article
Hacking habits

Access Common Crawl Data That is Stored on S3

Crawling billions of pages online is a large undertaking. Thanks to the non profit companies, Common Crawl does the heavy lifting for you…

interview
Library of Congress

Machine Scale Analysis of Digital Collections: An Interview with Lisa Green of Common Crawl

How do we make digital collections available at scale for today’s scholars and researchers? Lisa Green, director of Common Crawl, tackled this and related questions in her keynote address at Digital Preservation 2013…

article
MIT Technology Review

A Free Database of the Entire Web May Spawn the Next Google

A nonprofit called Common Crawl is now using its own Web crawler and making a giant copy of the Web that it makes accessible to anyone…

More Media

creative commons — Big changes for CC Search beta: updates released today!
Forbes — Common Crawl And Unlocking Web Archives For Research
Yelp Engineering Blog — Analyzing the Web For the Price of a Sandwich
Hacking Habits Blog — Access Common Crawl Data That is Stored on S3
Library of Congress — Machine Scale Analysis of Digital Collections: An Interview with Lisa Green of Common Crawl
Big Data News — Big data set – 3.5 billion web pages – made available for all of us
The McGill Daily — Doing the web crawl
ArnoldIT Blog — Proof Behind Common Crawl Claims
Swiftkey — Why Open Data Matters
NBC Press:Here — Gil Elbaz and Common Crawl [Video]
The Verge — Common Crawl: going after Google on a non-profit budget
FileHippo — Will Common Crawl Be the Next Google?
The Tech Panda — Common Crawl – Free Database Of The Entire Web, Competition For Google
NonProfit Quarterly — Meet Common Crawl, the Nonprofit That Could Reshape the Web
MIT Technology Review — A Free Database of the Entire Web May Spawn the Next Google
The AWS Report — The AWS Report – Lisa Green of Common Crawl
The H Open — Blekko donates 81 terabytes of data to Common Crawl
Blekko Blog — Blekko Donates Search Data to Common Crawl
Lucky Oyster Blog — Data Mining the Web: $100 Worth of Priceless
Data Driven Intelligence — How to crawl a quarter billion webpages in 40 hours
High Scalability — The Anatomy Of Search Technology: Crawling Using Combinators
Semantic Web — Common Crawl Founder Gil Elbaz Speaks About New Relationship With Amazon…
WebProNews — Blekko Makes “Donation” Of Search Data To Common Crawl
TWIST — Episode: 222: Gil Elbaz and Nova Spivack…
Occam's Machete Machine Learning — Common Crawl Envy
I Programmer — Common Crawl – now everyone can be Google
DEJANSEO — Common Crawl: The Open Search Engine
Read Write Web — New 5 Billion Page Web Index with Page Rank Now Available for Free from Common Crawl Foundation