The crawl archive for October 2017 is now available! The archive contains 3.65 billion web pages and over 300 TiB of uncompressed content.
Read More...The crawl archive for September 2017 is now available! The archive contains 3.01 billion web pages and over 250 TiB of uncompressed content.
Read More...The crawl archive for August 2017 is now available! The archive contains 3.28 billion+ web pages and over 280 TiB of uncompressed content.
Read More...We are pleased to announce the release of host-level and domain-level web graphs based on the published crawls of May, June, and July 2017. These graphs, along with ranked lists of hosts and domains, follow on our first host-level web graph (February, March, April 2017).
Read More...The crawl archive for July 2017 is now available! The archive contains 2.89 billion+ web pages and over 240 TiB of uncompressed content.
Read More...The crawl archive for June 2017 is now available! The archive contains 3.16 billion+ web pages and over 260 TiB of uncompressed content.
Read More...The crawl archive for May 2017 is now available! The archive contains 2.96 billion+ web pages and over 250 TiB of uncompressed content.
Read More...We are pleased to announce the release of a host-level web graph of recent monthly crawls (February, March, April 2017). The graph consists of 385 million nodes and 2.5 billion edges.
Read More...The crawl archive for April 2017 is now available! The archive contains 2.94 billion+ web pages and over 250 TiB of uncompressed content.
Read More...The crawl archive for March 2017 is now available! The archive contains 3.07 billion+ web pages and over 250 TiB of uncompressed content.
Read More...The crawl archive for February 2017 is now available! The archive contains 3.08 billion+ web pages and over 250 TiB of uncompressed content.
Read More...The crawl archive for January 2017 is now available! The archive contains more than 3.14 billion web pages and about 250 TiB of uncompressed content.
Read More...The crawl archive for December 2016 is now available! The archive contains more than 2.85 billion web pages.
Read More...The crawl archive for October 2016 is now available! The archive contains more than 3.25 billion web pages.
Read More...The crawl archive for September 2016 is now available! The archive contains more than 1.72 billion web pages.
Read More...We are pleased to announce the release of a new dataset containing news articles from news sites all over the world.
Read More...Together with the crawl archive for August 2016 we release two data sets containing robots.txt files and server responses with HTTP status code other than 200 (404s, redirects, etc.) The data may be useful to anyone interested in web science, with various applications in the field.
Read More...The crawl archive for August 2016 is now available! The archive contains more than 1.61 billion web pages.
Read More...The crawl archive for July 2016 is now available! The archive contains more than 1.73 billion web pages.
Read More...The crawl archive for June 2016 is now available! The archive contains more than 1.23 billion web pages.
Read More...The crawl archive for May 2016 is now available! More than 1.46 billion web pages are in the archive.
Read More...The crawl archive for April 2016 is now available! More than 1.33 billion webpages are in the archive.
Read More...It is a pleasure to officially announce that Sebastian Nagel joined Common Crawl as Crawl Engineer in April. Sebastian brings to Common Crawl a unique blend of experience, skills, knowledge (and enthusiasm!) to complement his role and the organization.
Read More...As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for February 2016 is now available! This crawl archive holds more than 1.73 billion urls.
Read More...As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for November 2015 is now available! This crawl archive is over 151TB in size and holds more than 1.82 billion urls.
Read More...
.svg)
