December 2014 Crawl Archive Available

The crawl archive for December 2014 is now available! This crawl archive is over 160TB in size and contains 2.08 billion webpages. The files are located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-52/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

Thanks again to blekko for their ongoing donation of URLs for our crawl!

November 2014 Crawl Archive Available

The crawl archive for November 2014 is now available! This crawl archive is over 135TB in size and contains 1.95 billion webpages. The files are located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-49/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

Thanks again to blekko for their ongoing donation of URLs for our crawl!

October 2014 Crawl Archive Available

The crawl archive for October 2014 is now available! This crawl archive is over 254TB in size and contains 3.72 billion webpages. The files are located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-42/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

Thanks again to blekko for their ongoing donation of URLs for our crawl!

September 2014 Crawl Archive Available

The crawl archive for September 2014 is now available! This crawl archive is over 220TB in size and contains 2.98 billion webpages. The files are located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-41/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

Thanks again to blekko for their ongoing donation of URLs for our crawl!

August 2014 Crawl Data Available

The August crawl of 2014 is now available! The new dataset is over 200TB in size containing approximately 2.8 billion webpages. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-35/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

Thanks again to blekko for their ongoing donation of URLs for our crawl!

July 2014 Crawl Data Available

The July crawl of 2014 is now available! The new dataset is over 266TB in size containing approximately 3.6 billion webpages. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-23/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

We’ve also released a Python library, gzipstream, that should enable easier access and processing of the Common Crawl dataset. We’d love for you to try it out!

Thanks again to blekko for their ongoing donation of URLs for our crawl!

Note: the original estimate for this crawl was 4 billion, but after full analytics were run, this estimate was revised.

April 2014 Crawl Data Available

The April crawl of 2014 is now available! The new dataset is over 183TB in size containing approximately 2.6 billion webpages. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-15/.

To assist with exploring and using the dataset, we’ve provided gzipped files that list:

By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

Thanks again to blekko for their ongoing donation of URLs for our crawl!

March 2014 Crawl Data Now Available

The March crawl of 2014 is now available! The new dataset contains approximately 2.8 billion webpages and is about 223TB in size. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-10/

We went a little deeper on this crawl than during our 2013 crawls so you’ll see more pages per domain.We’re working hard to get a few machines always crawling domains with large numbers of pages to go even deeper while still maintaining our politeness policy.

Thanks again to Blekko for their ongoing donation of URLs for our crawl.

Winter 2013 Crawl Data Now Available

The second crawl of 2013 is now available! In late November, we published the data from the first crawl of 2013 (see previous blog post for more detail on that dataset). The new dataset was collected at the end of 2013, contains approximately 2.3 billion webpages and is 148TB in size. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2013-48/

In 2013, we made changes to our crawling and post-processing systems. As detailed in the previous blog post, we switched file formats to the international standard WARC and WAT files. We also began using Apache Nutch to crawl – stay tuned for an upcoming blog post on our use of Nutch. The new crawling method relies heavily on the generous data donations from blekko and we are extremely grateful for blekko’s ongoing support!

In 2014 we plan to crawl much more frequently and publish fresh datasets at least once a month.

New Crawl Data Available!

We are very please to announce that new crawl data is now available! The data was collected in 2013, contains approximately 2 billion web pages and is 102TB in size (uncompressed).

We’ve made some changes to the data formats and the directory structure. Please see the details below and please share your thoughts and questions on the Common Crawl Google Group.

Format Changes

We have switched from ARC files to WARC files to better match what the industry has standardized on. WARC files allow us to include HTTP request information in the crawl data, add metadata about requests, and cross-reference the text extracts with the specific response that they were generated from. There are also many good open source tools for working with WARC files.

We have switched the metadata files from JSON to WAT files. The JSON format did not allow specifying the multiple offsets to files necessary for the WARC upgrade and WAT files provide more detail.


We have switched our text file format from Hadoop sequence files to WET files (WARC Encapsulated Text) that properly reference the original requests. This makes it far easier for your processes to disambiguate which text extracts belong to which specific page fetches.

Directory Structure

New crawl data is located in the commoncrawl bucket at /crawl-data/ path.

Under this base path, crawl data is organized hierarchically as follows:

  • CRAWL-NAME-YYYY-MM – The name of the crawl and year + week# initiated on

    • segments

      • SEGMENTNAME – A segment directory, typically a unix timestamp

        • warc – contains the WARC files with the HTTP request and responses for each fetch

          • CRAWL-NAME-YYYMMMDDSS-SEQ-MACHINE.warc.gz – individual WAT files

        • wat – contains WARC-encoded WAT files which describe the metadata of each request/response


          • CRAWL-NAME-YYYMMMDDSS-SEQ-MACHINE.warc.wat.gz – individual WAT files

        • wet – contains WARC-encoded WET files with text extractions from the HTTP responses

          • CRAWL-NAME-YYYMMMDDSS-SEQ-MACHINE.warc.wet.gz – individual WAT files

The 2013 wide web crawl data is located at /crawl-data/CC-MAIN-2013-20/ which represents the main CC crawl initiated during the 20th week of 2013.

Resources

More information about WARC can be found at http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf

Internet Archive publishes tools to process WARC and WAT files at https://github.com/internetarchive/ia-hadoop-tools and https://github.com/internetarchive/ia-web-commons

WET files can be treated as WARC files as they are simply conversion records as detailed in the WARC specification above.

More information about WAT files can be found at https://webarchive.jira.com/wiki/display/Iresearch/Web+Archive+Metadata+File+Specification.

Python WARC tools http://code.hanzoarchives.com/warc-tools


Erlang WARC sdk http://www.webarchivingbucket.com/#wsdk


A tool for exploring WARC files https://wiki.umiacs.umd.edu/adapt/index.php/WarcManager

A handy collection of links to tools for working with WARC files http://www.netpreserve.org/web-archiving/tools-and-software