< Back to Blog
March 26, 2014

March 2014 Crawl Data Now Available

The March crawl of 2014 is now available! The new dataset contains approximately 2.8 billion webpages and is about 223TB in size.
Common Crawl Foundation
Common Crawl Foundation
Common Crawl - Open Source Web Crawling data‍

The March crawl of 2014 is now available! The new dataset contains approximately 2.8 billion webpages and is about 223TB in size. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-10/

We went a little deeper on this crawl than during our 2013 crawls so you'll see more pages per domain.

Data Type File List #Files Total Size
Compressed (TiB)
Segments segment.paths.gz 557
WARC warc.paths.gz 55700 33.19
WAT wat.paths.gz 55699 11.43
WET wet.paths.gz 55699 4.01
URL index files cc-index.paths.gz 302 0.14
Columnar URL index files cc-index-table.paths.gz 300 0.15

We're working hard to get a few machines always crawling domains with large numbers of pages to go even deeper while still maintaining our politeness policy.

Thanks again to Blekko for their ongoing donation of URLs for our crawl.

Errata
No items found.
This release was authored by:
No items found.