The March crawl of 2014 is now available! The new dataset contains approximately 2.8 billion webpages and is about 223TB in size. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-10/
We went a little deeper on this crawl than during our 2013 crawls so you'll see more pages per domain.
We're working hard to get a few machines always crawling domains with large numbers of pages to go even deeper while still maintaining our politeness policy.
Thanks again to Blekko for their ongoing donation of URLs for our crawl.
.svg)