Data Sets Containing Robots.txt Files and Non-200 Responses
September 16, 2016
Together with the crawl archive for August 2016 we release two data sets containing robots.txt files and server responses with HTTP status code other than 200 (404s, redirects, etc.) The data may be useful to anyone interested in web science, with various applications in the field.
Read More...July 2016 Crawl Archive Now Available
August 9, 2016
The crawl archive for July 2016 is now available! The archive contains more than 1.73 billion web pages.
Read More...June 2016 Crawl Archive Now Available
July 14, 2016
The crawl archive for June 2016 is now available! The archive contains more than 1.23 billion web pages.
Read More...May 2016 Crawl Archive Now Available
June 19, 2016
The crawl archive for May 2016 is now available! More than 1.46 billion web pages are in the archive.
Read More...April 2016 Crawl Archive Now Available
May 24, 2016
The crawl archive for April 2016 is now available! More than 1.33 billion webpages are in the archive.
Read More...Welcome, Sebastian!
May 13, 2016
It is a pleasure to officially announce that Sebastian Nagel joined Common Crawl as Crawl Engineer in April. Sebastian brings to Common Crawl a unique blend of experience, skills, knowledge (and enthusiasm!) to complement his role and the organization.
Read More...February 2016 Crawl Archive Now Available
February 29, 2016
As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for February 2016 is now available! This crawl archive holds more than 1.73 billion urls.
Read More...November 2015 Crawl Archive Now Available
December 18, 2015
As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for November 2015 is now available! This crawl archive is over 151TB in size and holds more than 1.82 billion urls.
Read More...September 2015 Crawl Archive Now Available
November 16, 2015
As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for September 2015 is now available! This crawl archive is over 106TB in size and holds more than 1.32 billion urls.
Read More...August 2015 Crawl Archive Available
October 10, 2015
The crawl archive for August 2015 is now available! This crawl archive is over 149TB in size and holds more than 1.84 billion webpages.
Read More...Web Image Size Prediction for Efficient Focused Image Crawling
August 20, 2015
This is a guest blog post by Katerina Andreadou, a research assistant at CERTH, specializing in multimedia analysis and web crawling. In the context of using Web image content for analysis and retrieval, it is typically necessary to perform large-scale image crawling. In our web image crawler setup, we noticed that a serious bottleneck pertains to the fetching of image content, since for each web page a large number of HTTP requests need to be issued to download all included image elements.
Read More...July 2015 Crawl Archive Available
August 15, 2015
The crawl archive for June 2015 is now available! This crawl archive is over 145TB in size and holds more than 1.81 billion webpages.
Read More...June 2015 Crawl Archive Available
July 23, 2015
The crawl archive for June 2015 is now available! This crawl archive is over 131TB in size and holds more than 1.67 billion webpages.
Read More...May 2015 Crawl Archive Available
July 8, 2015
The crawl archive for May 2015 is now available! This crawl archive is over 159TB in size and holds more than 2.05 billion webpages.
Read More...April 2015 Crawl Archive Available
May 28, 2015
The crawl archive for April 2015 is now available! This crawl archive is over 168TB in size and holds more than 2.11 billion webpages.
Read More...March 2015 Crawl Archive Available
May 20, 2015
The crawl archive for March 2015 is now available! This crawl archive is over 124TB in size and holds more than 1.64 billion webpages.
Read More...Announcing the Common Crawl Index!
April 8, 2015
This is a guest post by Ilya Kreymer, a dedicated volunteer who has gifted large amounts of time, effort and talent to Common Crawl. He previously worked at the Internet Archive and led the Wayback Machine development, which included building large indexes of WARC files.
Read More...Evaluating graph computation systems
April 1, 2015
This is a guest blog post by Frank McSherry, a computer science researcher active in the area of large scale data analysis. While at Microsoft Research he co-invented differential privacy, and lead the Naiad streaming dataflow project. His current interests involve understanding and improving performance in scalable data processing systems.
Read More...February 2015 Crawl Archive Available
March 31, 2015
The crawl archive for February 2015 is now available! This crawl archive is over 145TB in size and over 1.9 billion webpages.
Read More...5 Good Reads in Big Open Data: March 26 2015
March 26, 2015
Analyzing the Web For the Price of a Sandwich - via Yelp Engineering Blog: a Common Crawl use case from the December 2014 Dataset finds 748 million US phone numbers “I wanted to explore the Common Crawl in more depth, so I came up with a (somewhat contrived) use case of helping consumers find the web pages for local businesses…”
Read More...5 Good Reads in Big Open Data: March 20 2015
March 20, 2015
Startup Orbital Insight uses deep learning and finds financially useful information in aerial imagery - via MIT Technology Review: “To predict retail sales based on retailers’ parking lots, humans at Orbital Insights use Google Street View images to pinpoint the exact location of the stores’ entrances. Satellite imagery is acquired from a number of commercial suppliers, some of it refreshed daily. Software then monitors the density of cars and the frequency with which they enter the lots.”
Read More...5 Good Reads in Big Open Data: March 13 2015
March 13, 2015
Jürgen Schmidhuber- Ask Me Anything - via Reddit: Jürgen has pioneered self-improving general problem solvers and Deep Learning Neural Networks for decades. He is the recipient of the 2013 Helmholtz Award of the International Neural Networks Society.
Read More...5 Good Reads in Big Open Data: March 6 2015
March 6, 2015
2015: What do you think about Machines that think? - via Edge: A.I isn't so artificial “With these kind of software challenges, and given the very real technology-driven threats to our species already at hand, why worry about malevolent A.I.? For decades to come, at least, we are clearly more threatened by like trans-species plagues, extreme resource depletion, global warming, and nuclear warfare…”
Read More...January 2015 Crawl Archive Available
March 4, 2015
The crawl archive for January 2015 is now available! This crawl archive is over 139TB in size and contains 1.82 billion webpages.
Read More...5 Good Reads in Big Open Data: February 27 2015
February 27, 2015
Hadoop is the Glue for Big Data - via StreetWise Journal: Startups trying to build a successful big data infrastructure should "welcome...and be protective" of open source software like Hadoop. The future and innovation of Big Data depends on it.
Read More...
.svg)
