Search results

Common Crawl - Blog - April 2015 Crawl Archive Available

May 28, 2015. April 2015 Crawl Archive Available. The crawl archive for April 2015 is now available! This crawl archive is over 168TB in size and holds more than 2.11 billion webpages. Stephen Merity.

Common Crawl - Blog - July 2015 Crawl Archive Available

August 15, 2015. July 2015 Crawl Archive Available. The crawl archive for June 2015 is now available! This crawl archive is over 145TB in size and holds more than 1.81 billion webpages. Stephen Merity.

Common Crawl - Blog - February 2015 Crawl Archive Available

March 31, 2015. February 2015 Crawl Archive Available. The crawl archive for February 2015 is now available! This crawl archive is over 145TB in size and over 1.9 billion webpages. Stephen Merity.

Common Crawl - Blog - August 2015 Crawl Archive Available

October 10, 2015. August 2015 Crawl Archive Available. The crawl archive for August 2015 is now available! This crawl archive is over 149TB in size and holds more than 1.84 billion webpages. Stephen Merity.

Common Crawl - Blog - January 2015 Crawl Archive Available

March 4, 2015. January 2015 Crawl Archive Available. The crawl archive for January 2015 is now available! This crawl archive is over 139TB in size and contains 1.82 billion webpages. Stephen Merity.

Common Crawl - Blog - March 2015 Crawl Archive Available

May 20, 2015. March 2015 Crawl Archive Available. The crawl archive for March 2015 is now available! This crawl archive is over 124TB in size and holds more than 1.64 billion webpages. Stephen Merity.

Common Crawl - Blog - June 2015 Crawl Archive Available

July 23, 2015. June 2015 Crawl Archive Available. The crawl archive for June 2015 is now available! This crawl archive is over 131TB in size and holds more than 1.67 billion webpages. Stephen Merity.

Common Crawl - Blog - May 2015 Crawl Archive Available

July 8, 2015. May 2015 Crawl Archive Available. The crawl archive for May 2015 is now available! This crawl archive is over 159TB in size and holds more than 2.05 billion webpages. Stephen Merity.

Common Crawl - Blog - November 2015 Crawl Archive Now Available

December 18, 2015. November 2015 Crawl Archive Now Available. As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for November 2015 is now available!

Common Crawl - Blog - September 2015 Crawl Archive Now Available

November 16, 2015. September 2015 Crawl Archive Now Available. As an interim crawl engineer for CommonCrawl, I am pleased to announce that the crawl archive for September 2015 is now available!

Common Crawl - Blog - 5 Good Reads in Big Open Data: Feb 13 2015

February 13, 2015. 5 Good Reads in Big Open Data: Feb 13 2015. What does it mean for the Open Web if users don't know they're on the internet? Via QUARTZ: “This is more than a matter of semantics.

Common Crawl - Blog - 5 Good Reads in Big Open Data: March 20 2015

March 20, 2015. 5 Good Reads in Big Open Data: March 20 2015.

Common Crawl - Blog - 5 Good Reads in Big Open Data: March 6 2015

March 6, 2015. 5 Good Reads in Big Open Data: March 6 2015. 2015: What do you think about Machines that think?

Common Crawl - Blog - 5 Good Reads in Big Open Data: February 27 2015

February 27, 2015. 5 Good Reads in Big Open Data: February 27 2015.

Common Crawl - Blog - 5 Good Reads in Big Open Data: Feb 20 2015

February 20, 2015. 5 Good Reads in Big Open Data: Feb 20 2015. A thriving ecosystem is the key for real viability of any technology.

Common Crawl - Blog - 5 Good Reads in Big Open Data: Feb 6 2015

February 6, 2015. 5 Good Reads in Big Open Data: Feb 6 2015.

Common Crawl - Blog - 5 Good Reads in Big Open Data: March 13 2015

March 13, 2015. 5 Good Reads in Big Open Data: March 13 2015. Jürgen Schmidhuber- Ask Me Anything - via Reddit: Jürgen has pioneered self-improving general problem solvers and Deep Learning Neural Networks for decades.

Common Crawl - Blog - 5 Good Reads in Big Open Data: March 26 2015

March 26, 2015. 5 Good Reads in Big Open Data: March 26 2015.

Common Crawl - Blog - December 2014 Crawl Archive Available

January 9, 2015. December 2014 Crawl Archive Available. The crawl archive for December 2014 is now available! This crawl archive is over 160TB in size and contains 2.08 billion webpages. Stephen Merity.

Common Crawl - Blog - Announcing the Common Crawl Index!

April 8, 2015. Announcing the Common Crawl Index! This is a guest post by Ilya Kreymer, a dedicated volunteer who has gifted large amounts of time, effort and talent to Common Crawl.

Common Crawl - Blog - WikiReverse- Visualizing Reverse Links with the Common Crawl Archive

February 18, 2015. WikiReverse- Visualizing Reverse Links with the Common Crawl Archive. This is a guest blog post by Ross Fairbanks, a software developer based in Barcelona. He mainly develops in Ruby and is interested in open data and cloud computing.

Common Crawl - Blog - Please Donate To Common Crawl!

December 10, 2014. Please Donate To Common Crawl! Big data has the potential to change the world. The talent exists and the tools are already there. What’s lacking is access to data.

Common Crawl - Blog - The Promise of Open Government Data & Where We Go Next

January 29, 2015. The Promise of Open Government Data & Where We Go Next.

Common Crawl - Blog - Web Image Size Prediction for Efficient Focused Image Crawling

August 20, 2015. Web Image Size Prediction for Efficient Focused Image Crawling. This is a guest blog post by Katerina Andreadou, a research assistant at CERTH, specializing in multimedia analysis and web crawling.

Common Crawl - Blog - Host- and Domain-Level Web Graphs Nov/Dec/Jan 2017-2018

February 8, 2018. Host- and Domain-Level Web Graphs Nov/Dec/Jan 2017-2018. We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of November, December 2017 and January 2018.

Common Crawl - Blog - Host- and Domain-Level Web Graphs Nov/Dec/Jan 2018 - 2019

February 20, 2019. Host- and Domain-Level Web Graphs Nov/Dec/Jan 2018 - 2019. We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of November, December 2018 and January 2019.

Common Crawl - Blog - OSCON 2012

June 19, 2012. OSCON 2012. We're just one month away from one of the biggest and most exciting events of the year, O'Reilly's Open Source Convention (OSCON). This year's conference will be held July 16th-20th in Portland, Oregon. Allison Domicone.

Common Crawl - Blog - Common Crawl's First In-House Web Graph

May 22, 2017. Common Crawl's First In-House Web Graph. We are pleased to announce the release of a host-level web graph of recent monthly crawls (February, March, April 2017). The graph consists of 385 million nodes and 2.5 billion edges. Sebastian Nagel.

Common Crawl - Blog - Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019

November 12, 2019. Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019. We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019.

Common Crawl - Blog - Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020.

Common Crawl - Blog - Host- and Domain-Level Web Graphs May/June/July 2019

August 8, 2019. Host- and Domain-Level Web Graphs May/June/July 2019. We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2019.

Common Crawl - Use Cases

Overview of the original Common Crawl crawler (in use 2008-2013) discussing the Hadoop data processing pipeline, PageRank implementation, and the techniques used to optimize Hadoop. The Web of Data and Web Data Commons.

Common Crawl - Blog - Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019

May 9, 2019. Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019. We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019.

Common Crawl - Blog - Analyzing a Web graph with 129 billion edges using FlashGraph

February 25, 2015. Analyzing a Web graph with 129 billion edges using FlashGraph. This is a guest blog post by Da Zheng, the architect and main developer of the FlashGraph project.

Common Crawl - Blog - Winter 2013 Crawl Data Now Available

January 8, 2014. Winter 2013 Crawl Data Now Available. The second crawl of 2013 is now available! In late November, we published the data from the first crawl of 2013.

Common Crawl - Blog - Evaluating graph computation systems

April 1, 2015. Evaluating graph computation systems. This is a guest blog post by Frank McSherry, a computer science researcher active in the area of large scale data analysis.

Common Crawl - Blog - November 2014 Crawl Archive Available

December 24, 2014. November 2014 Crawl Archive Available. The crawl archive for November 2014 is now available! This crawl archive is over 135TB in size and contains 1.95 billion webpages. Stephen Merity.

Common Crawl - Blog - August 2014 Crawl Data Available

September 22, 2014. August 2014 Crawl Data Available. The August crawl of 2014 is now available! The new dataset is over 200TB in size containing approximately 2.8 billion webpages. Stephen Merity.

Common Crawl - Blog - 2012 Crawl Data Now Available

July 16, 2012. 2012 Crawl Data Now Available. I am very happy to announce that Common Crawl has released 2012 crawl data as well as a number of significant enhancements to our example library and help pages. Common Crawl Foundation.

Common Crawl - Blog - October 2014 Crawl Archive Available

November 20, 2014. October 2014 Crawl Archive Available. The crawl archive for October 2014 is now available! This crawl archive is over 254TB in size and contains 3.72 billion webpages. Stephen Merity.

Common Crawl - Blog - July 2014 Crawl Data Available

August 7, 2014. July 2014 Crawl Data Available. The July crawl of 2014 is now available! The new dataset is over 266TB in size containing approximately 3.6 billion webpages. Stephen Merity.

Common Crawl - Blog - September 2014 Crawl Archive Available

November 12, 2014. September 2014 Crawl Archive Available. The crawl archive for September 2014 is now available! This crawl archive is over 220TB in size and contains 2.98 billion webpages. Stephen Merity.

Common Crawl - Blog - April 2014 Crawl Data Available

July 16, 2014. April 2014 Crawl Data Available. The April crawl of 2014 is now available! The new dataset is over 183TB in size containing approximately 2.6 billion webpages. Stephen Merity.

Common Crawl - Blog - September 2019 crawl archive now available

September 28, 2019. September 2019 crawl archive now available. The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th.

Common Crawl - Blog - February 2019 crawl archive now available

March 1, 2019. February 2019 crawl archive now available. The crawl archive for February 2019 is now available! It contains 2.9 billion web pages or 225 TiB of uncompressed content, crawled between February 15th and 24th. Sebastian Nagel.

Common Crawl - Blog - June 2018 Crawl Archive Now Available

July 2, 2018. June 2018 Crawl Archive Now Available. The crawl archive for June 2018 is now available! The archive contains 3.05 billion web pages and 235 TiB of uncompressed content, crawled between June 18th and 25th. Sebastian Nagel.

Common Crawl - Blog - August 2016 Crawl Archive Now Available

September 16, 2016. August 2016 Crawl Archive Now Available. The crawl archive for August 2016 is now available! The archive contains more than 1.61 billion web pages. Sebastian Nagel. Sebastian is a Distinguished Engineer with Common Crawl.

Common Crawl - Blog - September 2016 Crawl Archive Now Available

October 7, 2016. September 2016 Crawl Archive Now Available. The crawl archive for September 2016 is now available! The archive contains more than 1.72 billion web pages. Sebastian Nagel. Sebastian is a Distinguished Engineer with Common Crawl.

Common Crawl - Blog - September 2017 Crawl Archive Now Available

September 29, 2017. September 2017 Crawl Archive Now Available. The crawl archive for September 2017 is now available! The archive contains 3.01 billion web pages and over 250 TiB of uncompressed content. Sebastian Nagel.

Common Crawl - Blog - January 2018 Crawl Archive Now Available

January 29, 2018. January 2018 Crawl Archive Now Available. The crawl archive for January 2018 is now available! The archive contains 3.4 billion web pages and 270 TiB of uncompressed content, crawled between January 16th and Jan 24th. Sebastian Nagel.

Common Crawl - Blog - September 2018 crawl archive now available

October 3, 2018. September 2018 crawl archive now available. The crawl archive for September 2018 is now available! It contains 2.8 billion web pages and 220 TiB of uncompressed content, crawled between September 17th and 26th. Sebastian Nagel.

Common Crawl - Blog - February 2017 Crawl Archive Now Available

March 10, 2017. February 2017 Crawl Archive Now Available. The crawl archive for February 2017 is now available! The archive contains 3.08 billion+ web pages and over 250 TiB of uncompressed content. Sebastian Nagel.

Common Crawl - Blog - December 2017 Crawl Archive Now Available

December 22, 2017. December 2017 Crawl Archive Now Available. The crawl archive for December 2017 is now available! The archive contains 2.9 billion web pages and over 240 TiB of uncompressed content. Sebastian Nagel.

Common Crawl - Blog - May 2016 Crawl Archive Now Available

June 19, 2016. May 2016 Crawl Archive Now Available. The crawl archive for May 2016 is now available! More than 1.46 billion web pages are in the archive. Sebastian Nagel. Sebastian is a Distinguished Engineer with Common Crawl.

Common Crawl - Blog - November 2018 crawl archive now available

November 29, 2018. November 2018 crawl archive now available. The crawl archive for November 2018 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between November 12th and 22nd. Sebastian Nagel.

Common Crawl - Blog - October 2018 crawl archive now available

October 30, 2018. October 2018 crawl archive now available. The crawl archive for October 2018 is now available! It contains 3.0 billion web pages and 240 TiB of uncompressed content, crawled between October 15th and 24th. Sebastian Nagel.

Common Crawl - Blog - August 2019 crawl archive now available

August 30, 2019. August 2019 crawl archive now available. The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th. Sebastian Nagel.

Common Crawl - Blog - April 2018 Crawl Archive Now Available

May 2, 2018. April 2018 Crawl Archive Now Available. The crawl archive for April 2018 is now available! The archive contains 3.1 billion web pages and 230 TiB of uncompressed content, crawled between April 19th and 27th. Sebastian Nagel.

Common Crawl - Blog - July 2016 Crawl Archive Now Available

August 9, 2016. July 2016 Crawl Archive Now Available. The crawl archive for July 2016 is now available! The archive contains more than 1.73 billion web pages. Sebastian Nagel. Sebastian is a Distinguished Engineer with Common Crawl.

Common Crawl - Blog - December 2016 Crawl Archive Now Available

December 16, 2016. December 2016 Crawl Archive Now Available. The crawl archive for December 2016 is now available! The archive contains more than 2.85 billion web pages. Sebastian Nagel. Sebastian is a Distinguished Engineer with Common Crawl.