Search results

Common Crawl - Blog - 3.25 Billion Pages Crawled in July 2018

< Back to Blog. July 28, 2018. 3.25 Billion Pages Crawled in July 2018. The crawl archive for July 2018 is now available! The archive contains 3.25 billion web pages and 255 TiB of uncompressed content, crawled between July 15th and 23th. Sebastian Nagel.

Common Crawl - Blog

Common Crawl Blog. The latest news, interviews, technologies, and resources. Common Crawl Blog. The Data. Overview. Web Graphs. Latest Crawl. Resources. Get Started. Blog. Examples. Use Cases. CCBot. Infra Status. FAQ. Community. Research Papers.

Common Crawl - Blog - August Crawl Archive Introduces Language Annotations

< Back to Blog. August 26, 2018. August Crawl Archive Introduces Language Annotations. The crawl archive for August 2018 is now available! It contains 2.65 billion web pages and 220 TiB of uncompressed content, crawled between August 14th and 22th.

Common Crawl - Blog - February/March 2021 crawl archive now available

< Back to Blog. March 14, 2021. February/March 2021 crawl archive now available. The crawl archive for February/March 2021 is now available!

Common Crawl - Blog - January 2020 crawl archive now available

< Back to Blog. February 3, 2020. January 2020 crawl archive now available. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th.

Common Crawl - Blog - May/June 2020 crawl archive now available

< Back to Blog. June 10, 2020. May/June 2020 crawl archive now available. The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th.

Common Crawl - Blog - Twelve steps to running your Ruby code across five billion web pages

< Back to Blog. March 26, 2012. Twelve steps to running your Ruby code across five billion web pages. The following is a guest blog post by Pete Warden, a member of the Common Crawl Advisory Board.

Common Crawl - Blog - Analysis of the NCSU Library URLs in the Common Crawl Index

< Back to Blog. January 16, 2013. Analysis of the NCSU Library URLs in the Common Crawl Index. Last week we announced the Common Crawl URL Index.

Common Crawl - Blog - August 2019 crawl archive now available

< Back to Blog. August 30, 2019. August 2019 crawl archive now available. The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th. Sebastian Nagel.

Common Crawl - Blog - April 2019 crawl archive now available

< Back to Blog. April 30, 2019. April 2019 crawl archive now available. The crawl archive for April 2019 is now available! It contains 2.5 billion web pages or 198 TiB of uncompressed content, crawled between April 18th and 26th. Sebastian Nagel.

Common Crawl - Blog - Answers to Recent Community Questions

It was wonderful to see our first blog post and the. great piece. by. Marshall Kirkpatrick. on ReadWriteWeb generate so much interest in Common Crawl last week! There were many questions raised on Twitter and in the comment sections of our blog, RWW and.

Common Crawl - Blog - September 2018 crawl archive now available

< Back to Blog. October 3, 2018. September 2018 crawl archive now available. The crawl archive for September 2018 is now available! It contains 2.8 billion web pages and 220 TiB of uncompressed content, crawled between September 17th and 26th.

Common Crawl - Blog - OSCON 2012

< Back to Blog. June 19, 2012. OSCON 2012. We're just one month away from one of the biggest and most exciting events of the year, O'Reilly's Open Source Convention (OSCON). This year's conference will be held July 16th-20th in Portland, Oregon.

Common Crawl - Blog - Welcome, Sebastian!

< Back to Blog. May 13, 2016. Welcome, Sebastian! It is a pleasure to officially announce that Sebastian Nagel joined Common Crawl as Crawl Engineer in April.

Common Crawl - Blog - September 2019 crawl archive now available

< Back to Blog. September 28, 2019. September 2019 crawl archive now available. The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th.

Common Crawl - Blog - May/June 2023 crawl archive now available

< Back to Blog. June 21, 2023. May/June 2023 crawl archive now available. The crawl archive for May/June 2023 is now available! The data was crawled May 27 – June 11 and contains 3.1 billion web pages or 390 TiB of uncompressed content.

Common Crawl - Blog - June 2021 crawl archive now available

< Back to Blog. June 28, 2021. June 2021 crawl archive now available. The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content.

Common Crawl - Blog - April 2021 crawl archive now available

< Back to Blog. April 27, 2021. April 2021 crawl archive now available. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content.

Common Crawl - Blog - March/April 2023 crawl archive now available

< Back to Blog. April 6, 2023. March/April 2023 crawl archive now available. The crawl archive for March/April 2023 is now available! The data was crawled March 20 – April 2 and contains 3.1 billion web pages or 400 TiB of uncompressed content.

Common Crawl - Blog - August 2020 crawl archive now available

< Back to Blog. August 19, 2020. August 2020 crawl archive now available. The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th.

Common Crawl - Blog - January 2022 crawl archive now available

< Back to Blog. February 2, 2022. January 2022 crawl archive now available. The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content.

Common Crawl - Blog - May 2021 crawl archive now available

< Back to Blog. May 23, 2021. May 2021 crawl archive now available. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content.

Common Crawl - Blog - March/April 2020 crawl archive now available

< Back to Blog. April 14, 2020. March/April 2020 crawl archive now available. The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th.

Common Crawl - Blog - October 2021 crawl archive now available

< Back to Blog. November 1, 2021. October 2021 crawl archive now available. The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content.

Common Crawl - Blog - December 2019 crawl archive now available

< Back to Blog. December 19, 2019. December 2019 crawl archive now available. The crawl archive for December 2019 is now available! It contains 2.45 billion web pages or 234 TiB of uncompressed content, crawled between December 5th and 16th.

Common Crawl - Blog - October 2019 crawl archive now available

< Back to Blog. October 29, 2019. October 2019 crawl archive now available. The crawl archive for October 2019 is now available! It contains 3.0 billion web pages or 280 TiB of uncompressed content, crawled between October 13th and 24th.

Common Crawl - Blog - September 2021 crawl archive now available

< Back to Blog. October 4, 2021. September 2021 crawl archive now available. The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content.

Common Crawl - Blog - January 2021 crawl archive now available

< Back to Blog. February 2, 2021. January 2021 crawl archive now available. The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content.

Common Crawl - Blog - November/December 2022 crawl archive now available

< Back to Blog. December 14, 2022. November/December 2022 crawl archive now available. The crawl archive for November/December 2022 is now available!

Common Crawl - Blog - October 2020 crawl archive now available

< Back to Blog. November 7, 2020. October 2020 crawl archive now available. The crawl archive for October 2020 is now available!

Common Crawl - Blog - August 2022 crawl archive now available

< Back to Blog. August 22, 2022. August 2022 crawl archive now available. The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content.

Common Crawl - Blog - May 2022 crawl archive now available

< Back to Blog. June 2, 2022. May 2022 crawl archive now available. The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content.

Common Crawl - Blog - January/February 2023 crawl archive now available

< Back to Blog. February 16, 2023. January/February 2023 crawl archive now available. The crawl archive for January/February 2023 is now available!

Common Crawl - Blog - September 2020 crawl archive now available

< Back to Blog. October 7, 2020. September 2020 crawl archive now available. The crawl archive for September 2020 is now available!

Common Crawl - Blog - June/July 2022 crawl archive now available

< Back to Blog. July 13, 2022. June/July 2022 crawl archive now available. The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content.

Common Crawl - Blog - November/December 2020 crawl archive now available

< Back to Blog. December 10, 2020. November/December 2020 crawl archive now available. The crawl archive for November/December 2020 is now available!

Common Crawl - Blog - September/October 2022 crawl archive now available

< Back to Blog. October 11, 2022. September/October 2022 crawl archive now available. The crawl archive for September/October 2022 is now available!

Common Crawl - Blog - URL Search Tool!

< Back to Blog. March 5, 2013. URL Search Tool! A couple months ago we announced the creation of the Common Crawl URL Index and followed it up with a guest post by Jason Ronallo describing how he had used the URL Index.

Common Crawl - Blog - News Dataset Available

< Back to Blog. October 4, 2016. News Dataset Available. We are pleased to announce the release of a new dataset containing news articles from news sites all over the world. Sebastian Nagel. Sebastian is a Distinguished Engineer with Common Crawl.

Common Crawl - Blog - Web Data Commons

< Back to Blog. March 22, 2012. Web Data Commons.

Common Crawl - Blog - Data 2.0 Summit

< Back to Blog. March 28, 2012. Data 2.0 Summit. Next week a few members of the Common Crawl team are going the Data 2.0 Summit in San Francisco. Common Crawl Foundation. Common Crawl - Open Source Web Crawling data‍.

Common Crawl - Blog - November 2019 crawl archive now available

< Back to Blog. November 27, 2019. November 2019 crawl archive now available. The crawl archive for November 2019 is now available!

Common Crawl - Blog - July/August 2021 crawl archive now available

< Back to Blog. August 9, 2021. July/August 2021 crawl archive now available. The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content.

Common Crawl - Blog - February 2020 crawl archive now available

< Back to Blog. March 4, 2020. February 2020 crawl archive now available. The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th.

Common Crawl - Blog - Host- and Domain-Level Web Graphs Aug/Sept/Oct 2017

< Back to Blog. November 27, 2017. Host- and Domain-Level Web Graphs Aug/Sept/Oct 2017. We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September, and October 2017.

Common Crawl - Blog - July 2020 crawl archive now available

< Back to Blog. July 20, 2020. July 2020 crawl archive now available. The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th.

Common Crawl - Blog - Announcing the Common Crawl Index!

< Back to Blog. April 8, 2015. Announcing the Common Crawl Index! This is a guest post by Ilya Kreymer, a dedicated volunteer who has gifted large amounts of time, effort and talent to Common Crawl.

Common Crawl - Blog - November/December 2023 Crawl Archive Now Available

< Back to Blog. December 15, 2023. November/December 2023 Crawl Archive Now Available. The crawl archive for November/December 2023 is now available.

Common Crawl - Blog - September/October 2023 crawl archive now available

< Back to Blog. October 12, 2023. September/October 2023 crawl archive now available. The crawl archive for September/October 2023 is now available!

Common Crawl - Blog - February/March 2024 Crawl Archive Now Available

< Back to Blog. March 11, 2024. February/March 2024 Crawl Archive Now Available. The crawl archive for February/March 2024 is now available.

Common Crawl - Blog - November/December 2021 crawl archive now available

< Back to Blog. December 14, 2021. November/December 2021 crawl archive now available. The crawl archive for November/December 2021 is now available!

Common Crawl - Blog - Analyzing a Web graph with 129 billion edges using FlashGraph

< Back to Blog. February 25, 2015. Analyzing a Web graph with 129 billion edges using FlashGraph. This is a guest blog post by Da Zheng, the architect and main developer of the FlashGraph project.

Common Crawl - Blog - July 2019 crawl archive now available

< Back to Blog. July 30, 2019. July 2019 crawl archive now available. The crawl archive for July 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between July 15th and 24th. Sebastian Nagel.

Common Crawl - Blog - 2012 Crawl Data Now Available

< Back to Blog. July 16, 2012. 2012 Crawl Data Now Available. I am very happy to announce that Common Crawl has released 2012 crawl data as well as a number of significant enhancements to our example library and help pages. Common Crawl Foundation.

Common Crawl - Blog - May 2019 crawl archive now available

< Back to Blog. May 31, 2019. May 2019 crawl archive now available. The crawl archive for May 2019 is now available! It contains 2.65 billion web pages or 220 TiB of uncompressed content, crawled between May 19th and 27th. Sebastian Nagel.

Common Crawl - Blog - Common Crawl's Advisory Board

< Back to Blog. February 15, 2012. Common Crawl's Advisory Board. As part of our ongoing effort to grow Common Crawl into a truly useful and innovative tool, we recently formed an Advisory Board to guide us in our efforts.

Common Crawl - Blog - March/April 2024 Newsletter

< Back to Blog. March 26, 2024. March/April 2024 Newsletter. We're excited to share an update on some of our recent projects and initiatives in this newsletter! Common Crawl Foundation. Common Crawl - Open Source Web Crawling data‍. Table of Contents.

Common Crawl - Blog - Common Crawl URL Index

< Back to Blog. January 8, 2013. Common Crawl URL Index. We are thrilled to announce that Common Crawl now has a URL index! Scott Robertson, founder of triv.io graciously donated his time and skills to creating this valuable tool. Scott Robertson.

Common Crawl - Blog - Strata Conference + Hadoop World

< Back to Blog. August 3, 2012. Strata Conference + Hadoop World. This year's Strata Conference teams up with Hadoop World for what promises to be a powerhouse convening in NYC from October 23-25.

Common Crawl - Blog - February 2019 crawl archive now available

< Back to Blog. March 1, 2019. February 2019 crawl archive now available. The crawl archive for February 2019 is now available! It contains 2.9 billion web pages or 225 TiB of uncompressed content, crawled between February 15th and 24th. Sebastian Nagel.