Search results
June 2, 2022. May 2022 crawl archive now available. The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content.…
February 2, 2022. January 2022 crawl archive now available. The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content.…
August 22, 2022. August 2022 crawl archive now available. The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content.…
October 11, 2022. September/October 2022 crawl archive now available. The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content.…
December 14, 2022. November/December 2022 crawl archive now available. The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content.…
July 13, 2022. June/July 2022 crawl archive now available. The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content.…
March 16, 2022. Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022.…
March 15, 2023. Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023.…
September 23, 2022. Host- and Domain-Level Web Graphs May, June/July and August 2022. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022.…
A bad configuration was checked into our exclusion list on Sep 22, 2022 and was fixed on Oct 27, 2023. The configuration blocked a number of 2–level domains, meaning they were not included in certain crawls.…
February 11, 2026. CC-Citations: A Visualization of Research Papers Referencing Common Crawl. We are proud to release an interactive visualization of thousands of research papers using or citing Common Crawl data. Malte Ostendorff.…
May, June/July, August 2022 Web Graph (cc-main-2022-may-jun-aug-domain). and the following Web Graph releases. Affected Web Graphs. The Data. Overview. CDXJ Index. URL Index. Web Graphs. Latest Crawl. Crawl Stats. Graph Stats. Errata. Resources.…
March 1, 2022. Introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data. Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone.…
May 5, 2024. Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.…
February 1, 2025. Host- and Domain-Level Web Graphs November/December 2024 and January 2025. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2024 and January 2025.…
February 24, 2026. Host- and Domain-Level Web Graphs December 2025 and January/February 2026.…
January 30, 2026. Host- and Domain-Level Web Graphs November/December 2025 and January 2026.…
February 25, 2025. Host- and Domain-Level Web Graphs December 2024 and January/February 2025.…
A. 2024 Mozilla Foundation study. described the dataset as having "laid the infrastructural foundation" for the generative AI boom, and found that at least 64% of 47 large language models published between 2019 and 2023 were trained on filtered versions of…
February 10, 2021. Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021.…
March 14, 2024. Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024.…
July 16, 2024. The Environmental Impact of the Cloud - the Common Crawl Case Study. Looking at tools (Green Software) and methodologies to evaluate the environmental impact of the cloud (a nascent activity coined GreenOps). Julien Nioche.…
March 26, 2024. March/April 2024 Newsletter. We're excited to share an update on some of our recent projects and initiatives in this newsletter! Common Crawl Foundation.…
February 3, 2025. January/February 2025 Newsletter. We’re happy to share our January/February 2025 newsletter with updates and insights from the world of open data and web archiving. Jen English.…
August 26, 2025. July/August 2025 Newsletter. We are pleased to release our newsletter for July and August 2025, with updates on our team's activities. Jen English.…
June 25, 2024. May/June 2024 Newsletter. We’re pleased to share our newsletter for May/June 2024, featuring the latest updates, events, and highlights from our community. Greg Lindahl. Greg is Chief Technology Officer at the Common Crawl Foundation.…
September 10, 2024. August/September 2024 Newsletter. We're pleased to announce our newsletter for August and September 2024. Jen English.…
November 3, 2025. October/November 2025 Newsletter. Check out our newsletter for October/November 2025, with updates on what we've been up to. Common Crawl Foundation.…
June 24, 2025. May/June 2025 Newsletter. We're happy to share our newsletter for May/June 2025 with updates from our team. Jen English.…
November 25, 2024. October/November 2024 Newsletter. We’re pleased to announce this month's newsletter, featuring key updates, upcoming events, and community highlights. Jen English.…
April 14, 2025. March/April 2025 Newsletter. We're excited to share our newsletter for March/April 2025 with updates from our team. Jen English.…
November 15, 2023. Oct/Nov 2023 Performance Issues. Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.…
April 6, 2026. April 2026 Common Crawl Newsletter. Check out our newsletter for April 2026, with updates on what we've been up to. Jen English.…
August 24, 2026. August 2026 Crawl Archive Now Available. We are pleased to announce that the crawl archive for August 2026 is now available, containing 2.14 billion web pages or 360 TiB of uncompressed content. Michael Paris.…
August 13, 2025. Common Crawl Foundation at ACL 2025. The Common Crawl team attended the 63rd Annual Meeting of the Association of Computational Linguistics in Vienna, presenting recent published work and strengthening links with the research community.…
April 1, 2025. March 2025 Crawl Archive Now Available. We are pleased to announce that the crawl archive for March 2025 is now available.…
November 1, 2021. October 2021 crawl archive now available. The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content.…
October 7, 2020. September 2020 crawl archive now available. The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content.…
February 3, 2020. January 2020 crawl archive now available. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th.…
June 28, 2024. June 2024 Crawl Archive Now Available. The crawl archive for June 2024 is now available. The data was crawled between June 12th and June 26th, and contains 2.7 billion web pages (or 382 TiB of uncompressed content).…
November 18, 2024. November 2024 Crawl Archive Now Available. The crawl archive for November 2024 is now available. The data was crawled between November 1st and November 15th, and contains 2.68 billion web pages (or 405 TiB of uncompressed content).…
May 25, 2026. May 2026 Crawl Archive Now Available. We are happy to announce the release of the May 2026 crawl archive, consisting of 2.16 billion web pages, or 365.56 TiB of uncompressed content. Michael Paris.…
December 20, 2025. December 2025 Crawl Archive Now Available. The crawl archive for December 2025 is now available, consisting of 2.16 billion web pages (or 364 TiB of uncompressed content). Thom Vaughan.…
June 3, 2024. May 2024 Crawl Archive Now Available. The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!…
February 2, 2021. January 2021 crawl archive now available. The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content.…
July 20, 2020. July 2020 crawl archive now available. The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th.…
June 28, 2021. June 2021 crawl archive now available. The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content.…
April 27, 2021. April 2021 crawl archive now available. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content.…
May 4, 2025. April 2025 Crawl Archive Now Available. Announcing the release of the April 2025 crawl archive. The data was crawled between April 17th and May 1st, and contains 2.74 billion web pages (or 468 TiB of uncompressed content).…
March 4, 2020. February 2020 crawl archive now available. The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th.…
June 28, 2024. Dialog and Discovery at AI_dev 2024. This month members from the Common Crawl Foundation attended the AI_dev: Open Source GenAI & ML Summit in Paris, where discussions focused on AI advancements, ethics, and Open Source solutions.…
April 28, 2026. April 2026 Crawl Archive Now Available. We are pleased to announce that the crawl archive for April 2026 is now available, containing 2.19 billion web pages or 379.2 TiB of uncompressed content. Luca Foppiano.…
July 28, 2026. July 2026 Crawl Archive Now Available. The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content).…
June 30, 2026. Common Crawl Foundation at LREC 2026.…
October 21, 2026. October 2026 Crawl Archive Now Available. We are pleased to announce the release of the October 2026 crawl archive, consisting of 2.16 billion pages (or 371.19 TiB of uncompressed content). Hande Çelikkanat.…
July 23, 2025. July 2025 Crawl Archive Now Available. The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content. Thom Vaughan.…
July 28, 2024. July 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for July 2024 is now available, containing 2.5 billion web pages, or 360 TiB of uncompressed content. Thom Vaughan.…
May 1, 2024. April 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for April 2024 is now available.…
September 24, 2024. September 2024 Crawl Archive Now Available. The crawl archive for September 2024 is now available.…
October 4, 2021. September 2021 crawl archive now available. The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content.…