Search results

August 22, 2022. August 2022 crawl archive now available. The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content.…

February 2, 2022. January 2022 crawl archive now available. The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content.…

June 2, 2022. May 2022 crawl archive now available. The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content.…

December 14, 2022. November/December 2022 crawl archive now available. The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content.…

October 11, 2022. September/October 2022 crawl archive now available. The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content.…

July 13, 2022. June/July 2022 crawl archive now available. The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content.…

March 16, 2022. Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022.…

March 15, 2023. Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023.…

September 23, 2022. Host- and Domain-Level Web Graphs May, June/July and August 2022. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022.…

A bad configuration was checked into our exclusion list on Sep 22, 2022 and was fixed on Oct 27, 2023. The configuration blocked a number of 2–level domains, meaning they were not included in certain crawls.…

May, June/July, August 2022 Web Graph (cc-main-2022-may-jun-aug-domain). and the following Web Graph releases. Affected Web Graphs. The Data. Overview. CDXJ Index. Columnar Index. Web Graphs. Latest Crawl. Crawl Stats. Graph Stats. Errata. Resources.…

February 11, 2026. CC-Citations: A Visualization of Research Papers Referencing Common Crawl. We are proud to release an interactive visualization of thousands of research papers using or citing Common Crawl data. Malte Ostendorff.…

May 5, 2024. Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.…

March 1, 2022. Introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data. Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone.…

February 25, 2025. Host- and Domain-Level Web Graphs December 2024 and January/February 2025.…

February 24, 2026. Host- and Domain-Level Web Graphs December 2025 and January/February 2026.…

February 1, 2025. Host- and Domain-Level Web Graphs November/December 2024 and January 2025. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2024 and January 2025.…

January 30, 2026. Host- and Domain-Level Web Graphs November/December 2025 and January 2026.…

February 10, 2021. Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021.…

March 14, 2024. Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024.…

A. 2024 Mozilla Foundation study. described the dataset as having "laid the infrastructural foundation" for the generative AI boom, and found that at least 64% of 47 large language models published between 2019 and 2023 were trained on filtered versions of…

November 25, 2024. October/November 2024 Newsletter. We’re pleased to announce this month's newsletter, featuring key updates, upcoming events, and community highlights. Jen English.…

April 14, 2025. March/April 2025 Newsletter. We're excited to share our newsletter for March/April 2025 with updates from our team. Jen English.…

June 24, 2025. May/June 2025 Newsletter. We're happy to share our newsletter for May/June 2025 with updates from our team. Jen English.…

September 10, 2024. August/September 2024 Newsletter. We're pleased to announce our newsletter for August and September 2024. Jen English.…

November 3, 2025. October/November 2025 Newsletter. Check out our newsletter for October/November 2025, with updates on what we've been up to. Common Crawl Foundation.…

June 25, 2024. May/June 2024 Newsletter. We’re pleased to share our newsletter for May/June 2024, featuring the latest updates, events, and highlights from our community. Greg Lindahl. Greg is Chief Technology Officer at the Common Crawl Foundation.…

August 26, 2025. July/August 2025 Newsletter. We are pleased to release our newsletter for July and August 2025, with updates on our team's activities. Jen English.…

March 26, 2024. March/April 2024 Newsletter. We're excited to share an update on some of our recent projects and initiatives in this newsletter! Common Crawl Foundation.…

February 3, 2025. January/February 2025 Newsletter. We’re happy to share our January/February 2025 newsletter with updates and insights from the world of open data and web archiving. Jen English.…

July 16, 2024. The Environmental Impact of the Cloud - the Common Crawl Case Study. Looking at tools (Green Software) and methodologies to evaluate the environmental impact of the cloud (a nascent activity coined GreenOps). Julien Nioche.…

November 15, 2023. Oct/Nov 2023 Performance Issues. Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.…

April 6, 2026. April 2026 Common Crawl Newsletter. Check out our newsletter for April 2026, with updates on what we've been up to. Jen English.…

December 18, 2024. December 2024 Crawl Archive Now Available. The crawl archive for December 2024 is now available. The data was crawled between December 1st and December 15th, and contains 2.64 billion web pages (or 394 TiB of uncompressed content).…

August 18, 2025. August 2025 Crawl Archive Now Available. We are pleased to announce the release of our August 2025 crawl, containing 2.44 billion web pages (or 424 TiB of uncompressed content). Thom Vaughan.…

October 20, 2024. October 2024 Crawl Archive Now Available. The data was crawled between October 3rd and October 16th, and contains 2.49 billion web pages (or 365 TiB of uncompressed content).…

October 23, 2025. October 2025 Crawl Archive Now Available. We are pleased to announce the release of the October 2025 crawl, containing 2.61 billion web pages or 468 TiB of uncompressed content. Hande Çelikkanat.…

March 19, 2026. March 2026 Crawl Archive Now Available. We are pleased to announce the release of the March 2026 crawl, containing 1.97 billion web pages, or 344.64 TiB of uncompressed content.…

May 23, 2021. May 2021 crawl archive now available. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content.…

January 31, 2025. January 2025 Crawl Archive Now Available. We're pleased to announce our first crawl of 2025, containing 3.0 billion pages, and 460 TiB uncompressed content. Thom Vaughan. Thom is Principal Engineer at the Common Crawl Foundation.…

June 28, 2021. June 2021 crawl archive now available. The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content.…

June 3, 2024. May 2024 Crawl Archive Now Available. The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!…

April 27, 2021. April 2021 crawl archive now available. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content.…

March 4, 2020. February 2020 crawl archive now available. The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th.…

May 4, 2025. April 2025 Crawl Archive Now Available. Announcing the release of the April 2025 crawl archive. The data was crawled between April 17th and May 1st, and contains 2.74 billion web pages (or 468 TiB of uncompressed content).…

February 2, 2021. January 2021 crawl archive now available. The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content.…

July 20, 2020. July 2020 crawl archive now available. The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th.…

December 20, 2025. December 2025 Crawl Archive Now Available. The crawl archive for December 2025 is now available, consisting of 2.16 billion web pages (or 364 TiB of uncompressed content). Thom Vaughan.…

June 28, 2024. Dialog and Discovery at AI_dev 2024. This month members from the Common Crawl Foundation attended the AI_dev: Open Source GenAI & ML Summit in Paris, where discussions focused on AI advancements, ethics, and Open Source solutions.…

April 28, 2026. April 2026 Crawl Archive Now Available. We are pleased to announce that the crawl archive for April 2026 is now available, containing 2.19 billion web pages or 379.2 TiB of uncompressed content. Luca Foppiano.…

May 1, 2024. April 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for April 2024 is now available.…

July 23, 2025. July 2025 Crawl Archive Now Available. The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content. Thom Vaughan.…

September 22, 2025. September 2025 Crawl Archive Now Available. We are pleased to announce the release of our September 2025 crawl, containing 2.39 billion web pages, or 421 TiB of uncompressed content. Thom Vaughan.…

July 28, 2024. July 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for July 2024 is now available, containing 2.5 billion web pages, or 360 TiB of uncompressed content. Thom Vaughan.…

October 4, 2021. September 2021 crawl archive now available. The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content.…

September 24, 2024. September 2024 Crawl Archive Now Available. The crawl archive for September 2024 is now available.…

November 1, 2021. October 2021 crawl archive now available. The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content.…

February 3, 2020. January 2020 crawl archive now available. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th.…

June 28, 2024. June 2024 Crawl Archive Now Available. The crawl archive for June 2024 is now available. The data was crawled between June 12th and June 26th, and contains 2.7 billion web pages (or 382 TiB of uncompressed content).…

April 1, 2025. March 2025 Crawl Archive Now Available. We are pleased to announce that the crawl archive for March 2025 is now available.…