Search results

October 4, 2021. September 2021 crawl archive now available. The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content.…

May 23, 2021. May 2021 crawl archive now available. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content.…

April 27, 2021. April 2021 crawl archive now available. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content.…

June 28, 2021. June 2021 crawl archive now available. The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content.…

February 2, 2021. January 2021 crawl archive now available. The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content.…

November 1, 2021. October 2021 crawl archive now available. The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content.…

March 14, 2021. February/March 2021 crawl archive now available. The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content.…

December 14, 2021. November/December 2021 crawl archive now available. The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content.…

August 9, 2021. July/August 2021 crawl archive now available. The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content.…

March 16, 2022. Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022.…

February 10, 2021. Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021.…

October 8, 2021. Host- and Domain-Level Web Graphs June, July/August and September 2021. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July/August and September 2021.…

May 31, 2021. Host- and Domain-Level Web Graphs February/March, April and May 2021. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February/March, April and May 2021.…

May 5, 2024. Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.…

February 1, 2025. Host- and Domain-Level Web Graphs November/December 2024 and January 2025. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2024 and January 2025.…

February 25, 2025. Host- and Domain-Level Web Graphs December 2024 and January/February 2025.…

March 14, 2024. Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024.…

March 15, 2023. Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023.…

June 25, 2024. May/June 2024 Newsletter. We’re pleased to share our newsletter for May/June 2024, featuring the latest updates, events, and highlights from our community. Greg Lindahl. Greg is the Chief Technology Officer at the Common Crawl Foundation.…

March 26, 2024. March/April 2024 Newsletter. We're excited to share an update on some of our recent projects and initiatives in this newsletter! Common Crawl Foundation.…

February 3, 2025. January/February 2025 Newsletter. We’re happy to share our January/February 2025 newsletter with updates and insights from the world of open data and web archiving. Jen English.…

September 10, 2024. August/September 2024 Newsletter. We're pleased to announce our newsletter for August and September 2024. Jen English.…

November 25, 2024. October/November 2024 Newsletter. We’re pleased to announce this month's newsletter, featuring key updates, upcoming events, and community highlights. Jen English.…

April 14, 2025. March/April 2025 Newsletter. We're excited to share our newsletter for March/April 2025 with updates from our team. Jen English.…

November 15, 2023. Oct/Nov 2023 Performance Issues. Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.…

October 7, 2020. September 2020 crawl archive now available. The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content.…

February 3, 2020. January 2020 crawl archive now available. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th.…

April 1, 2025. March 2025 Crawl Archive Now Available. We are pleased to announce that the crawl archive for March 2025 is now available.…

June 28, 2024. June 2024 Crawl Archive Now Available. The crawl archive for June 2024 is now available. The data was crawled between June 12th and June 26th, and contains 2.7 billion web pages (or 382 TiB of uncompressed content).…

November 18, 2024. November 2024 Crawl Archive Now Available. The crawl archive for November 2024 is now available. The data was crawled between November 1st and November 15th, and contains 2.68 billion web pages (or 405 TiB of uncompressed content).…

June 3, 2024. May 2024 Crawl Archive Now Available. The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!…

June 28, 2024. Dialog and Discovery at AI_dev 2024. This month members from the Common Crawl Foundation attended the AI_dev: Open Source GenAI & ML Summit in Paris, where discussions focused on AI advancements, ethics, and Open Source solutions.…

May 4, 2025. April 2025 Crawl Archive Now Available. Announcing the release of the April 2025 crawl archive. The data was crawled between April 17th and May 1st, and contains 2.74 billion web pages (or 468 TiB of uncompressed content).…

July 20, 2020. July 2020 crawl archive now available. The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th.…

March 4, 2020. February 2020 crawl archive now available. The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th.…

February 23, 2025. February 2025 Crawl Archive Now Available. The crawl archive for February 2025 is now available. The data was crawled between February 6th and February 20th, and contains 2.6 billion web pages (or 402 TiB of uncompressed content).…

August 18, 2024. August 2024 Crawl Archive Now Available. The crawl archive for August 2024 is now available. The data was crawled between August 3rd and August 16th, and contains 2.3 billion web pages (or 327.4 TiB of uncompressed content). Thom Vaughan.…

August 19, 2020. August 2020 crawl archive now available. The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th.…

November 7, 2020. October 2020 crawl archive now available. The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content.…

February 2, 2022. January 2022 crawl archive now available. The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content.…

August 22, 2022. August 2022 crawl archive now available. The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content.…

May 1, 2024. April 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for April 2024 is now available.…

July 28, 2024. July 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for July 2024 is now available, containing 2.5 billion web pages, or 360 TiB of uncompressed content. Thom Vaughan.…

September 24, 2024. September 2024 Crawl Archive Now Available. The crawl archive for September 2024 is now available.…

December 18, 2024. December 2024 Crawl Archive Now Available. The crawl archive for December 2024 is now available. The data was crawled between December 1st and December 15th, and contains 2.64 billion web pages (or 394 TiB of uncompressed content).…

January 31, 2025. January 2025 Crawl Archive Now Available. We're pleased to announce our first crawl of 2025, containing 3.0 billion pages, and 460 TiB uncompressed content. Thom Vaughan. Thom is Principal Technologist at the Common Crawl Foundation.…

October 20, 2024. October 2024 Crawl Archive Now Available. The data was crawled between October 3rd and October 16th, and contains 2.49 billion web pages (or 365 TiB of uncompressed content).…

June 2, 2022. May 2022 crawl archive now available. The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content.…

A bad configuration was checked into our exclusion list on Sep 22, 2022 and was fixed on Oct 27, 2023. The configuration blocked a number of 2–level domains, meaning they were not included in certain crawls.…

December 10, 2020. November/December 2020 crawl archive now available. The crawl archive for November/December 2020 is now available!…

July 13, 2022. June/July 2022 crawl archive now available. The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content.…

June 10, 2020. May/June 2020 crawl archive now available. The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th.…

April 14, 2020. March/April 2020 crawl archive now available. The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th.…

December 15, 2023. November/December 2023 Crawl Archive Now Available. The crawl archive for November/December 2023 is now available.…

October 11, 2022. September/October 2022 crawl archive now available. The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content.…

December 14, 2022. November/December 2022 crawl archive now available. The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content.…

June 21, 2023. May/June 2023 crawl archive now available. The crawl archive for May/June 2023 is now available! The data was crawled May 27 – June 11 and contains 3.1 billion web pages or 390 TiB of uncompressed content.…

April 6, 2023. March/April 2023 crawl archive now available. The crawl archive for March/April 2023 is now available! The data was crawled March 20 – April 2 and contains 3.1 billion web pages or 400 TiB of uncompressed content.…

October 12, 2023. September/October 2023 crawl archive now available. The crawl archive for September/October 2023 is now available! The data was crawled Sept 21 – October 5 and contains 3.4 billion web pages or 456 TiB of uncompressed content.…

March 11, 2024. February/March 2024 Crawl Archive Now Available. The crawl archive for February/March 2024 is now available.…