Search results

February 3, 2020. January 2020 crawl archive now available. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th.…

October 7, 2020. September 2020 crawl archive now available. The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content.…

July 20, 2020. July 2020 crawl archive now available. The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th.…

March 4, 2020. February 2020 crawl archive now available. The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th.…

August 19, 2020. August 2020 crawl archive now available. The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th.…

November 7, 2020. October 2020 crawl archive now available. The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content.…

June 10, 2020. May/June 2020 crawl archive now available. The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th.…

December 10, 2020. November/December 2020 crawl archive now available. The crawl archive for November/December 2020 is now available!…

April 14, 2020. March/April 2020 crawl archive now available. The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th.…

February 10, 2021. Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021.…

October 16, 2020. Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020.…

June 16, 2020. Host- and Domain-Level Web Graphs Feb/Mar/May 2020. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March/April and May/June 2020.…

February 10, 2020. Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020.…

In 2020, Factual merged with Foursquare and today Gil is Co-Chairman of the board of a combined entity which generated $150m in combined revenue at the time of the merger.…

May 5, 2024. Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.…

February 1, 2025. Host- and Domain-Level Web Graphs November/December 2024 and January 2025. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2024 and January 2025.…

October 28, 2020. Interactive Webgraph Statistics Notebook Released. We are pleased to announce the release of an interactive Jupyter notebook that is used to provide visualization of webgraph statistics, and a way to interact with the webgraph. Alex Xue.…

February 25, 2025. Host- and Domain-Level Web Graphs December 2024 and January/February 2025.…

March 16, 2022. Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022. We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022.…

March 14, 2024. Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024.…

March 15, 2023. Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023.…

June 25, 2024. May/June 2024 Newsletter. We’re pleased to share our newsletter for May/June 2024, featuring the latest updates, events, and highlights from our community. Greg Lindahl. Greg is the Chief Technology Officer at the Common Crawl Foundation.…

March 26, 2024. March/April 2024 Newsletter. We're excited to share an update on some of our recent projects and initiatives in this newsletter! Common Crawl Foundation.…

February 3, 2025. January/February 2025 Newsletter. We’re happy to share our January/February 2025 newsletter with updates and insights from the world of open data and web archiving. Jen English.…

September 10, 2024. August/September 2024 Newsletter. We're pleased to announce our newsletter for August and September 2024. Jen English.…

November 25, 2024. October/November 2024 Newsletter. We’re pleased to announce this month's newsletter, featuring key updates, upcoming events, and community highlights. Jen English.…

April 14, 2025. March/April 2025 Newsletter. We're excited to share our newsletter for March/April 2025 with updates from our team. Jen English.…

November 15, 2023. Oct/Nov 2023 Performance Issues. Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.…

June 28, 2024. June 2024 Crawl Archive Now Available. The crawl archive for June 2024 is now available. The data was crawled between June 12th and June 26th, and contains 2.7 billion web pages (or 382 TiB of uncompressed content).…

November 18, 2024. November 2024 Crawl Archive Now Available. The crawl archive for November 2024 is now available. The data was crawled between November 1st and November 15th, and contains 2.68 billion web pages (or 405 TiB of uncompressed content).…

April 1, 2025. March 2025 Crawl Archive Now Available. We are pleased to announce that the crawl archive for March 2025 is now available.…

November 1, 2021. October 2021 crawl archive now available. The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content.…

April 27, 2021. April 2021 crawl archive now available. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content.…

June 28, 2021. June 2021 crawl archive now available. The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content.…

February 2, 2021. January 2021 crawl archive now available. The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content.…

June 3, 2024. May 2024 Crawl Archive Now Available. The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!…

June 28, 2024. Dialog and Discovery at AI_dev 2024. This month members from the Common Crawl Foundation attended the AI_dev: Open Source GenAI & ML Summit in Paris, where discussions focused on AI advancements, ethics, and Open Source solutions.…

May 4, 2025. April 2025 Crawl Archive Now Available. Announcing the release of the April 2025 crawl archive. The data was crawled between April 17th and May 1st, and contains 2.74 billion web pages (or 468 TiB of uncompressed content).…

October 4, 2021. September 2021 crawl archive now available. The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content.…

September 24, 2024. September 2024 Crawl Archive Now Available. The crawl archive for September 2024 is now available.…

May 1, 2024. April 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for April 2024 is now available.…

July 28, 2024. July 2024 Crawl Archive Now Available. We are pleased to announce that the crawl archive for July 2024 is now available, containing 2.5 billion web pages, or 360 TiB of uncompressed content. Thom Vaughan.…

June 2, 2022. May 2022 crawl archive now available. The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content.…

May 23, 2021. May 2021 crawl archive now available. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content.…

October 20, 2024. October 2024 Crawl Archive Now Available. The data was crawled between October 3rd and October 16th, and contains 2.49 billion web pages (or 365 TiB of uncompressed content).…

December 18, 2024. December 2024 Crawl Archive Now Available. The crawl archive for December 2024 is now available. The data was crawled between December 1st and December 15th, and contains 2.64 billion web pages (or 394 TiB of uncompressed content).…

January 31, 2025. January 2025 Crawl Archive Now Available. We're pleased to announce our first crawl of 2025, containing 3.0 billion pages, and 460 TiB uncompressed content. Thom Vaughan. Thom is Principal Technologist at the Common Crawl Foundation.…

August 18, 2024. August 2024 Crawl Archive Now Available. The crawl archive for August 2024 is now available. The data was crawled between August 3rd and August 16th, and contains 2.3 billion web pages (or 327.4 TiB of uncompressed content). Thom Vaughan.…

February 23, 2025. February 2025 Crawl Archive Now Available. The crawl archive for February 2025 is now available. The data was crawled between February 6th and February 20th, and contains 2.6 billion web pages (or 402 TiB of uncompressed content).…

February 2, 2022. January 2022 crawl archive now available. The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content.…

August 22, 2022. August 2022 crawl archive now available. The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content.…

A bad configuration was checked into our exclusion list on Sep 22, 2022 and was fixed on Oct 27, 2023. The configuration blocked a number of 2–level domains, meaning they were not included in certain crawls.…

December 15, 2023. November/December 2023 Crawl Archive Now Available. The crawl archive for November/December 2023 is now available.…

July 13, 2022. June/July 2022 crawl archive now available. The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content.…

October 11, 2022. September/October 2022 crawl archive now available. The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content.…

August 9, 2021. July/August 2021 crawl archive now available. The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content.…

December 14, 2022. November/December 2022 crawl archive now available. The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content.…

March 11, 2024. February/March 2024 Crawl Archive Now Available. The crawl archive for February/March 2024 is now available.…

April 16, 2025. IIPC General Assembly & Web Archiving Conference 2025. The Common Crawl team attended the 2025 IIPC General Assembly and Web Archiving Conference in Oslo, presenting recent work and participating in discussions on web preservation.…

December 14, 2021. November/December 2021 crawl archive now available. The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content.…