Common Crawl Blog

The latest news, interviews, technologies, and resources.

Filter by Category or Search by Title

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Host- and Domain-Level Web Graphs February/March, April, and May 2024

Host- and Domain-Level Web Graphs February/March, April, and May 2024

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, April, and May 2024.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
May 2024 Crawl Archive Now Available

May 2024 Crawl Archive Now Available

The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024

Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
April 2024 Crawl Archive Now Available

April 2024 Crawl Archive Now Available

We are pleased to announce that the crawl archive for April 2024 is now available. The data was crawled between April 12th and April 25th, and contains 2.7 billion web pages (or 386 TiB of uncompressed content). Page captures are from 47.24 million hosts or 37.65 million registered domains and include 0.98 billion new URLs not visited in any of our prior crawls.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
March/April 2024 Newsletter

March/April 2024 Newsletter

We're excited to share an update on some of our recent projects and initiatives in this newsletter!
Common Crawl Foundation
Common Crawl - Open Source Web Crawling data‍
Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024

Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of September, November, February 2023-24.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
February/March 2024 Crawl Archive Now Available

February/March 2024 Crawl Archive Now Available

The crawl archive for February/March 2024 is now available. The data was crawled between February 20th and March 5th, and contains 3.16 billion web pages (or 424.7 TiB of uncompressed content).
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
Web Archiving File Formats Explained

Web Archiving File Formats Explained

In the ever–evolving landscape of digital archiving and data analysis, it is helpful to understand the various file formats used for web crawling. From the early ARC format to the more advanced WARC, and the specialised WET and WAT files, each plays an important role in the field of web archiving. In this post, we explain these formats, exploring their unique features, applications, and the enhancements they offer.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
A Further Look Into the Prevalence of Various ML Opt–Out Protocols

A Further Look Into the Prevalence of Various ML Opt–Out Protocols

This post details some experiments that we have done regarding Machine Learning Opt–Out protocols. We decided to investigate the prevalence of some of these protocols, by taking a deeper look at our WARC files, and finding which proportions of domains are using which opt–out protocols.
Alex Xue
Alex is a Computer Science graduate from the University of Waterloo, Canada, and an emeritus member of the Common Crawl Foundation.
Balancing Discovery and Privacy: A Look Into Opt–Out Protocols

Balancing Discovery and Privacy: A Look Into Opt–Out Protocols

What opt–out protocols are, their importance, how you can use them, how we respect them, and what the emerging initiatives are that surround them.
Alex Xue
Alex is a Computer Science graduate from the University of Waterloo, Canada, and an emeritus member of the Common Crawl Foundation.
Host- and Domain-Level Web Graphs May/Sep/Nov 2023

Host- and Domain-Level Web Graphs May/Sep/Nov 2023

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, September, and November of 2023.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
November/December 2023 Crawl Archive Now Available

November/December 2023 Crawl Archive Now Available

The crawl archive for November/December 2023 is now available. The data was crawled between November 28th and December 12th, and contains 3.35 billion web pages (or 454 TiB of uncompressed content).
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
Oct/Nov 2023 Performance Issues

Oct/Nov 2023 Performance Issues

Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.
Greg Lindahl
Greg is the Chief Technology Officer at the Common Crawl Foundation.
Host- and Domain-Level Web Graphs Mar/May/Oct 2023

Host- and Domain-Level Web Graphs Mar/May/Oct 2023

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, May, and October 2023. The host-level graph consists of 378.7 million nodes and 2.6 billion edges, and the domain-level graph has 94.2 million nodes and 1.7 billion edges.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
September/October 2023 crawl archive now available

September/October 2023 crawl archive now available

The crawl archive for September/October 2023 is now available! The data was crawled Sept 21 – October 5 and contains 3.4 billion web pages or 456 TiB of uncompressed content.
Julien Nioche
Julien is a member of the Apache Software Foundation, Emeritus member of the Common Crawl Foundation, and is the creator of StormCrawler.
Bridging Digital Exploration and Scientific Frontiers

Bridging Digital Exploration and Scientific Frontiers

This month Common Crawl Foundation members had the privilege of attending 5th International Open Search Symposium at CERN in Geneva, Switzerland.
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.
May/June 2023 crawl archive now available

May/June 2023 crawl archive now available

The crawl archive for May/June 2023 is now available! The data was crawled May 27 – June 11 and contains 3.1 billion web pages or 390 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.0 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
March/April 2023 crawl archive now available

March/April 2023 crawl archive now available

The crawl archive for March/April 2023 is now available! The data was crawled March 20 – April 2 and contains 3.1 billion web pages or 400 TiB of uncompressed content. Page captures are from 43 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023

Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023

We are pleased to announce a new release of host-level and domain-level web graphs based on the September/October, November/December 2022 and January/February 2023 crawls. For more information about the data formats and the processing pipeline, please see the announcements of previous webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
January/February 2023 crawl archive now available

January/February 2023 crawl archive now available

The crawl archive for January/February 2023 is now available! The data was crawled January 26 – February 9 and contains 3.15 billion web pages or 400 TiB of uncompressed content. Page captures are from 40 million hosts or 33 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
September/October 2022 crawl archive now available

September/October 2022 crawl archive now available

The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls. This crawl includes improvements made in extracting clean text in WET files and WAT anchor texts.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs February/March, April and May 2021

Host- and Domain-Level Web Graphs February/March, April and May 2021

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February/March, April and May 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
November/December 2022 crawl archive now available

November/December 2022 crawl archive now available

The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
June/July 2022 crawl archive now available

June/July 2022 crawl archive now available

The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs May, June/July and August 2022

Host- and Domain-Level Web Graphs May, June/July and August 2022

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
August 2022 crawl archive now available

August 2022 crawl archive now available

The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content. Page captures are from 46 million hosts or 37 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
July/August 2021 crawl archive now available

July/August 2021 crawl archive now available

The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
May 2022 crawl archive now available

May 2022 crawl archive now available

The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content. Page captures are from 45 million hosts or 36 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022

Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
January 2022 crawl archive now available

January 2022 crawl archive now available

The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data

Introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data

Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
November/December 2021 crawl archive now available

November/December 2021 crawl archive now available

The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
October 2021 crawl archive now available

October 2021 crawl archive now available

The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs June, July/August and September 2021

Host- and Domain-Level Web Graphs June, July/August and September 2021

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July/August and September 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
September 2021 crawl archive now available

September 2021 crawl archive now available

The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
April 2021 crawl archive now available

April 2021 crawl archive now available

The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
June 2021 crawl archive now available

June 2021 crawl archive now available

The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
May 2021 crawl archive now available

May 2021 crawl archive now available

The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.28 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
February/March 2021 crawl archive now available

February/March 2021 crawl archive now available

The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021

Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2020 and January 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020

Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
November/December 2020 crawl archive now available

November/December 2020 crawl archive now available

The crawl archive for November/December 2020 is now available! The data was crawled between November 23 and December 6 and contains 2.64 billion web pages or 270 TiB of uncompressed content. It includes page captures of 1.4 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
January 2021 crawl archive now available

January 2021 crawl archive now available

The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content. It includes page captures of 1.15 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
October 2020 crawl archive now available

October 2020 crawl archive now available

The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Interactive Webgraph Statistics Notebook Released

Interactive Webgraph Statistics Notebook Released

We are pleased to announce the release of an interactive Jupyter notebook that is used to provide visualization of webgraph statistics, and a way to interact with the webgraph.
Alex Xue
Alex is a Computer Science graduate from the University of Waterloo, Canada, and an emeritus member of the Common Crawl Foundation.
September 2020 crawl archive now available

September 2020 crawl archive now available

The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
August 2020 crawl archive now available

August 2020 crawl archive now available

The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th. It includes page captures of 940 million URLs unknown in any of our prior crawl archives.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
July 2020 crawl archive now available

July 2020 crawl archive now available

The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th. It includes page captures of 1.1 billion URLs unknown in any of our prior crawl archives.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Feb/Mar/May 2020

Host- and Domain-Level Web Graphs Feb/Mar/May 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March/April and May/June 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
May/June 2020 crawl archive now available

May/June 2020 crawl archive now available

The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th. It includes page captures of 1.2 billion URLs unknown in any of our prior crawl archives.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
February 2020 crawl archive now available

February 2020 crawl archive now available

The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
March/April 2020 crawl archive now available

March/April 2020 crawl archive now available

The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
January 2020 crawl archive now available

January 2020 crawl archive now available

The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. It includes page captures of 960 million URLs not contained in any crawl archive before.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
December 2019 crawl archive now available

December 2019 crawl archive now available

The crawl archive for December 2019 is now available! It contains 2.45 billion web pages or 234 TiB of uncompressed content, crawled between December 5th and 16th. It includes page captures of 850 million URLs not contained in any crawl archive before.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs May/June/July 2019

Host- and Domain-Level Web Graphs May/June/July 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
November 2019 crawl archive now available

November 2019 crawl archive now available

The crawl archive for November 2019 is now available! It contains 2.55 billion web pages or 250 TiB of uncompressed content, crawled between November 11th and 23rd with a short operational break on Nov 16th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
October 2019 crawl archive now available

October 2019 crawl archive now available

The crawl archive for October 2019 is now available! It contains 3.0 billion web pages or 280 TiB of uncompressed content, crawled between October 13th and 24th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
September 2019 crawl archive now available

September 2019 crawl archive now available

The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th. It includes page captures of 1.0 billion URLs not contained in any crawl archive before. The other 1.5 billion pages have been already captured in prior crawls and are now revisited.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
August 2019 crawl archive now available

August 2019 crawl archive now available

The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
July 2019 crawl archive now available

July 2019 crawl archive now available

The crawl archive for July 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between July 15th and 24th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
May 2019 crawl archive now available

May 2019 crawl archive now available

The crawl archive for May 2019 is now available! It contains 2.65 billion web pages or 220 TiB of uncompressed content, crawled between May 19th and 27th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
June 2019 crawl archive now available

June 2019 crawl archive now available

The crawl archive for June 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between June 16th and 27th with an operational break from 21st to 24th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019

Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
April 2019 crawl archive now available

April 2019 crawl archive now available

The crawl archive for April 2019 is now available! It contains 2.5 billion web pages or 198 TiB of uncompressed content, crawled between April 18th and 26th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
February 2019 crawl archive now available

February 2019 crawl archive now available

The crawl archive for February 2019 is now available! It contains 2.9 billion web pages or 225 TiB of uncompressed content, crawled between February 15th and 24th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
March 2019 crawl archive now available

March 2019 crawl archive now available

The crawl archive for March 2019 is now available! It contains 2.55 billion web pages or 210 TiB of uncompressed content, crawled between March 18th and 27th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Nov/Dec/Jan 2018 - 2019

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2018 - 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of November, December 2018 and January 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
October 2018 crawl archive now available

October 2018 crawl archive now available

The crawl archive for October 2018 is now available! It contains 3.0 billion web pages and 240 TiB of uncompressed content, crawled between October 15th and 24th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
January 2019 crawl archive now available

January 2019 crawl archive now available

The crawl archive for January 2019 is now available! It contains 2.85 billion web pages or 240 TiB of uncompressed content, crawled between January 15th and 24th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
November 2018 crawl archive now available

November 2018 crawl archive now available

The crawl archive for November 2018 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between November 12th and 22nd.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
December 2018 crawl archive now available

December 2018 crawl archive now available

The crawl archive for December 2018 is now available! It contains 3.1 billion web pages or 250 TiB of uncompressed content, crawled between December 9th and 19th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Aug/Sep/Oct 2018

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2018

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Common Crawl Foundation
Common Crawl - Open Source Web Crawling data‍
June 2018 Crawl Archive Now Available

June 2018 Crawl Archive Now Available

The crawl archive for June 2018 is now available! The archive contains 3.05 billion web pages and 235 TiB of uncompressed content, crawled between June 18th and 25th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
September 2018 crawl archive now available

September 2018 crawl archive now available

The crawl archive for September 2018 is now available! It contains 2.8 billion web pages and 220 TiB of uncompressed content, crawled between September 17th and 26th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
August Crawl Archive Introduces Language Annotations

August Crawl Archive Introduces Language Annotations

The crawl archive for August 2018 is now available! It contains 2.65 billion web pages and 220 TiB of uncompressed content, crawled between August 14th and 22th. Together with an upgrade of the crawler software we've plugged in a language detector and now provide as annotation the language a web page is written in.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
3.25 Billion Pages Crawled in July 2018

3.25 Billion Pages Crawled in July 2018

The crawl archive for July 2018 is now available! The archive contains 3.25 billion web pages and 255 TiB of uncompressed content, crawled between July 15th and 23th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs May/June/July 2018

Host- and Domain-Level Web Graphs May/June/July 2018

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Common Crawl Foundation
Common Crawl - Open Source Web Crawling data‍
May 2018 Crawl Archive Now Available

May 2018 Crawl Archive Now Available

The crawl archive for May 2018 is now available! The archive contains 2.75 billion web pages and 215 TiB of uncompressed content, crawled between May 20th and 28th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Feb/Mar/Apr 2018

Host- and Domain-Level Web Graphs Feb/Mar/Apr 2018

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
April 2018 Crawl Archive Now Available

April 2018 Crawl Archive Now Available

The crawl archive for April 2018 is now available! The archive contains 3.1 billion web pages and 230 TiB of uncompressed content, crawled between April 19th and 27th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Index to WARC Files and URLs in Columnar Format

Index to WARC Files and URLs in Columnar Format

We're happy to announce the release of an index to WARC files and URLs in a columnar format. The columnar format (we use Apache Parquet) allows to efficiently query or process the index and saves time and computing resources. Especially, if only few columns are accessed, recent big data tools will run impressively fast.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
February 2018 Crawl Archive Now Available

February 2018 Crawl Archive Now Available

The crawl archive for February 2018 is now available! The archive contains 3.4 billion web pages and 270+ TiB of uncompressed content, crawled between February 17th and Feb 26th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
March 2018 Crawl Archive Now Available

March 2018 Crawl Archive Now Available

The crawl archive for March 2018 is now available! The archive contains 3.2 billion web pages and 250+ TiB of uncompressed content, crawled between March 17th and 25th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Nov/Dec/Jan 2017-2018

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2017-2018

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of November, December 2017 and January 2018. These graphs, along with ranked lists of hosts and domains, follow the prior web graph releases (Feb/Mar/Apr 2017, May/Jun/Jul 2017 and Aug/Sep/Oct 2017).
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
January 2018 Crawl Archive Now Available

January 2018 Crawl Archive Now Available

The crawl archive for January 2018 is now available! The archive contains 3.4 billion web pages and 270 TiB of uncompressed content, crawled between January 16th and Jan 24th.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
December 2017 Crawl Archive Now Available

December 2017 Crawl Archive Now Available

The crawl archive for December 2017 is now available! The archive contains 2.9 billion web pages and over 240 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
November 2017 Crawl Archive Now Available

November 2017 Crawl Archive Now Available

The crawl archive for November 2017 is now available! The archive contains 3.2 billion web pages and 260 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Host- and Domain-Level Web Graphs Aug/Sept/Oct 2017

Host- and Domain-Level Web Graphs Aug/Sept/Oct 2017

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September, and October 2017. These graphs, along with ranked lists of hosts and domains, follow the first (February, March, April 2017) and second (May, June, July 2017) web graph releases.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
October 2017 Crawl Archive Now Available

October 2017 Crawl Archive Now Available

The crawl archive for October 2017 is now available! The archive contains 3.65 billion web pages and over 300 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
September 2017 Crawl Archive Now Available

September 2017 Crawl Archive Now Available

The crawl archive for September 2017 is now available! The archive contains 3.01 billion web pages and over 250 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
August 2017 Crawl Archive Now Available

August 2017 Crawl Archive Now Available

The crawl archive for August 2017 is now available! The archive contains 3.28 billion+ web pages and over 280 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
June 2017 Crawl Archive Now Available

June 2017 Crawl Archive Now Available

The crawl archive for June 2017 is now available! The archive contains 3.16 billion+ web pages and over 260 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Now Available: Host- and Domain-Level Web Graphs

Now Available: Host- and Domain-Level Web Graphs

We are pleased to announce the release of host-level and domain-level web graphs based on the published crawls of May, June, and July 2017. These graphs, along with ranked lists of hosts and domains, follow on our first host-level web graph (February, March, April 2017).
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
July 2017 Crawl Archive Now Available

July 2017 Crawl Archive Now Available

The crawl archive for July 2017 is now available! The archive contains 2.89 billion+ web pages and over 240 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
May 2017 Crawl Archive Now Available

May 2017 Crawl Archive Now Available

The crawl archive for May 2017 is now available! The archive contains 2.96 billion+ web pages and over 250 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
Common Crawl's First In-House Web Graph

Common Crawl's First In-House Web Graph

We are pleased to announce the release of a host-level web graph of recent monthly crawls (February, March, April 2017). The graph consists of 385 million nodes and 2.5 billion edges.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
April 2017 Crawl Archive Now Available

April 2017 Crawl Archive Now Available

The crawl archive for April 2017 is now available! The archive contains 2.94 billion+ web pages and over 250 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.
March 2017 Crawl Archive Now Available

March 2017 Crawl Archive Now Available

The crawl archive for March 2017 is now available! The archive contains 3.07 billion+ web pages and over 250 TiB of uncompressed content.
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.

Common Crawl Blog

Host- and Domain-Level Web Graphs February/March, April, and May 2024

June 4, 2024

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, April, and May 2024.

Read More...

May 2024 Crawl Archive Now Available

June 3, 2024

The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!

Read More...

Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024

May 5, 2024

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.

Read More...

April 2024 Crawl Archive Now Available

May 1, 2024

We are pleased to announce that the crawl archive for April 2024 is now available. The data was crawled between April 12th and April 25th, and contains 2.7 billion web pages (or 386 TiB of uncompressed content). Page captures are from 47.24 million hosts or 37.65 million registered domains and include 0.98 billion new URLs not visited in any of our prior crawls.

Read More...

March/April 2024 Newsletter

March 26, 2024

We're excited to share an update on some of our recent projects and initiatives in this newsletter!

Read More...

Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024

March 14, 2024

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of September, November, February 2023-24.

Read More...

February/March 2024 Crawl Archive Now Available

March 11, 2024

The crawl archive for February/March 2024 is now available. The data was crawled between February 20th and March 5th, and contains 3.16 billion web pages (or 424.7 TiB of uncompressed content).

Read More...

Web Archiving File Formats Explained

March 1, 2024

In the ever–evolving landscape of digital archiving and data analysis, it is helpful to understand the various file formats used for web crawling. From the early ARC format to the more advanced WARC, and the specialised WET and WAT files, each plays an important role in the field of web archiving. In this post, we explain these formats, exploring their unique features, applications, and the enhancements they offer.

Read More...

A Further Look Into the Prevalence of Various ML Opt–Out Protocols

February 22, 2024

This post details some experiments that we have done regarding Machine Learning Opt–Out protocols. We decided to investigate the prevalence of some of these protocols, by taking a deeper look at our WARC files, and finding which proportions of domains are using which opt–out protocols.

Read More...

Balancing Discovery and Privacy: A Look Into Opt–Out Protocols

February 13, 2024

What opt–out protocols are, their importance, how you can use them, how we respect them, and what the emerging initiatives are that surround them.

Read More...

Host- and Domain-Level Web Graphs May/Sep/Nov 2023

December 22, 2023

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, September, and November of 2023.

Read More...

November/December 2023 Crawl Archive Now Available

December 15, 2023

The crawl archive for November/December 2023 is now available. The data was crawled between November 28th and December 12th, and contains 3.35 billion web pages (or 454 TiB of uncompressed content).

Read More...

Oct/Nov 2023 Performance Issues

November 15, 2023

Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.

Read More...

Host- and Domain-Level Web Graphs Mar/May/Oct 2023

October 18, 2023

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, May, and October 2023. The host-level graph consists of 378.7 million nodes and 2.6 billion edges, and the domain-level graph has 94.2 million nodes and 1.7 billion edges.

Read More...

September/October 2023 crawl archive now available

October 12, 2023

The crawl archive for September/October 2023 is now available! The data was crawled Sept 21 – October 5 and contains 3.4 billion web pages or 456 TiB of uncompressed content.

Read More...

Bridging Digital Exploration and Scientific Frontiers

October 10, 2023

This month Common Crawl Foundation members had the privilege of attending 5th International Open Search Symposium at CERN in Geneva, Switzerland.

Read More...

May/June 2023 crawl archive now available

June 21, 2023

The crawl archive for May/June 2023 is now available! The data was crawled May 27 – June 11 and contains 3.1 billion web pages or 390 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.0 billion new URLs, not visited in any of our prior crawls.

Read More...

March/April 2023 crawl archive now available

April 6, 2023

The crawl archive for March/April 2023 is now available! The data was crawled March 20 – April 2 and contains 3.1 billion web pages or 400 TiB of uncompressed content. Page captures are from 43 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.

Read More...

Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023

March 15, 2023

We are pleased to announce a new release of host-level and domain-level web graphs based on the September/October, November/December 2022 and January/February 2023 crawls. For more information about the data formats and the processing pipeline, please see the announcements of previous webgraph releases.

Read More...

January/February 2023 crawl archive now available

February 16, 2023

The crawl archive for January/February 2023 is now available! The data was crawled January 26 – February 9 and contains 3.15 billion web pages or 400 TiB of uncompressed content. Page captures are from 40 million hosts or 33 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.

Read More...

November/December 2022 crawl archive now available

December 14, 2022

The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.

Read More...

September/October 2022 crawl archive now available

October 11, 2022

The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls. This crawl includes improvements made in extracting clean text in WET files and WAT anchor texts.

Read More...

Host- and Domain-Level Web Graphs May, June/July and August 2022

September 23, 2022

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.

Read More...

August 2022 crawl archive now available

August 22, 2022

The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content. Page captures are from 46 million hosts or 37 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.

Read More...

June/July 2022 crawl archive now available

July 13, 2022

The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.

Read More...