May 2022 crawl archive now available

The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content. Page captures are from 45 million hosts or 36 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-21/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively, please see accessing the data for detailed instructions.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2022-21/segment.paths.gz100
WARC filesCC-MAIN-2022-21/warc.paths.gz8000092.81
WAT filesCC-MAIN-2022-21/wat.paths.gz8000023.30
WET filesCC-MAIN-2022-21/wet.paths.gz800009.80
Robots.txt filesCC-MAIN-2022-21/robotstxt.paths.gz800000.15
Non-200 responses filesCC-MAIN-2022-21/non200responses.paths.gz800002.26
URL index filesCC-MAIN-2022-21/cc-index.paths.gz3020.25
Columnar URL index filesCC-MAIN-2022-21/cc-index-table.paths.gz9000.28

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-21/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

November/December 2021 crawl archive now available

The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.

Improvements

The column url_host_name_reversed was added to the columnar index. It holds the host name in reverse domain name notation (com.example.www) which is more efficient to query. In order to make use of the new column please use the updated table schema. On Amazon Athena you need to recreate table by running the latest table creation statement. Further details are found in the corresponding pull request.

The columnar index is now built using Spark version 3.2.0 and Parquet MR 1.12.1 – these upgrades allow us to go for further improvements next year. Let us know if you observe any incompatibilities using this index to query data from the Nov/Dec 2021 crawl.

The crawler is now based on Tika 2.1.0 for detection of MIME/content types and character sets used to encode HTML documents.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-49/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-49/segment.paths.gz100
WARC filesCC-MAIN-2021-49/warc.paths.gz6400068.66
WAT filesCC-MAIN-2021-49/wat.paths.gz6400016.66
WET filesCC-MAIN-2021-49/wet.paths.gz640007.18
Robots.txt filesCC-MAIN-2021-49/robotstxt.paths.gz640000.15
Non-200 responses filesCC-MAIN-2021-49/non200responses.paths.gz640002.29
URL index filesCC-MAIN-2021-49/cc-index.paths.gz3020.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-49/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

October 2021 crawl archive now available

The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-43/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-43/segment.paths.gz100
WARC filesCC-MAIN-2021-43/warc.paths.gz7200085.11
WAT filesCC-MAIN-2021-43/wat.paths.gz7200022.42
WET filesCC-MAIN-2021-43/wet.paths.gz720009.79
Robots.txt filesCC-MAIN-2021-43/robotstxt.paths.gz720000.15
Non-200 responses filesCC-MAIN-2021-43/non200responses.paths.gz720002.11
URL index filesCC-MAIN-2021-43/cc-index.paths.gz3020.25

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-43/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

September 2021 crawl archive now available

The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-39/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-39/segment.paths.gz100
WARC filesCC-MAIN-2021-39/warc.paths.gz7200072.16
WAT filesCC-MAIN-2021-39/wat.paths.gz7200019.81
WET filesCC-MAIN-2021-39/wet.paths.gz720008.7
Robots.txt filesCC-MAIN-2021-39/robotstxt.paths.gz720000.15
Non-200 responses filesCC-MAIN-2021-39/non200responses.paths.gz720002.19
URL index filesCC-MAIN-2021-39/cc-index.paths.gz3020.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-39/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

July/August 2021 crawl archive now available

The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls.

Improvements

Archiving of robots.txt files was improved. Robots.txt files are not archived if

  • the robots.txt of the target host does not allow it (in case of a HTTP redirect) or
  • URL filters exclude the entire site, eg. if it’s known ahead that a site does not allow crawling or
  • the MIME type is not applicable for robots.txt files (eg. HTML, PDF)

More details are found in the corresponding issue report. The change reduces the size of the robots.txt subset (since August 2016) by removing content which should not contained in this dataset.

Archive Location and Download

The July/August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-31/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-31/segment.paths.gz100
WARC filesCC-MAIN-2021-31/warc.paths.gz7200075.34
WAT filesCC-MAIN-2021-31/wat.paths.gz7200021.67
WET filesCC-MAIN-2021-31/wet.paths.gz720009.43
Robots.txt filesCC-MAIN-2021-31/robotstxt.paths.gz720000.14
Non-200 responses filesCC-MAIN-2021-31/non200responses.paths.gz720001.98
URL index filesCC-MAIN-2021-31/cc-index.paths.gz3020.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-31/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

June 2021 crawl archive now available

The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-25/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-25/segment.paths.gz100
WARC filesCC-MAIN-2021-25/warc.paths.gz6400057.9
WAT filesCC-MAIN-2021-25/wat.paths.gz6400016.54
WET filesCC-MAIN-2021-25/wet.paths.gz640007.24
Robots.txt filesCC-MAIN-2021-25/robotstxt.paths.gz640000.16
Non-200 responses filesCC-MAIN-2021-25/non200responses.paths.gz640001.76
URL index filesCC-MAIN-2021-25/cc-index.paths.gz3020.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-25/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

May 2021 crawl archive now available

The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.28 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-21/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-21/segment.paths.gz100
WARC filesCC-MAIN-2021-21/warc.paths.gz6400066.17
WAT filesCC-MAIN-2021-21/wat.paths.gz6400017.61
WET filesCC-MAIN-2021-21/wet.paths.gz640007.65
Robots.txt filesCC-MAIN-2021-21/robotstxt.paths.gz640000.17
Non-200 responses filesCC-MAIN-2021-21/non200responses.paths.gz640001.86
URL index filesCC-MAIN-2021-21/cc-index.paths.gz3020.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-21/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

April 2021 crawl archive now available

The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-17/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-17/segment.paths.gz100
WARC filesCC-MAIN-2021-17/warc.paths.gz6400069.78
WAT filesCC-MAIN-2021-17/wat.paths.gz6400021.05
WET filesCC-MAIN-2021-17/wet.paths.gz640009.25
Robots.txt filesCC-MAIN-2021-17/robotstxt.paths.gz640000.16
Non-200 responses filesCC-MAIN-2021-17/non200responses.paths.gz640001.76
URL index filesCC-MAIN-2021-17/cc-index.paths.gz3020.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-17/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

February/March 2021 crawl archive now available

The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.

Improvements and Fixes

The ISO639-3 code for the Hmong language was updated to "hmn" – the code "blu" used so far was already deprecated in 2008. Crawl archives prior to this crawl will still use the code "blu". More details about this update are found here.

Archive Location and Download

The February/March crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-10/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-10/segment.paths.gz100
WARC filesCC-MAIN-2021-10/warc.paths.gz6400062.51
WAT filesCC-MAIN-2021-10/wat.paths.gz6400018.44
WET filesCC-MAIN-2021-10/wet.paths.gz640008.06
Robots.txt filesCC-MAIN-2021-10/robotstxt.paths.gz640000.2
Non-200 responses filesCC-MAIN-2021-10/non200responses.paths.gz640001.58
URL index filesCC-MAIN-2021-10/cc-index.paths.gz3020.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-10/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

January 2021 crawl archive now available

The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content. It includes page captures of 1.15 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-04/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2021-04/segment.paths.gz100
WARC filesCC-MAIN-2021-04/warc.paths.gz7984078.98
WAT filesCC-MAIN-2021-04/wat.paths.gz7984022.92
WET filesCC-MAIN-2021-04/wet.paths.gz7984010.04
Robots.txt filesCC-MAIN-2021-04/robotstxt.paths.gz798400.23
Non-200 responses filesCC-MAIN-2021-04/non200responses.paths.gz798402.11
URL index filesCC-MAIN-2021-04/cc-index.paths.gz3020.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-04/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.