Common Crawl - Blog - November 2024 Crawl Archive Now Available

November 2024 Crawl Archive Now Available

The crawl archive for November 2024 is now available. The data was crawled between November 1st and November 15th, and contains 2.68 billion web pages (or 405 TiB of uncompressed content). Page captures are from 47.5 million hosts or 38.3 million registered domains and include 1 billion new URLs, not visited in any of our prior crawls.

Erratum:

WARC Content-Type header in revisit records

Originally reported by:

Sebastian Nagel

More details

Common Crawl's WARC revisit records use Content-Type: message/http (following the WARC 1.1 spec's example), but per iipc/warc-specifications#55 it should be application/http;msgtype=response for consistency with other HTTP response records.

Erratum:

Redirect target URL in URL indexes may be a relative URL

Originally reported by:

Sebastian Nagel

More details

When the HTTP “Location” header includes a relative URL, the corresponding “redirect” field in the CDX index and “fetch_redirect” field in the columnar index will also store a relative URL. In all other cases, redirect targets in the URL indexes should be recorded as absolute URLs.

Erratum:

Content is truncated

Originally reported by:

More details

Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.

Erratum:

SURT URLs do not properly encode non-UTF-8 percent-encoded characters

Originally reported by:

Tom Morris

More details

When constructing SURT (Sort-friendly URI Reordering Transform) URLs, percent-encoded characters that are not valid UTF-8 sequences were not being correctly handled. This could lead to inconsistencies in URL normalization and sorting, potentially causing incorrect deduplication or retrieval issues in datasets that rely on SURT-based indexing.

Erratum:

WAT data: repeated WARC and HTTP headers are not preserved

Originally reported by:

More details

Repeated HTTP and WARC headers were not represented in the JSON data in WAT files.

Erratum:

WARC revisit metadata records

Originally reported by:

More details

The revisit records in the Common Crawl WARC archives (since Aug 2018) lack the metadata record which is attached to all response records.

	File List	#Files	Total Size Compressed (TiB)
Segments	segment.paths.gz	100
WARC	warc.paths.gz	90000	82.67
WAT	wat.paths.gz	90000	18.83
WET	wet.paths.gz	90000	7.39
Robots.txt	robotstxt.paths.gz	90000	0.14
Non-200 responses	non200responses.paths.gz	90000	2.58
URL index	cc-index.paths.gz	302	0.20
Columnar URL index	cc-index-table.paths.gz	900	0.23

Erratum:

WARC Content-Type header in revisit records

Erratum:

Redirect target URL in URL indexes may be a relative URL

Erratum:

Content is truncated

Erratum:

SURT URLs do not properly encode non-UTF-8 percent-encoded characters

Erratum:

WAT data: repeated WARC and HTTP headers are not preserved

Erratum:

WARC revisit metadata records

The Data

Resources

Community

About