Errata
Here you can find comprehensive information about errata that affect our data releases, including crawl data, and web graphs. If you have any problems to report please Contact Us.
Erroneous title field in WAT records
The title field in WAT record is extracted from last but not first <title> element in an HTML page
Read More...Missing fetch_status fields
In our columnar index for this crawl, the `content_mime_type` is missing and `fetch_status` is always -1. In the cdx index (columnar: `content_mime_type`), fields "mime" and "status" are missing.
Read More...Redundant extra line in response records
The WARC files of the August 2018 crawl contain a redundant empty line between the HTTP headers and the payload of WARC response records.
Read More...Incorrect fetch_time metadata
In crawls CC-MAIN-2016-36 to CC-MAIN-2016-50, the fetch_time metadata for robots.txt is incorrect.
Read More...Charset Detection Bug in WET Records
The charset detection required to properly transform non-UTF-8 HTML pages in WARC files into WET records didn't work before November 2016 (CC-MAIN-2016-50) due to a bug in IIPC Web Archive Commons.
Read More...ARC Format (Legacy) Crawls
Our early crawls were archived using the ARC (Archive) format, not the WARC (Web ARChive) format.
Read More...Missing Language Classification
Starting with crawl CC-MAIN-2018-39 we added a language classification field (‘content-languages’) to the columnar indexes, WAT files, and WARC metadata for all subsequent crawls.
Read More...Some 2–Level CCTLDs Excluded
A bad configuration was checked into our exclusion list on Sep 22, 2022 and was fixed on Oct 27, 2023. The configuration blocked a number of 2–level domains, meaning they were not included in certain crawls.
Read More...
.svg)
