Search results

CDXJ Index

Note: In other documentation, this index is often referred to as the CDX index for legacy reasons. What is the CDXJ Index? The CDXJ index is one of the indices available for querying the Common Crawl corpus.…

Common Crawl - Erratum - Missing content_truncated flag in URL indexes

The flag in our indexes (CDX Index and URL Index) that indicates whether or not a WARC record payload was truncated was added in CC-MAIN-2019-47. This indicator is missing in our indexes for all previous crawl releases.…

Common Crawl - Blog - Announcing the Common Crawl Index!

ZipNum’ CDX format. and it is the same format that is used by the Wayback Machine at the Internet Archive. Index Query API.…

Common Crawl - Blog - You can now build directly on Common Crawl from the browser

Look up a URL in the CDX index. const. cdx =. await. fetch(. 'https://index.commoncrawl.org/CC-MAIN-2026-17-index'. +. '?url=example.com&output=json&limit=1'. ).then(. r. =>. r.text()); const. { filename, offset, length } =. JSON. .parse(cdx.split(.…

Common Crawl - Erratum - Redirect target URL in URL indexes may be a relative URL

When the HTTP “Location” header includes a relative URL, the corresponding “redirect” field in the CDX Index and “fetch_redirect” field in the URL Index (previously called the "Columnar Index") will also store a relative URL.…

Common Crawl - Blog - November 2019 crawl archive now available

We've added two new fields to the URL indexes (CDX and columnar): the redirect target location is stored in the CDX JSON field "redirect" resp. the column "fetch_redirect".…

Common Crawl - Erratum - Missing fetch_status fields

In the cdx index (columnar: `. content_mime_type. `), fields `. mime. ` and `. status. ` are missing. Affected Crawls. The Data. Overview. CDXJ Index. URL Index. Web Graphs. Latest Crawl. Crawl Stats. Graph Stats. Errata. Resources. Get Started. AI Agent.…

Common Crawl - Blog - Introducing the New Examples & Resources Browser

Looking for something that works with CDX indexes? Type "CDX" and you'll have your answer in a keystroke. Filter by what matters.…

Common Crawl - Erratum - No Truncation Indicator in WARC Records

The "length" in the CDX index is the length of the gzip-compressed WARC record. The name in the columnar index. warc_record_length. reflects this better. It is also worth noting that PDFs end with. %%EOF. perhaps followed by a linefeed. Affected Crawls.…

Common Crawl - Blog - Introducing Common Crawl AI Agent by ReadyAI

Questions about Common Crawl’s indexes, both cdx and columnar. Questions about example uses of Common Crawl data. Generic questions about web archiving. The end of most answers contains a link to a specific webpage with more information about the answer.…

Common Crawl - FAQ

Our CDX API endpoint is frequently abused and therefore heavily rate limited. If your client sends too many requests in a short period of time, your IP address may be temporarily blocked. To avoid connection issues, always use.…

Common Crawl - Blog - May/June 2024 Newsletter

If you use the cdx index, https://index.commoncrawl.org/collinfo.json. now has 2 new fields, “from” and “to”, giving the exact dates when the crawling started and ended. Volunteer for Common Crawl!…

Common Crawl - Blog - September 2018 crawl archive now available

The. columnar index. has been updated to contain two new fields added to WARC and CDX files starting with the. August crawl. : content_charset: the character encoding used by the HTML page. content_languages: a comma-separated list of.…

Common Crawl - Blog - Announcing the Whirlwind Tour of Common Crawl's Datasets using Python

We also play with some useful Python packages for interacting with the data: warcio. , cdxj-indexer. , cdx_toolkit. , and. duckdb. By the end of the Tour, users should have the foundation they need to start using Common Crawl's data in their own projects.…

Common Crawl - Blog

CDXJ Index. URL Index. Web Graphs. Latest Crawl. Crawl Stats. Graph Stats. Errata. Resources. Get Started. AI Agent. Blog. Examples. CCBot. Infra Status. Opt-Out Registry. FAQ. Community. Research Papers. Mailing List Archive. Hugging Face. Discord.…

Common Crawl - Our Team

Common Crawl - Collaborators

Common Crawl - Blog - The Columnar Index Is Now the URL Index

CDXJ Index. As its new name makes clear, it is an index to the URLs and WARC files in the corpus, stored in a columnar format (Apache Parquet™).…

Common Crawl - Team - Grogmar "Grog" Thunderfist

Common Crawl - Open Repository of Web Crawl Data

Common Crawl - Erratum - Charset Detection Bug in WET Records

Common Crawl - Team - Hugh Marbury

Common Crawl - Erratum - WAT data: repeated WARC and HTTP headers are not preserved

Common Crawl - Erratum - Nodes in Domain-Level Webgraphs Not Sorted and May Include Duplicates

Common Crawl - Research Papers

Common Crawl - Team - Alex Xue