Search results

The column url_host_name_reversed was added to the. columnar index. It holds the host name in reverse domain name notation (com.example.www) which is more efficient to query. In order to make use of the new column please use the. updated table schema.…

The following improvements and fixes to the data formats have been made: the. columnar index. contains the content language of a web page as a new field. Please read the instructions below how to upgrade your tools to read newly added fields.…

Improvements and Fixes. date time values in the column "fetch_time" of the. columnar index. are now stored using the "int64" data type. For details and compatibility issues please see. cc-index-table#7.…

In our columnar index for this crawl, the `. content_mime_type. ` is missing and `. fetch_status. ` is always -1. In the cdx index (columnar: `. content_mime_type. `), fields `. mime. ` and `. status. ` are missing. Affected Crawls. The Data. Overview.…

The URL index fields "redirect" and "mime" haven't been filled if the corresponding HTTP headers Location and Content-Type are written in lower-case letters or any other variant not matching case.…

The flag in our URL indexes (CDX and columnar) that indicates whether or not a WARC record payload was truncated was added in CC-MAIN-2019-47. This indicator is missing in our indexes for all previous crawl releases.…

ISO-639-3 code. are shown in the URL index as a new field, e.g. "languages": "zho,eng". The WARC metadata records contain the full CLD2 response including scores and text coverage: On github you'll find the.…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-43/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-39/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-39/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-21/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-21/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-34/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-45/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-05/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-33/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2023-14/. Also the. columnar index. has been updated to contain this Crawl. The Data. Overview. Web Graphs. Latest Crawl. Crawl Stats. Graph Stats. Errata.…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-25/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-04/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-17/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-51/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-16/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-40/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-50/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-43/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2023-06/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-10/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-49/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-13/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-27/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-13/. Also the. columnar index. has been updated to contain this crawl.…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-47/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-30/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-40/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-10/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-30/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-51/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-22/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-04/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-22/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-43/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-18/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-31/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-24/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-26/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-35/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-26/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-17/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

We've added two new fields to the URL indexes (CDX and columnar): the redirect target location is stored in the CDX JSON field "redirect" resp. the column "fetch_redirect".…

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-09/. Also the. columnar index. has been updated to contain this crawl. Please. donate. to Common Crawl if you appreciate our free datasets!…

Starting with crawl CC-MAIN-2018-39 we added a language classification field (‘content-languages’) to the columnar indexes, WAT files, and WARC metadata for all subsequent crawls.…

Index to WARC Files and URLs in Columnar Format. We're happy to announce the release of an index to WARC files and URLs in a columnar format.…

Columnar (Parquet) Indexes. In addition to the above, we provide an index for WARC files and URLs in a columnar format using. Apache Parquet™. This enables more efficient querying and data analysis.…

You may wish to use our columnar index via Amazon Athena or Apache Spark if your query involves broad or large-scale filtering. These tools are better suited to high-volume access patterns and provide more flexibility for complex queries.…

If you’re sending small requests for index information or single webpages contained in WARC files, we can handle a few thousand requests per second total for everyone combined, so you’ll want to stay below 10 per second, or if things someday become better,…

Common Crawl URL Index. Note: this post has been marked as obsolete. We are thrilled to announce that Common Crawl now has a URL index! Scott Robertson, founder of triv.io graciously donated his time and skills to creating this valuable tool.…

We are pleased to announce a new index and query api system for Common Crawl. The raw index data is available, per crawl, at: s3://commoncrawl/cc-index/collections/CC-MAIN-YYYY-WW/indexes/. There is now an index for the Jan 2015 and Feb 2015 crawls.…

Analysis of the NCSU Library URLs in the Common Crawl Index. Note: this post has been marked as obsolete. Last week we announced the Common Crawl URL Index.…

Questions about Common Crawl’s indexes, both cdx and columnar. Questions about example uses of Common Crawl data. Generic questions about web archiving. The end of most answers contains a link to a specific webpage with more information about the answer.…

A couple months ago we announced the creation of the Common Crawl URL Index and followed it up with a guest post by Jason Ronallo describing how he had used the URL Index.…