< Back to Blog
October 18, 2023

Host- and Domain-Level Web Graphs Mar/May/Oct 2023

Note: this post has been marked as obsolete.
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, May, and October 2023. The host-level graph consists of 378.7 million nodes and 2.6 billion edges, and the domain-level graph has 94.2 million nodes and 1.7 billion edges.
Thom Vaughan
Thom Vaughan
Thom is Principal Technologist at the Common Crawl Foundation.

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, May, and October 2023. The crawls used to generate the graphs were CC-MAIN-2023-14, CC-MAIN-2023-23, and CC-MAIN-2023-40. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior web graph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of web graph notebooks.

Host-level Graph

The host-level graph consists of 378.7 million nodes and 2.6 billion edges.

There are 321.4 million dangling nodes (84.85%) and the largest strongly connected component contains 44.2 million (11.67%) nodes. Dangling nodes stem from:

- Hosts that have not been crawled, yet are pointed to from a link on a crawled page
- Hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404).

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 378.7 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2023-mar-may-oct/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2023-mar-may-oct/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 40 gzip-compressed files listed in two path listings - one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using "gzip -d" or "gunzip". By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download Files of the Common Crawl March, May, October 2023 Host-level Web Graph

Size File Description
176 Bytes cc-main-2023-mar-may-oct-host-vertices.paths.gz nodes ⟨id, rev host⟩, paths of 16 vertices files
195 Bytes cc-main-2023-mar-may-oct-host-edges.paths.gz edges ⟨from_id, to_id⟩, paths of 24 edges files
5.5 GiB cc-main-2023-mar-may-oct-host.graph graph in BVGraph format
1.3 KiB cc-main-2023-mar-may-oct-host.properties
6.1 GiB cc-main-2023-mar-may-oct-host-t.graph transpose of the graph (outlinks inverted to inlinks)
1.3 KiB cc-main-2023-mar-may-oct-host-t.properties
816 Bytes cc-main-2023-mar-may-oct-host.stats WebGraph statistics
6.4 GiB cc-main-2023-mar-may-oct-host-ranks.txt.gz harmonic centrality and pagerank

Domain-level Graph


The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) 78bff2b of the public suffix list was used (commit date 2023-11-03T16:47:17Z).

The domain-level graph has 94.2 million nodes and 1.7 billion edges. 53.6% or 50.5 million nodes are dangling nodes, the largest strongly connected component covers 35.7 million or 37.95% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2023-mar-may-oct/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2023-mar-may-oct/domain/.

Download Files of the Common Crawl March, May, October 2023 Domain-level Web Graph

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc.

Let us know about your results via Common Crawl's Google Group!

This post was updated on Saturday 4th November 2023 to correct some erroneous metrics — the previous version was lacking the counts for the number of hosts (subdomains) per domain.

This release was authored by:
Julien is a member of the Apache Software Foundation, Emeritus member of the Common Crawl Foundation, and is the creator of StormCrawler.
Julien Nioche
Sebastian is a Distinguished Engineer with Common Crawl.
Sebastian Nagel