< Back to Blog
May 9, 2019

Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What's new?

The software which builds the graph from WAT and WARC files has been extended to extract more links from the HTML <head> element:

  • more links are taken from <metadata> elements, e.g, the thumbnail meta name, Open Graph or twitter:* properties
  • links from <script> elements are now included

Note that previous web graph releases already include all kinds of links: not only <a href="..."> but also links to images and multi-media content, links from <form> elements, canonical links, and many more. While the domain-level graph shows almost the same size and metrics as the previous one released three months ago, the host-level graph has increased in size by 85 million nodes but is less densely connected. The growth in the number of nodes is mainly caused by a link spam cluster of 190 million hosts distributed over 15k domains. Thanks to the webgraph these domains (e.g., 24340.tw) are detected and the crawler is advised not to visit them again.

Host-level graph

The graph consists of 492 million nodes and 3.0 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 426 million dangling nodes (87%) and the largest strongly connected component contains 52 million (10.5%) nodes. You can download the graph and the ranks of all 492 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Feb/Mar/Apr 2019 host-level Webgraph

Size File Description
3.36 GB cc-main-2019-feb-mar-apr-host-vertices.paths.gz nodes ⟨id, rev host⟩, paths of 28 vertices files
14.40 GB cc-main-2019-feb-mar-apr-host-edges.paths.gz edges ⟨from_id, to_id⟩, paths of 56 edges files
6.33 GB cc-main-2019-feb-mar-apr-host.graph graph in BVGraph format
2 kB cc-main-2019-feb-mar-apr-host.properties
7.02 GB cc-main-2019-feb-mar-apr-host-t.graph transpose of the graph (outlinks inverted to inlinks)
2 kB cc-main-2019-feb-mar-apr-host-t.properties
1 kB cc-main-2019-feb-mar-apr-host.stats WebGraph statistics
7.85 GB cc-main-2019-feb-mar-apr-host-ranks.txt.gz harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 91 million nodes and 1.89 billion edges. 51% or 46 million nodes are dangling nodes, the largest strongly connected component covers 38 million or 42% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/domain/.

Download files of the Common Crawl Feb/Mar/Apr 2019 domain-level Webgraph

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl's Google Group!

This release was authored by:
No items found.