We are pleased to announce the release of host-level and domain-level web graphs based on the published crawls of May, June, and July 2017. These graphs, along with ranked lists of hosts and domains, follow on our first host-level web graph (February, March, April 2017). Detailed information about the data formats, the processing pipeline, our objectives, and credits can be found in the prior announcement.
Host-level graph
The graph consists of 1.3 billion nodes and 5.25 billion edges. The graph includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.
You can download the graph and the ranks of all 1.3 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/hostgraph/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/hostgraph/ as prefix to access the files from everywhere.
The following files and formats are provided:
Note that differences in the rankings and the structure of the web graph are due our objective to make monthly crawls more diverse and to reduce the overlap between consecutive crawls. During both February/March/April and May/June/July we crawled about 9 billion pages. As an indicator of less overlap the number of unique URLs increased from 5.0 to 6.2 billion and the number of unique hosts went up from 70 to 90 million (or from 65 to 82 million with leading "www." removed). The largest strongly connected component contains now 59 million nodes/hosts (45 million in the February/March/April graph). However, the May/June/July host-level graph has doubled its size in terms of edges and more than tripled in terms of nodes. This growth is caused by a significant increase in the number of dangling nodes. 1.2 billion dangling nodes provide a solid foundation to extend the next crawls but we need a closer look at the distribution of these hosts among domains and TLDs.
Domain-level graph
The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs). The extraction of PLDs is based on the public suffix list from publicsuffix.org. Only "ICANN" domains are accepted, "private" domains are not (cf. section "divisions" in the documentation on publicsuffix.org). In short, foo.blogspot.com and data.commoncrawl.org are not accepted as pay-level domains, they are aggregated as domains blogspot.com resp. amazonaws.com.
The domain-level graph has 91 million nodes and 1,071 million edges. 55% are dangling nodes, the largest strongly connected component covers 30 million or 33% of the nodes.
All files related to the domain graph are placed on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/domaingraph/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/domaingraph/.
Download files of the Common Crawl May/June/July 2017 domain-level Webgraph
Credits
Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.
We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link SPAM detection, etc. Let us know about your results via Common Crawl's Google Group!
.svg)
