Interactive Webgraph Statistics Notebook Released

We are pleased to announce the release of an interactive Jupyter notebook that is used to provide: Visualization of web graph statistics An interface for interacting with the webgraph The visualization of the web graph statistics is done by leveraging the WebGraph framework, which provides means of gathering many interesting data points of a web […]

Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2020 and January 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also […]

Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit […]

Host- and Domain-Level Web Graphs Feb/Mar/May 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March/April and May/June 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit […]

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also […]

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also […]

August 2019 crawl archive now available

The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th. The August crawl contains page captures of 1.1 billion URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks […]

July 2019 crawl archive now available

The crawl archive for July 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between July 15th and 24th. The July crawl contains page captures of 810 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks […]

June 2019 crawl archive now available

The crawl archive for June 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between June 16th and 27th with an operational break from 21st to 24th. The June crawl contains page captures of 880 million URLs not contained in any crawl archive before. New URLs are […]