How much of the web does a crawler see?
Every web crawl is a sample. Common Crawl collects billions of pages a month, a focused academic crawl collects a hundred million URLs. Both are samples of something larger - everything the crawler’s configuration could reach but did not sample. The crawl size is known exactly. The size of that reachable web is not, and neither is the share of it held by a crawl.
One approach comes from ecology; catch, mark, release and catch again: the overlap between two samples estimates the population. The early ecologist assumed that nothing enters or leaves the population between the two catches (Petersen 1896, Lincoln 1930). But our crawl policy does. And unlike estimating the size of the Web through the overlap of search query results (Lawrence and Giles 1998) a large scale crawler does not have a parallel crawl to which to compare. So the two samples come in sequence, and the web keeps changing between them. Pages the first sample missed and pages that no longer exist both show up as "not seen again", and a naïve overlap mixes them.
A result we presented at the 30th TPDL (Theory and Practice for Digital Libraries) "Measuring What the Crawler Sees”, separates the two (missing and gone) with an urn model with turnover. Two model parameters determine the intersections and union of a crawl series, coverage and retention rate. Coverage expresses what fraction of the reachable web each crawl holds. Retention is defined as the fraction replaced over time. Both are retrieved via a regression for Common Crawl's domains. The simple model breaks in an instructive way and the repair reveals a persistent core of roughly 40% of all domains.
The simulation below is the model at work. Entities drift from right to left, their speed indicating how long they remain in the policy/alive; the circle is defining the edge of the crawl policy, catching whatever fraction \(c\) within its range. Head ranks (the core) are visualized in the center, while tail ranks (the shell/frontier) are at the extremities of the Web. The tail ranks decrease as moving from the center toward the top/bottom of the figure.
We observe that in these two known regimes 1) of the homogeneous and the 2) core + shell urn, both the containment (i.e. intersections) and discovery (i.e. union) curves below yield correct predictions about the latent input conditions to the Web’s behavior.
Move the sliders, give the web a persistent core, spread the speeds, and watch the fitted numbers follow. The preprint is available on arXiv. The data used for this work is published in the Common Crawl Statistics.
This is a complementary extension of our effort on which we have written in Measuring Crawled Coverage of a Website in Common Crawl. There, instead of looking at coverage indirectly, we analyzed it internally relative to the crawldb.

