You might have the question of how many pages of any particular website are in Common Crawl’s dataset. This question is harder to answer than it seems – it’s not just the number of pages we crawled in the most recent crawl. Instead, our crawler forms an idea of how fast particular pages change, notices pages that change very slowly or never, and crawls those pages less and less often. This is good because it allows the crawler to focus on newly discovered pages, and pages that frequently change. But it’s bad in that it makes it hard to understand how complete the dataset is.
A glimpse inside the CrawlDB
In this diagram, webpages that the crawler thinks are fresh enough to not need recrawling pile up in the “fetched” bubble. The size of this list of pages is our estimate of how many fresh webpages we have crawled from a website.
This number of fetched pages is not currently visible in any production data products – but it is in our (in public testing) Host Index v2.
In this chart, which comes from the testing Host Index v2, the purple line is the number of fetched pages according to CrawlDB, and the blue is the number fetched in that particular crawl. Common Crawl’s website has become significantly more popular over time, which explains why the number of pages retained was so small in 2020. By the end of this test dataset, our rank and the CrawlDB number of fetched pages are both steadily rising, while the number of pages fetched per crawl is much smaller.
The Next Question
The next question you might want to ask is: what fraction of the total pages on a website have we crawled? Not only the ones that we know about, but also the ones that we’ve never seen? Some websites have “sitemaps” and some of those sitemaps are actually accurate, so there’s one way to compute the fraction for some websites.
Another way to compute the fraction is to use math – a lot of math. For that, three Common Crawl engineers recently wrote this paper:
Michael Paris, Hande Celikkanat, Luca Foppiano. Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls.
https://arxiv.org/abs/2607.13636
Abstract:
A longitudinal web crawl is a sequence of partial samples of an evolving URL population. Pairwise containment between two crawls is the standard probe; under the homogeneous urn model it recovers per-round survival α and coverage c, but treats the population as uniform and consumes one pair at a time. We extend this analysis with the discovery curve U(s,T), the cumulative URL footprint over a sliding window of T crawls starting at s, which under the same urn model is also a closed-form function of (α,c). Containment and the discovery curve are then two projections of one process: independent fits agree on (α,c) when the urn is homogeneous, so any disagreement is itself a measurement. Applied to Common Crawl (2020–2025, domain granularity) and to the German Academic Web (GAW, URL granularity), the two projections disagree on both archives, and a two-component urn with a persistent core fraction κ alongside shell parameters (α∂,c∂) reconciles the disagreement. A residual on c∂ remains, signalling that the shell itself is not homogeneous; κ is recorded as the scalar entry point to a rank-resolved generalisation, which is left to follow-up work.
.svg)
