You might have the question of how many pages of any particular website are in Common Crawl’s dataset. This question is harder to answer than it seems – it’s not just the number of pages we crawled in the most recent crawl. Instead, our crawler forms an idea of how fast particular pages change, notices pages that change very slowly or never, and crawls those pages less and less often. This is good because it allows the crawler to focus on newly discovered pages, and pages that frequently change. But it’s bad in that it makes it hard to understand how complete the dataset is.
A glimpse inside the CrawlDb
In this diagram, webpages that the crawler thinks are fresh enough to not need recrawling pile up in the “fetched” bubble. The size of this list of pages is our estimate of how many fresh webpages we have crawled from a website.
In this chart, which comes from the testing Host Index v2, the purple line is the number of fetched pages according to CrawlDb, and the blue is the number fetched in that particular crawl. Common Crawl’s website has become significantly more popular over time, which explains why the number of pages retained was so small in 2020. By the end of this test dataset, our rank and the CrawlDb number of fetched pages are both steadily rising, while the number of pages fetched per crawl is much smaller.
The Next Question
The next question you might want to ask is: what fraction of the total pages on a website have we crawled? Not only the ones that we know about, but also the ones that we’ve never seen? Some websites have “sitemaps” and some of those sitemaps are actually accurate, so there’s one way to compute the fraction for some websites.
Another way to compute the fraction is to use math – a lot of math. For that, three Common Crawl engineers recently wrote this paper:
.svg)
