Common Crawl - Open Repository of Web Crawl Data

Latest Blog Post

CommonsDB, ISCC, Spaghetti, and Meatballs

Notes from the CommonsDB final conference in Alicante: a registry of 6.5 million open works keyed on ISCC, and the EUIPO's push for federated copyright infrastructure.

Thom Vaughan

Thom is Principal Engineer at the Common Crawl Foundation.

Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

Common Crawl is a 501(c)(3) non–profit founded in 2007.‍We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.

Over 300 billion pages spanning 15 years.

Free and open corpus since 2007.

Cited in over 10,000 research papers.

3–5 billion new pages added each month.

Featured Papers

Nathan Godey, et al.

Gaperon: A Peppered English-French Generative Language Model Suite

Sheriff Issaka, et al.

The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP

Badal Nyalang

Stereotyped by Silence: How LLMs Erase Northeast Indian Languages Through Omission and Orthographic Corruption

Xiangyu Wu, et al.

MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents

Michael Paris, Hande Celikkanat, Luca Foppiano

Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls

Julio Garbers, Terry Gregory

The Diffusion of Artificial Intelligence Across Firms: Evidence from Europe

Pedro Ortiz Suarez, et al.

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Asier Gutiérrez-Fandiño, et al.

esCorpius: A Massive Spanish Crawling Corpus

Latest Blog Post

CommonsDB, ISCC, Spaghetti, and Meatballs

The Data

Resources

Community

About

Common Crawl is a 501(c)(3) non–profit founded in 2007.
‍
We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.