Common Crawl - Open Repository of Web Crawl Data

Latest Blog Post

Getting Started with Common Crawl Data on Hugging Face

We made parts of our crawl archive available experimentally on a Hugging Face Storage Bucket. This blog post explains how to use the data on Hugging Face, with tools and examples on which you can build.

Malte Ostendorff

Malte is a Senior Research Engineer at Common Crawl.

Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

Common Crawl is a 501(c)(3) non–profit founded in 2007.‍We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.

Over 300 billion pages spanning 15 years.

Free and open corpus since 2007.

Cited in over 10,000 research papers.

3–5 billion new pages added each month.

Featured Papers

Nathan Godey, et al.

Gaperon: A Peppered English-French Generative Language Model Suite

Sheriff Issaka, et al.

The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP

Badal Nyalang

Stereotyped by Silence: How LLMs Erase Northeast Indian Languages Through Omission and Orthographic Corruption

Xiangyu Wu, et al.

MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents

Michael Paris, Hande Celikkanat, Luca Foppiano

Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls

Julio Garbers, Terry Gregory

The Diffusion of Artificial Intelligence Across Firms: Evidence from Europe

Pedro Ortiz Suarez, et al.

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Asier Gutiérrez-Fandiño, et al.

esCorpius: A Massive Spanish Crawling Corpus

Latest Blog Post

Getting Started with Common Crawl Data on Hugging Face

The Data

Resources

Community

About

Common Crawl is a 501(c)(3) non–profit founded in 2007.
‍
We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.