Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

Common Crawl is a 501(c)(3) non–profit founded in 2007.
‍
We make wholesale extraction, transformation and analysis of open web data accessible to researchers.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, Daya Guo

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Computation and Language

Asier Gutiérrez-Fandiño, David Pérez-Fernández, Jordi Armengol-Estapé, David Griol, Zoraida Callejas

esCorpius: A Massive Spanish Crawling Corpus

The Web as a Graph (Master's Thesis)

Marius Løvold Jørgensen, UiT Norges Arktiske Universitet

BacklinkDB: A Purpose-Built Backlink Database Management System

Internet Security: Phishing Websites

Asadullah Safi, Satwinder Singh

A Systematic Literature Review on Phishing Website Detection Techniques

More on Google Scholar Curated BibTeX Dataset

Latest Blog Post:

News

The First WMDQS-Masakhane LangID Hackathon

In June 2025 the Common Crawl Foundation, MLCommons, and EleutherAI had the pleasure of hosting a virtual hackathon in partnership with Masakhane in order to collect language identification annotations for African languages.

Pedro Ortiz Suarez

Pedro is a French-Colombian mathematician, computer scientist, and researcher. He holds a PhD in computer science and Natural Language Processing from Sorbonne Université.

Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

Common Crawl is a 501(c)(3) non–profit founded in 2007.‍We make wholesale extraction, transformation and analysis of open web data accessible to researchers.

Over 250 billion pages spanning 15 years.

Free and open corpus since 2007.

Cited in over 10,000 research papers.

3–5 billion new pages added each month.

Featured Papers:

A study on web crawlers facing inconsistent and poorly-signalled blocking

Mostafa Ansar, Anna Sperotto, Ralph Holz

Web Crawl Refusals: Insights From Common Crawl

Research on Free Expression Online

Jeffrey Knockel, Jakub Dalek, Noura Aljizawi, Mohamed Ahmed, Levi Meletti, and Justin Lau

Banned Books: Analysis of Censorship on Amazon.com

Analyzing the Australian Web with Web Graphs: Harmonic Centrality at the Domain Level

Xian Gong, Paul X. McCarthy, Marian-Andrei Rizoiu, Paolo Boldi

Harmony in the Australian Domain Space

The Dangers of Hijacked Hyperlinks

Kevin Saric, Felix Savins, Gowri Sankar Ramachandran, Raja Jurdak, Surya Nepal

Hyperlink Hijacking: Exploiting Erroneous URL Links to Phantom Domains

Enhancing Computational Analysis