Common Crawl maintains a free,open repository of web crawl data that can be used by anyone.
Common Crawl is a 501(c)(3) non–profit founded in 2007. We make wholesale extraction, transformation and analysis of open web data accessible to researchers.
Common Crawl was invited to the AI Plumbers unconference held at FOSDEM this year. The contrast between the 100 people at the unconference, compared to the 10,000 people at the main event, couldn't be bigger.
Thijs Dalhuijsen
Thijs Dalhuijsen is a Senior Software Engineer at Common Crawl. He works on backend systems, automation, and data infrastructure to power large-scale web access and analysis.
.svg)
