Search results

Common Crawl - Privacy Policy

For the purposes of this Privacy Policy: “Organization”. (referred to as either. "the Organization". , “Common Crawl”. , "We". , "Us". or.…

Common Crawl - Blog - Web Archives for Social Sciences Datathon, Bristol

Web Archives for Social Sciences Datathon. , a two-day, hands-on event exploring how large-scale web archive data can be used for policy-relevant social and economic research. The Datathon was organised by the.…

Common Crawl - Blog - White House Briefing on Open Data’s Role in Technology

We recently had the honor of briefing the White House Office of Science and Technology Policy (OSTP) on the role of The Common Crawl Foundation as critical infrastructure in the artificial intelligence ecosystem and how we can support U.S. federal efforts in…

Common Crawl - Blog - The Promise of Open Government Data & Where We Go Next

Allison Domicone was formerly a Program and Policy Consultant to Common Crawl and previously worked for Creative Commons.…

Common Crawl - Blog - Common Crawl at the United Nations Open Source Week, June 2025

The Common Crawl Foundation team took part in the United Nations Open Source Week in New York City this June, meeting with global developers, researchers, and policymakers to discuss all things open source and AI. Common Crawl Foundation.…

Common Crawl - Team - Thom Vaughan

He works across infrastructure engineering, policy research, and open source advocacy.…

Common Crawl - Team - Lisa Green

She has worked in the areas of Open Access publishing, Open Science, Open Data, copyright, digital rights and policy. Lisa was Chief of Staff at Creative Commons and served as the director of Common Crawl from 2011 to 2015.…

Common Crawl - Blog - 5 Good Reads in Big Open Data: Feb 13 2015

If the majority of the world’s online population spends time on Facebook, then policymakers, businesses, startups, developers, nonprofits, publishers, and anyone else interested in communicating with them will also, if they are to be effective, go to Facebook…

Common Crawl - Blog - August/September 2024 Newsletter

Updates on our Policy Efforts. Roadmap and Future Plans. Common Crawl Citations in Academic Research. Common Crawl's impact on research has grown substantially since its beginning.…

Common Crawl - Blog - March 2014 Crawl Data Now Available

We're working hard to get a few machines always crawling domains with large numbers of pages to go even deeper while still maintaining our politeness policy. Thanks again to. Blekko. for their ongoing donation of URLs for our crawl. The Data. Overview.…

Common Crawl - Blog - Common Crawl Foundation at Stanford HAI

The presentation provided an introduction to Common Crawl and our data, and covered topics around crawler politeness and the Robots Exclusion Protocol, legal and policy issues, and web data and language coverage.…

Common Crawl - Blog - Amazon Web Services sponsoring $50 in credit to all contest entrants!

Allison Domicone was formerly a Program and Policy Consultant to Common Crawl and previously worked for Creative Commons. Did you know that every entry to the. First Ever Common Crawl Code Contest. gets $50 in Amazon Web Services (AWS) credits?…

Common Crawl - Blog - OSCON 2012

Allison Domicone was formerly a Program and Policy Consultant to Common Crawl and previously worked for Creative Commons. We're just one month away from one of the biggest and most exciting events of the year, O'Reilly's Open Source Convention (OSCON).…

Common Crawl - Research Papers

Privacy Policy. Terms of Use. Text Link…

Common Crawl - Blog - Common Crawl's Brand Spanking New Video and First Ever Code Contest!

Allison Domicone was formerly a Program and Policy Consultant to Common Crawl and previously worked for Creative Commons. At Common Crawl we’ve been busy recently!…

Common Crawl - Blog - Strata Conference + Hadoop World

Allison Domicone was formerly a Program and Policy Consultant to Common Crawl and previously worked for Creative Commons. This year’s Strata Conference teams up with Hadoop World for what promises to be a powerhouse convening in NYC from October 23-25.…

Common Crawl - Errata