Over a year ago we released an experimental version of cc-downloader, a command-line tool for politely downloading Common Crawl data via HTTPS. We introduced cc-downloader in response to a significant increase in downloads and requests for our datasets, which made it harder for many users to successfully access the data in our bucket due to the high traffic.
cc-downloader, which was initially meant to be an experimental tool, used a polite retry mechanism that implemented jitter and exponential backoff strategies to avoid overwhelming our infrastructure. It was also developed in Rust, allowing us to enable parallel downloads easily, while also keeping our tool memory-safe.
Over the course of a year, we monitored the usage of cc-downloader and saw significant adoption from the community.
cc-downloader is now one of the top 10 most used clients to download Common Crawl data through CloudFront, accounting for almost 14 PiB downloaded in one and a half years.
As such, we decided to completely refactor cc-downloader, turning it into a library with its own documentation. And seeing that python-requests is by far the most used client to download our data, we also decided to release Python bindings, in the hope that this will encourage the community to adopt cc-downloader more widely, instead of using potentially impolite implementations of python-requests.
Finally, we added support for downloading contrib content in our bucket, as well as for filtering the URL index by subset. We also fixed various issues opened by the community over the course of last year, and we completely rewrote the documentation for the standalone binary.
We hope to continue developing and supporting cc-downloader in the future, so if you find any issues or would like to submit a feature request, please visit its GitHub repository, where you will find our new documentation as well as easy instructions to install and start using cc-downloader right away.
.svg)
