Optimising Web Delivery

Squid use at Flickr

(Last updated Feburary 12 2007)

Flickr uses Squid as a reverse proxy accelerator in front of its photo storage
service. All photos are stored on persistent, centralised storage which makes
it easy to perform capacity planning - storage is added to handle more photos,
Squid servers are added to serve increasing numbers of users and page views.

Flickr currently serves 30 million objects across its farm, with a hit rate of
roughly 75-80%. Layer-7 URL hashing is used to send requests to the best cache
to achieve such an optimal hit rate. They keep 3-5 hours of hot objects in each
cache. Each Squid server handles approximately 1000 requests a second with
the Squid-2.6 codebase.

Each cache is configured to use LRU as the memory and disk replacement policy
and purposefully miss on objects above a certain size to optimise throughput
and performance over higher cache hit rates.