Everyone already heard about scalability at least once. Everyone already heard about memcached as well. What not everyone might heard is the dog-pile effect and how to avoid it. But before we start, let’s take a look on how to use Rails with memcached.

Rails + Memcached = <3

First, if you never used memcached with rails or never read/heard a lot about scalability, I recommend checking out Scaling Rails episodes done by Gregg Pollack, in special the episode about memcached.

Assuming that you have your memcached installed and want to use it on your application, you just need to add the following to your configuration files (for example production.rb):

config.cache_store = :mem_cache_store

By default, Rails will search for a memcached process running on localhost:11211.

But wait, why would I want to use memcached? Well, imagine that your application has a page where a slow query is executed against the database to generate a ranking of blog posts based on the author’s influence and this query takes on average 5 seconds. In this case, everytime an user access this page, the query will be executed and your application will end up having a very high response time.

Since you don’t want the user to wait 5 seconds everytime he wants to see the ranking, what do you do? You store the query results inside memcached. Once your query result is cached, your app users do not have to wait for those damn 5 seconds anymore!

What is the dog-pile effect?

Nice, we start to cache our query results, our application is responsive and we can finally sleep at night, right?

That depends. Let’s suppose we are expiring the cache based on a time interval, for example 5 minutes. Let’s see how it will work in two scenarios:

1 user accessing the page after the cache was expired:

In this first case, when the user access the page after the cache was expired, the query will be executed again. After 5 seconds the user will be able to see the ranking, your server worked a little and your application is still working.

N users accessing the page after the cache was expired:

Imagine that in a certain hour, this page on your application receives 4 requests per second on average. In this case, between the first request and the query results being returned, 5 seconds will pass and something around 20 requests will hit your server. The problem is, all those 20 requests will miss the cache and your application will try to execute the query in all of them, consuming a lot of CPU and memory resources. This is the dog-pile effect.

Depending on how many requests hit your server and the amount of resources needed to process the query, the dog-pile effect can bring your application down. Holy cow!

Luckily, there are a few solutions to handle this effect. Let’s take a look at one of them.

Dog pile effect working on your application!

How to avoid the dog-pile effect?

The dog-pile effect is triggered because we allowed more than one request to execute the expensive query. So, what if we isolate this operation to just the first request and let the next requests use the old cache until the new one is available? Looks like a good idea, so let’s code it!

In the next minutes, the first request will notice that the cache is stale (line 17) and will create a lock so only it will calculate the new cache;

In the next 5 seconds, the new query is calculated and all requests, instead of missing the cache, will access the old cache and return it to the client (lines 17 and 21)k;

When the query result is returned, it will overwrite the old cache with the new value and remove the lock (lines 31 and 32);

From now on, all new requests in the next five minutes will access the fresh cache and return it (lines 17 and 21).

Fallbacks and a few things to keep in mind

First, is not recommend to set the :expires_in value in your cache:

Rails.cache.write('my_key', 'my_value', :expires_in => 300)

With the solution proposed above, you just need to set :expires_delta. This is due to the fact that our application will now be responsible to expire the cache and not memcached.

Rails.cache.write('my_key', 'my_value', :expires_delta => 300)

However, there are a few cases where memcached can eventually expire the cache. When you initialize memcached, it allocates by default 64MB in memory. If eventually those 64MB are filled, what will memcached do when you try to save a new object? It uses the LRU algorithm and deletes the less accessed object in memory.

In such cases, where memcached removes a cache on its own, the dog pile effect can appear again. Suppose that the ranking is not accessed for quite some time and the cached ranking is discarded due to LRU. If suddenly a lot of people access the page in the five initial seconds where the query is being calculated, requests will accumulate and once again the dog-pile effect can bring your application down.

It’s important to have this scenario in mind when you are sizing your memcached, mainly on how many memory will be allocated.

Now I can handle the dog-pile effect and sleep again!

Summarizing, when your are using a cache strategy, you will probably need to expire your cache. In this process, the dog-pile effect can appear and haunt you down. Now you have one (more) tool to solve it.

You just need to add the SmartMemCacheStore code above to your application (for example in lib/), set your production.rb (or any appropriated environment) to use the :smart_mem_cache_store. If you use Rails default API to access the cache (Rails.cache.read, Rails.cache.write) and designed well your memcached structure, you will be protected from the dog-pile effect.

Hugo, great article. One question though. How would you deal with the situation where there was no previously cached response, which can happen if it is the very first request, or the cache was swept clear? Would it make sense to block the other requests until the cache is ready, or do you have another approach?

The dogpile effect is pretty common in high traffic sites, should the cache built into Rails handle it automatically?

Hugo, great article. One question though. How would you deal with the situation where there was no previously cached response, which can happen if it is the very first request, or the cache was swept clear? Would it make sense to block the other requests until the cache is ready, or do you have another approach?

The dogpile effect is pretty common in high traffic sites, should the cache built into Rails handle it automatically?

@Dan We could have some solutions. One of them is the one you said. You could make the other requests sleep for some time and after that, let them try to access the cache again.

Another one, you can pre-warm the cache yourself instead of wait for a user request for it.

It depends on your app requirments.

I’m not sure about putting that solution inside the cache built into Rails, mainly because there are different solutions, that depends on the situation. For example, is common to have another process generating the cache or use a namespaced memcached. All of them have positive and negative aspects. =)

Hugo Baraúna

@Dan We could have some solutions. One of them is the one you said. You could make the other requests sleep for some time and after that, let them try to access the cache again.

Another one, you can pre-warm the cache yourself instead of wait for a user request for it.

It depends on your app requirments.

I’m not sure about putting that solution inside the cache built into Rails, mainly because there are different solutions, that depends on the situation. For example, is common to have another process generating the cache or use a namespaced memcached. All of them have positive and negative aspects. =)

Hugo Baraúna

@Priit, I also find very interesting what we can do with just a simple key-value based data store. Actually, we are using that namespace solution too.