Menu

Monitor Dalli Connection Changes + Failures

Memcached servers changing leads to a tiny split-brain scenario since some servers might read from different caches then the others … good to keep an eye on it and alert when it happens too often. Here is a tiny snippet to report when it happens.

# config/initializers/dalli.rb
# Whenever the alive-ness of a server changes we read keys from a different server
# which leads to stale keys on the old server and cache-misses on the new servers
# so this should not happen often
# see lib/dalli/server.rb
#
# reproduce: rails c + Rails.cache.get + zdi memcached stop & start
Dalli::Server.prepend(Module.new do
def down!
$statsd.increment "dalli.connection_changed", tags: ["state:down"] unless @down_at
super
end
def up!
$statsd.increment "dalli.connection_changed", tags: ["state:up"] if @down_at
super
end
def failure!(*)
$statsd.increment "dalli.failed"
super
end
end)