Custom Redis Command: Performance of C vs Lua

24 Dec 2012

Over the last fewdays, we've been writing a custom Redis command in C. The command, xdiff, is tailored to a specific use-case: diff'ing a set from a sorted set with an offset and count. A reasonable question is: why not just write it in Lua?

Keeping in mind the dangers of such synthetic benchmarks, I decided to compare the C implementation to one written in Lua. The full C implementation can be seen at the end of part 2. Here's the Lua version:

local zset = KEYS[1]
local diff = KEYS[2]
local from = ARGV[1]
local to = tonumber(ARGV[2])
local count = to - from
local results = {}
while true do
local values = redis.call('zrevrange', zset, from, to)
if #values == 0 then
return results
end
for index,value in ipairs(values) do
if redis.call('sismember', diff, value) == 0 then
table.insert(results, value)
if #results == count then
return results
end
end
end
from = to + 1
to = from + count
end

Essentially, I'm interested in finding out how each performs based on the number of similarities between our sorted set and our set. The more they have in common, the more work is involved (since we need to find X that exist in zset which don't exist in our set).

Unsurprisingly, the C implementation is faster and is much better at dealing with a large overlap. The Lua version is still plenty fast though. The exact performance difference will depend on what you're doing. Simple queries aren't likely to gain much from being written in C. For a complex query though, we've seen more than a 2x performance improvement (but we still favor Lua in all but extreme cases).