about

writing

contact

JSON Benchmarks in jRuby

By eric

2010-10-25

I am in the process of switching a major application from MRI Ruby (specifically 1.8.7-p302) using many C extensions to jRuby (currently trying 1.5.3-master). In my application, performance is extremely important. It is so important in fact, that I will be writing about some of my experiences in troubleshooting the speed and getting those important milliseconds back. When I am trying to keep an entire transaction from start to finish under 40ms and just the decoding of a JSON object into a Ruby object in jRuby takes roughly 30ms using json_pure, we may have to explore other avenues.
Just to verify that JSON was indeed as slow as I thought it was, here is a quick benchmark of it:

That wasn’t so great. We’re looking at about 30ms on average to decode and an additional about 10ms to re-encode. If we want to keep our total transactions to 40ms and we haven’t even done anything with our data, we’re in pretty bad shape. Let’s try the same thing again, only this time with json-jruby.

These benchmarks are by no means exact, but they do give you a feel for the fact that jRuby is slower than MRI with C extensions. The comparison is essentially 3.5ms round trip in jRuby and about 1.5ms in MRI. It might not seem like a lot, but when every millisecond counts, there is a lot that can happen in those extra 2 milliseconds that could be available every transaction.

Note: I reran the code by hand and formatted the results by hand, so the code won’t exactly produce the results above. These benchmarks were all done on my 2008 MacBook Pro with 4G of RAM and an Intel 2.53GHz Core 2 Duo.

UPDATE: It has been pointed out to me that I didn’t take advantage of jRuby’s hotspot feature (Thanks NickSeiger and Charles Remes). Hotspot basically kicks in after it figures out the best ways to optimize the code. So here are some new results and it’s safe to note that json-jruby is so fast, that the computer can’t time it.

I should clarify exactly what I did. I apologize for the confusion. I actually ran this doing 25_000.times {} and 100_000.times {}. The reason it came out so small (I believe) is that the JSON object I used was about 10k rather then n Megs that are usually used for tests.nnTo this end, I decided to try doing 100_000.times { total_time += Benchmark.realtime { JSON.parse(json) } }; avg = total_time / 100_000; puts “Average decode time”;nnThis way the optimizations could take place. I also did it where it wasn’t an average and I could watch the times decrease as the optimizer took effect. The other way I tried things was to do 100_000 iterations prior to the benchmark and then run the benchmark once. This way the optimizer had already kicked in as well. Both yielded the same results of 0.00000ms.

Max

A better way might be instead of running a fixed number of iterations, run for a fixed length of time, i.e. figure out how many iterations you can get through in 10 seconds or something. A little more complicated to write, but probably a better benchmark.

On jRuby, another highly performant approach could be using Jackson (http://jackson.codehaus.org) — I’m pretty sure it is being used, and one link I found was this [https://github.com/guyboertje/jrjackson]