This blog will be covering my attendance at the QCon 2011 conference in San Francisco, California from November 16 to 18. QCon is a software development conference on many useful topics which I will try my best to summarize through my posts. Please, feel free to post your comments and get some conversation started. Enjoy!

Wednesday, November 16, 2011

JVM performance optimizations at Twitter's scale

Complete presentation about application optimization based on twitter experiments; it was Java-centric, but I tried to generalize the presented principles.

and for those interested... I still do not like Java!

Takeaway: Twitter went from 60 second of Garbage Collection (GC) per hour to 5 seconds per 3.5 days, mainly because of the internal slab allocator.

Twitter's biggest enemy: LatencyWhat are the main latency contributors (from worst to less):

Garbage collector (GC)

in process locking and thread scheduling

I/O

application inefficient algorithms

but only the first one can be addressed by JVM tuning (which happens to be the expertise of the speaker)

Areas of performance tuning:

Memory tuning (footprint, allocation rate, garbage collector rate)

Lock contention

Memory footprint tuning

Maybe you just use too much data!

Maybe your data representation is fat! (bad data model)

Maybe you have a memory leak (not covered in remaining presentation)

Global advice for performance tuning:

Profile your applications, especially when 3rd parties are involved abd not well known (ex: primitive SCALA array types eating lots of memory; another ex: Guava's concurrent map eating a lot of resource - especially if you do not need concurrency).

Even more important advice:

Writing a new memory manager is not a good idea unless you already optimized your data and your JVM... if you start doing this, it is a strong smell that you are doing something wrong!

Fighting latency

Performance tradeoff: Memory vs Time (convenient and valid, but oversimplified view); the performance triangle (press memory usage down, push throughput up and keep latency low) is a better model, but still too simple for Twitter optimization team. In the end they use a (optimization increase) = C x T x R where C=compactness (good data model), T= throughput and R=responsiveness.

The rest of the presentation was about tuning and optimizing Java JVM Garbage Collector (GC) - if you are interested, let me know (I will not document all of this in this post - lucky you!).

Side note:

At Twitter, they seem to also have a big gap between developer's "done" and "production ready" done.