At work, we have an XML API that gets quite a lot of traffic. Last week I looked into improving its performance since we are expecting more traffic soon and want to make sure our response time is optimized.

I run some benchmarks using ab, httperf and jmeter but the results were not that great. I was so sure that switching from rexml to nokogiri would give me awesome results that I was very disappointed.

I was about to call Nick Adams (Nokogiri’s author) to insult him, blame him for _why’s disappearance and tell him that all his pro bono efforts were useless since my own app was not running much faster when switched to his library. As I was about to dial his number on my iPhone I had a crazy thought… maybe it was not Aaron’s fault, maybe it was mine.

So I took a break went to play some fuzzball and as I was being ridiculed by Italian coworker, Emanuele, I realized that most of our API calls were just simple HTTP requests with no XML payload, just some query params. However, we were generating a lot of XML to send back to the client and AS::XmlMini only takes care of the XML parsing, not the rendering.

The XML rendering is done by Jim Weirich’s pure Ruby builder library which is vendored in Rails. Builder does a good job, but I thought that maybe a C based library might improve the speed. A coworker of mine (James Bunch) recommended to look into faster-builder, a drop-in Builder replacement based on libxml. Unfortunately, the project doesn’t seem to be maintained and I decided to look into using Nokogiri XML builder instead. (Also, faster-builder’s author doesn’t like me very much while Aaron knows he’s one of my Ruby heroes so asking for help could be easier)

Some people reported having tried using Nokogiri as a XML builder but didn’t see much speed improvement. Because of the amount of magic required to render a rxml template, I was not really surprised but I decided to contact Aaron and ask him if he tried using his lib instead of builder in a Rails app. Aaron told me he gave it a try a while back and he helped me get my Rails app setup to render xml templates using Nokogiri.

The next step was simple, create a benchmark app and benchmark Builder vs Nokogiri using various templates. Here are the results I got using Ruby 1.9.1 (the Ruby version we use in production) and two sets of templates:

Wow, @tenderlove’s Nokogori just brought us a 4.5X speed improvement for this specific template. 100ms per request is probably not a big deal for most people and I have to say that Jim did a great job with Builder. However in my specific case, 100ms on a request being called thousands of times per hour is quite important.

Which provides you with the url or image tag (or downloaded file) that produces the following graph:

This release works great with Ruby 1.9 and MacRuby, lots of bugs got fixed and some new features were added. Something a lot of people complained was that the gem was called googlecharts but that the main class was called Gchart. The problem was that I wrote my gem and called it Gchart and when I went to register the rubyforge project page, the name was already taken. In this release, I fixed this problem by allowing users to require and use the constant name they want, Gchart or Googlecharts. I also spent quite a lot of time cleaning up the old code which I was a bit ashamed of. Class variables are now removed and overall, the code should be a bit more sane.

This is huge!

While people still try to find some drama an in hypothetical war between rails and merb.

The Rails team and the Merb team announced working together on a joined version of the 2 frameworks. This is so exciting, nobody believed it could ever happen (I personally, had my serious doubt).

Yehuda had a great post laying down the plan and explaining things in details. Check out David’s post explaining why he wants us to work together and his vision of a better Ruby world.

I have to say that I have been impressed by the Rails core team and especially David (DHH).

I’ve known David for few years now and we had long/heated discussions on topics like i18n/l10n for Rails. David is known to be a very opinionated person. But if you come up with the right arguments, he can be convinced and when that happens, he is willing to move forward and shake things up.

This merge is a concrete example that David and the rest of the Rails team cares about Rails and the Ruby community more than we usually give them credit for. As a unified team, we are going to change the way web development in Ruby is done!

But what does it mean for you?

I put together a FAQ video here is the transcript:

Hi, I’m Matt Aimonetti from the merb core team and as you might have heard, a big announcement was made earlier today.

I did this video to hopefully answer the questions you might have.

Q: So what’s the big news?

merb and rails team will work together on the next version of their frameworks

merb 2.0 and rails 3.0 share the same common endpoint

we realized we now have the same objectives and agreed on all the key principles.

merb will provide Rails with agnosticism, modularity, improved performance and a public API.

The end product will be called Rails 3.0 but what real matters is that it’s a huge gain for the entire community.

Q: What??? I thought there was a war going on between Rails and merb, what happened?

There was no war between rails and merb

We created merb because rails was not fitting us

We wanted better performance, more choices/ more modularity and a Public API.

The Rails team now embraces these concepts and want Rails to follow them, so why not work together?

Q: Wait, does that mean that merb is dead?

Absolutely not!

merb development won’t stop, we are going to keep on releasing updates until rails 3.0

clear migration path, and upgrading to rails 3.0 will be as trivial as upgrading from rails 2.x to Rails 3.0

Q: What does the timeline look like?

We just started getting together and discuss the technical details. We are shooting for a release at RailsConf 2009. However it’s hard to estimate this kind of things so again, that’s just an estimate

Q: I just started a merb project, so what now?

I’m sure you had valid reasons to use merb, you needed modularity, performance and a solid API.

Keep on using Merb, we won’t let you down. The good news is that the next version of merb (rails 3.0) will be even awesomer!

Q: What about my client who was looking at using merb for his new project?

If your client is going to be using merb for valid reasons (and not, just because it’s not rails) he should still use merb, but with the full understanding that he/she will end up using Rails in 6 months or so. Again, Rails 3.0 will have what pushed you to use merb.

Q: I’ve been involved with the merb-book, what will happen with this project?

rails 3.0 won’t get released right away

still need awesome documentation

if we look at rails 3.0 as merb 2.0, we can easily imagine how the current work can be extended to the new version.

rails team will also include an evangelism team which I will be part of, so will be able to focus more on projects like the book.

Q: I’ve been working on a merb plugin, what should I do?

Keep working on it! We’ll assist you with the migration process and the new solid API.

Q: What if I still have questions?

Come talk with me, or any members from the new team. We’ll be open to hear your questions, worries, frustrations.

Concretely, nothing changes for Merb users. People loving Merb should not worry. The way you build merb apps won’t change until merb2.0/rails3.0. We will still work on your favorite framework and improve it.

Lori Holden worked on merb_sequel and we should release a 1.0 release of the plugin in a few days.

I’m sure this news comes as a shock for many of you, but try to not see Rails 3.0 as the way Rails is right now. Imagine a version of Rails with true modularity and agnosticism (sequel, DM and AR will still be supported) and the same type of performance as what you get with Merb. In other words, the rails world is about to discover the power of merb!

Here is what Yehuda explicitly says in his blog post:

Rails will become more modular, starting with a rails-core, and including the ability to opt in or out of specific components. [...]

We will port all of our performance improvements into Rails. This includes architectural decisions that are big performance wins.[..]

As of Rails 3, Rails will have a defined public API with a test suite for that API. [..]

Rails will still ship with the “stack” version as the default (just as Merb does since 1.0), but the goal is to make it easy to do with Rails what people do with Merb today.

Rails will be modified to more easily support DataMapper or Sequel as first-class ORMs. [..]

Rails will continue their recent embrace of Rack, which [..] will improve the state of modular, sharable logic between applications.

In general, we will take a look at features in Merb that are not in Rails (the most obvious example is the more robust router) and find a way to bring them into Rails.

Personal perspective

I’m personally really excited about this opportunity. I had a hard time believing that we could work together but I was proved wrong. We have many challenges in front of us, but watching the two teams working together is very reassuring.

I’m also glad to see that we will have a Rails Evangelism team that I will be part of. I strongly believe that one of the many reasons why merb has been so successful is because we work and listen to our community. We have put a tremendous amount of energy trying to understand what you guys need and what you like and dislike. In return, we saw many people working hard on the wiki and the merb-book.

Can you imagine doing that with almost 4 Million Rails developers?

I’m also looking forward to working with a team and reaching to even more people.