The Ruby ecosystem is constantly evolving. There have been many changes in the engineering world since our comparison of Ruby frameworks in 2014. During the two years, we received a few requests from fellow engineers asking for an updated benchmark. So, here is the 2016 edition with more things tested.

What’s under evaluation?

Of course, there are a lot of performance benchmarks on the web. For instance, this one is very good and compares different frameworks while they operate in a viable production configuration.

However, I wanted to understand how Ruby frameworks behave when used as basic solutions with default settings. The idea was to measure only framework speed, not the performance of the whole setup. All my tests were extremely simple, which allowed me to compare almost anything and to avoid side effects caused by very specific optimizations.

So, I implemented bare minimum testing samples for this benchmark and evaluated the performance of:

Ruby frameworks (compared with Ruby)

Ruby template engines

Rack application servers

Ruby ORM tools

other languages and frameworks

All of the frameworks and tools were tested in the production mode and with disabled logging. The performance of the technologies was measured while they were executing the following tasks:

Languages: Print a “Hello, World!” message.

Frameworks: Generate a “Hello, World!” web page.

Template engines: Render a template with one variable.

Application servers: Run successively five simple apps that carry out one action each, such as using database records, showing environment variables, or just sleeping for a second.

ORMs: Do different database operations—for example, loading all records with associations, selecting a fragment of a saved JSON object, and updating JSON.

Performance of Ruby frameworks

For comparing Ruby frameworks, I chose the ones that are popular enough and quite actively developed: Grape, Hanami, Padrino, Rack, Ruby on Rails, and Sinatra. If you have been following my series of benchmarks, you might have noticed the absence of Goliath on the list. Sadly, it has been more than two years since its latest release, so I did not include the framework into the tests. For all Rack-based frameworks, Puma was used as an application server.

Benchmarking tool. I began my study from selecting a testing tool, and the search was not in vain—I found GoBench. Really, I liked this tool better than ApacheBench, which hasn’t been updated for 10 years. Wrk is also good for local testing. There are more benchmarking tools available, which even allow you to create a cluster.

To get started with GoBench, I ran the gobench -t 10 -u http://127.0.0.1 command and set the concurrency level to 100 (the -c parameter).

Testing samples. In addition, I prepared a number of “Hello, World!” web applications and used them to measure the number of HTTP requests that the technologies under comparison could serve per second.

Performance results. As you might expect, Rack and Sinatra demonstrated the best results.

A simple, single-thread Ruby server was a bit faster than Rails 5 in API mode, while a modern Rails killer—Hanami—got just a bit better results. In my previous benchmark, Sinatra was three times faster than Rails, and now it is seven times faster. Good progress, Sinatra!

Performance of Ruby template engines

The list of the tested template engines includes ERB, Erubis, Haml, Liquid, Mustache, and Slim. Here is the Ruby script I used to measure their performance—the number of templates that can be compiled per second. The image below sums up the results of testing the template engines speed.

Erubis had the best performance among the tested solutions. Slim, as always, was the most compact, and it was also faster than ERB.

Performance of Rack application servers

Among the Rack application servers included in the benchmark are Phusion Passenger, Puma, Rhebok, Thin, and Unicorn. Similar to the frameworks, I measured the number of HTTP requests served per second when each of the servers was used. Here, you can find the source code of the tests.

As you can see in the image below, the fastest Rack application server was Passenger, and Rhebok came really close to it.

Puma, which was recently made the default server in Rails, won just one test. Compared to my previous benchmark, Unicorn has done a good job and now demonstrates results very similar to Thin.

Performance of ORMs

In this part of the benchmark, I focused on ORMs and tested how much time Active Record, Sequel, and Mongoid needed to process a given request.

Seven years ago, Jeremy Evans created an amazing benchmarking tool, simple_orm_benchmark. I have updated it and tested Active Record and Sequel with MySQL, PostgreSQL, and SQLite, as well as Mongoid with MongoDB. The six new tests I added to the existing 34 are related to the feature common for all modern databases—the support of JSON types.

The versions that I used in the benchmark include MySQL 5.7.15, PostgreSQL 9.5.4, SQLite 3.14.2, MongoDB 3.2.9, Active Record 5, Sequel 4.38, Mongoid 6, and Ruby 2.3.1.

Here, I include the most interesting results of the tests (fewer seconds means better performance). Red-colored rows in the images below represent measurements with transactions while blue-colored—without transactions. You can find all the performance results for the Ruby ORMs in this CSV file.

Time needed to perform the test with selecting records by a JSON fragment:

Despite the fact that Active Record announced the support of JSON types, it is not quite true. In reality, the support only means creating a new JSON object painlessly and getting it back from the database. If you want to update the object in the database, it will override the whole object. To update a part of the object or select the object using a part of it, you have to use raw SQL, which is different for every database.

Moreover, for having the support of JSON fields in SQLite, you should compile SQLite with the json1 module enabled via a SQL query. Alternatively, if you do not need Active Record, you can simply use the Amalgalite gem, because the activerecord-amalgalite-adapter gem is obsolete.

Operations

Mongoid

Active Record

Sequel

Inserting JSON

Yes

Yes

Yes

Reading JSON

Yes

Yes

Yes

Supporting JSON types in models

Yes

Yes (except SQLite)

Yes (only for PostgreSQL)

Updating a JSON object partially

Yes

Raw SQL

Raw SQL

Searching by JSON fragments

Yes

Raw SQL

Raw SQL

In general, Sequel was faster than Active Record and Mongoid, but I cannot say that I enjoyed using it. I really liked working with Mongoid: everything was intuitively clear, and I spent less time implementing the tests compared to Active Record and Sequel.

Performance of other languages and frameworks

In the final part of the benchmark, I compared Ruby frameworks with a number of other languages and frameworks, such as Crystal, Python, Elixir, Go, Java, Express, Meteor, Django, Phoenix, and Spring.

Test environment. Similar to the evaluation of Ruby frameworks described earlier, GoBench was used as a testing tool, and the behavior of the frameworks was maximally close to the default. See these “Hello, World” web apps for the testing samples.

Performance results. Here is the surprising part of this benchmark.

You can see that the Go language and a Go framework Iris are absolute winners that manage to handle about 60,000 requests per second. They are followed by Ring (Clojure), Scalatra (Scala), and Warp (Haskell). Elixir, Scala, and Crystal (a Ruby-like language inspired by the speed of C) share the third place. They are followed by Spring (Java), Hyper (Rust), C, Phoenix (Elixir), Ur/Web, Dart, and Node.js.

Conclusion

The frameworks and tools in this benchmark were configured as close as possible to their default behavior, which essentially allowed me to create equal conditions for all of them.

Even though Rack, Sinatra, Padrino, Grape, Rails, and Hanami demonstrated relatively low performance compared to other frameworks in our “Hello, World” benchmark, it does not mean that Ruby is the wrong choice. In real life—when you scale your app horizontally, use multiple workers and threads, and know all the advantages of a framework—the results may be very different. It is also true that your preferences and needs may change at some point.

But, first of all, you should enjoy your language. If you don’t, just try something else.

Find the full version of this study (16 pages, 17 graphs) in this document.

About the author

Eugene Melnikov is a senior software engineer at Altoros. He mainly specializes in Ruby and Ruby-based frameworks, as well as in JavaScript development, including Node.js, jQuery, and AngularJS. Working at Altoros, Eugene has also designed and implemented a variety of SQL and NoSQL database solutions. He recently became engaged in creating data-driven software for IoT applications. Check out Eugene’s GitHub profile.

For the next parts of this series, subscribe to our blog or follow @altoros.

The last benchmark does not make any sense. You are comparing languages and frameworks with different properties. For example, for PHP you are using web server which should be used only for development (why not php-fpm?), nobody uses it for production. Rails you are running in production environment (why not development then?). Go program by default would use all available cores to run the server (since go1.5), and gobench, the tool you are using to run benchmark, uses 100 concurrent request (you haven’t even mentioned that).

I think your benchmark is very shallow and results do not worth trust.

Eugene Melnikov

Hi Sergey

> Isn’t Mongoid using BSON? ActiveRecord cannot work with BSON. ActiveRecord uses Ruby to decode JSON and to encode its own implementation.

> I think your benchmark is very shallow and results do not worth trust. I think any comparison has its own assumptions. If you run more realistic cases, you will get completely different results, because every language and framework have different implementations, behave differently under a heavy load, and scale differently with a process manager or in a cluster. My scenario is as simple as possible, which makes the comparison very synthetic. But I selected this scenario, because I was curious what results I could get without using optimizations that are specific to a particular language or framework (I just disabled logging and enabled production mode for the frameworks). Plus, it was easy to prepare so many examples by implementing such simple applications.

> why not php-fpm? I didn’t use php-fpm because I assume it as a kind of optimization.

> gobench, the tool you are using to run benchmark, uses 100 concurrent request (you haven’t even mentioned that) mentioned)

>> why not php-fpm? > I didn’t use php-fpm because I assume it as a kind of optimization. In text you are saying “All of the frameworks and tools were tested in the production mode”, what makes you think that php -S is production mode?

>> gobench, the tool you are using to run benchmark, uses 100 concurrent request (you haven’t even mentioned that) >mentioned) but why not writing the code to be ready to handle 100 concurrent requests? Your java version runs unbounded number of threads and not using NIO or thread pooling. The C version does not even use pthreads. Is it production mode? I thought that on linux using epoll is more wide-spread in production. Ruby has Kernel#select to deal with asynchronous IO. Why did you choose to use pure synchronous IO in some cases, and threaded or even threaded asynchronous in the others? NIO/epoll/Kernel.select all of them accessible out of the box without external dependencies, but you chose not to consider them production enough?

Eugene Melnikov

Now I see what you mean by “production” mode. Meaning that I put into this word is more narrow. Some frameworks have predefined configurations for production, development and test environment, so I used production.

As for using threads. It was hard to choose between simplicity and performance. I did prefer simplicity for most cases, but for some languages the most simple example on the first page of documentation does support threads. From my point of view if documentation recommend to use threads from first page it’s not an optimization, it’s just basic way to use language.

I understand that you think it’s unfair and will think about it. May be in the next version of this benchmark all examples will support threads.

If you cannot choose between simplicity and performance, why do you put everything on the same chart? You have to render two charts then: “Simple solution, you will never see in the wild” and “Proper solution, which is complex, but similar to production case”. And you will see that almost all your tools/languages/libraries will go into first bucket.

> The frameworks and tools in this benchmark were configured > as close as possible to their default behavior, which essentially > allowed me to create equal conditions for all of them.

equal what? the number of lines? or number of commands in the terminal? Why does it matter at all?

You call results of you “benchmark” surprising, they are not surprising at all, you are comparing apples to oranges. The chart and “study” does not mean or worth anything.

mega_tux

I found Hanami results somewhat unexpected. I’ll try to test it here. Regards

Eugene Melnikov

I also expected to see Hanami more close to Rails

mega_tux

can you update the results with the above suggestions from the Hanami maintainer?

Eugene Melnikov

I got 1091 hits/sec on the same environment with Luca’s suggestions. I will update images a bit later. Thanks a lot for comments.

You can run using your own versions. Just clone https://github.com/melnikaite/app-server-arena and execute setup.rb then run.rb

nori3tsu

Thank you, I’ll do that. But I want to hear that Passenger used by this benchmark is the enterprise edition? The reason is whether Passenger’s evaluation in this benchmark will change depending on whether it is the enterprise version or not. There is a difference in the thread model between the enterprise edition and the community edition.

Eugene Melnikov

I used community edition, but it’d be interesting to see your comparison results with both editions.

Hi there, I’m the author of Hanami. There are two issues with the Hanami benchmark:

1. It’s fundamental to generate the assets manifest. Without it, I get all the requests as not successful. To fix it `HANAMI_ENV=production bundle exec hanami assets precompile` 2. The server is started with code reloading. Please note that `hanami server` is for development only. For production, I suggest to use directly servers commands (eg. `puma`). To fix this `HANAMI_ENV=production bundle exec hanami server –no-code-reloading`

The difference in results is huge. It goes from 59 req/s to 1039 req/s.

Subscribe to new posts

Subscribe to new posts

Get new posts right in your inbox!

Follow us

About Altoros

Altoros is a 300+ people strong consultancy that helps Global 2000 organizations with a methodology, training, technology building blocks, and end-to-end solution development. The company turns cloud-native app development, customer analytics, blockchain, and AI into products with a sustainable competitive advantage. Altoros assists enterprises on their way to digital transformation, standing behind some of the world's largest Cloud Foundry deployments.