16:57 sobel: i can't find it now, but arstechnica benched all the popular dynamic languages
16:58 sobel: using C/C++ as the standard (1.0) they ranked other languages as multiples of C/C++ performance
16:58 sobel: java was a 2
16:58 sobel: twice as slow as C. and it was the winner, head and shoulders above the rest
16:58 sobel: i think Erlang placed somewhat favorably
16:59 sobel: python/perl were near the middle, at something like 25-35x slower than C
16:59 sobel: Ruby: a full 72x slower than C

Ruby loses against invisible Ars Technica benchmarks in the sky with unknown URLs! 72x slower than C! Over twice as slow as Python and Perl!

Fortunately, we don't have to rely on some one-off benchmark Ars Technica may or may not have done at some indeterminate point in time whose URL cannot be located offhand, because there's a site with a fairly decent reputation which provides ongoing benchmarks across multiple programming languages using implementations submitted by fans of said language. It's been around for awhile and is relatively well-trusted.

According to this benchmark suite, JRuby is 34.5 times slower than (not C, not C++, but) Fortran. Ruby 1.9 (MRI/YARV) is 43.80 times slower than (not C, not C++, but) Fortran. Both JRuby and Ruby 1.9 beat Python, PHP, and Perl by a considerable margin. The nearest competitor is Python 3, at 47.93 times slower than (not C, not C++, but) Fortran. By the way, did I mention that the fastest language on their benchmark is... not C... not C++, but Fortran? (nothing personal sobel, but unsubstantiated hearsay is bad!)

Yes, that's right folks: according to the Programming Languages Shootout, Python, PHP, and Perl are all slower than Ruby. Did you think Ruby was slower than Python? Guess what, you're wrong. Ruby used to be one of the slowest popular languages, but that has changed. Ruby performance has advanced considerably over the years, so if you're still repeating some offhand information you may or may not have gotten from Ars Technica at some point but can't find the link to as your metric of Ruby performance, you may want to try again, and find modern, relevant information you can actually get a link to.

There are many future VM improvements in the pipe for Ruby, Python, and PHP (and I guess Perl users might continue dreaming of a Parrot-powered Perl 6). Rubyists can look forward to the upcoming JRuby 1.7 release which features InvokeDynamic support and allows for Java-speed method dispatch... in Ruby. InvokeDynamic is a game changer for the JVM in general, and it's a game changer for Ruby, because InvokeDynamic makes JRuby dispatch potentially as fast as Java.

Python users can look forward to PyPy, which is posting some incredibly impressive numbers, especially around numerical computing. Python users can also look forward to resumed work on Jython, which is adding InvokeDynamic support which can potentially make Python as fast as Java. Finally, PHP users can look forward to the HipHop VM developed at Facebook, which will provide much improved performance for PHP. These are all great projects, but none of them are really ready for general consumption yet (including JRuby 1.7).

All that said, the Programming Language Shootout doesn't include any of these unreleased development versions in the benchmarks you see when you visit their site. They show the numbers for the latest production releases, and those numbers show Ruby is faster than Python, PHP, and Perl.

The game has changed: you just weren't paying attention.

Last but not least, if you've seen some benchmark somewhere, even if you have an eidetic memory and remember but the numbers were, but can't even dredge up a link to it, please, please, don't quote said benchmark, even if you have an eidetic memory and remember what the numbers were.

For benchmarks to be remotely scientific, they must be both reproducible and falsifiable, and hopefully in addition to both those things they have a good methodology. If you can't even dredge up a link to the benchmark in question, please don't go quoting numbers off the top of your head to people who might be influenced by them.

Let's advance computer science beyond the state of witch doctors telling people to bleed themselves with leeches because at some point someone said they might make you feel better maybe.

38 comments:

Comparing Python to Ruby directly, we see this chart: Python vs. Ruby. You can see that they are on the same order of magnitude, and to say that one is faster than the other is an illusion. They operate at the same speed when averaged over many different types of tasks.

When Ruby is slow it's over 5X slower than it's median time. So the critical question would seem to be, how representative is the median timing of this test of real world use? C++ is always fast. PHP, at it's slowest, is only 2X slower than it's median timings. If the real world performance tends more toward the slower end then Ruby is slow.

Think it's a little unfair that the x86 architecture was cherry-picked of the four results sets to prove your point :/ on x64 quad-core, it was actually the slowest of all.

That's nit-picking, though, in general my opinion on this piece is that amongst the dynamic languages, these results mostly make the point that they all perform pretty similarly, and picking between them should probably be a choice more of team preference and library support for the target tasks. If performance is of the most supreme importance, the right choice between all those listed is none of them :)

I can't comment on the other languages' shootout implementations, but last time I checked some of the Ruby ones were astonishingly inept. I honestly wouldn't trust the aggregate rankings of the Shootout at all.

I mean, I don't even know where to start in figuring out if this is efficiently implemented, but just look at it!

Also, the shootout scripts were overhauled at some point. They used to be more general, and now they're much more focused on computation. That's part of why Fortran (which is optimized for mathematics) gets first place.

I do agree that the conventional wisdom that Ruby is the slowest of the 'big four' (Perl, Python, PHP, Ruby) is unjustified.

It's very important to acknowledge that program performance is likely to different when we run the same programs on x64 rather than x86, and when we have 4 cores available rather than one core, and ...

@Isaac Gouy I was referring to the fact that the style is a mess and the author refers to himself as a "Ruby novice". I find the Python version of that test easier to read, and I've been using Ruby almost 10 years longer than Python.

If that's your code and I hurt your feelings, I sincerely apologize.

And yes, the shootout was formerly more generalized. I recall many more micro-benchmarks on things like object instantiation, recursion, function calls, data structures, etc.

"After all, facts are facts, and although we may quote one to another with a chuckle the words of the Wise Statesman, 'Lies--damned lies--and statistics,' still there are some easy figures the simplest must understand, and the astutest cannot wriggle out of."

While Ruby 1.9 got increased performance as seen in the benchmarks, I was more interested in the Lua, Go and C# numbers. Is sad C# haven't got much attention in the open source (outside Windows) world while having numbers like those.

Yes, Ruby is pretty fast, considering how dynamic and powerful it is, but it may just be too powerful.From what I've seen, Ruby developers very often tend to value inner library code beauty and nice outer API design over performance.Look at Cucumber for example, that is a two layers of DSL over regular Ruby code.People actually construct damn fat classes in runtime based on results of introspection ( that is slow ), nifty DSL's ( they are slow too ) and, of course, then they use string manipulation to build code, and later eval it.Also, things like long inheritance trees and breaking up code in modules make method lookup more complex, and if that is not bad enough, people use method_missing a lot, and blocks are everywhere.In the end, this gives us just insane Rails startup times, and people go around "Bundler so slow!". So while Ruby can be fast ( if you pay a lot of attention to optimizing the code ), most Ruby libraries are slow.And you will probably end up using them.

More seriously, anton describes a far more realistic approach. I'd bet that well over 90% of real-world web apps would operate acceptably irrespective of the chosen language.

Web App performance bottlenecks ordinarily don't become significant unless the app is scaling to large transactional volumes, or the developers did some seriously faulty designs. In my experience, faulty design is highly portable. :-)

I'm a project manager, so on a certain level I think of developers and servers as costs. When the decision is about to hire Java developers for 2X in money or PHP/Ruby devs for X and if I can buy a HP DL360G7 with 2 CPUs (that is 8 cores at least, but can be 16) from 10 days of a PHP/Ruby developer's cost, what will I choose if the project has about 80 developer days? The PHP/Ruby one. When slow means "it cannot handle too many requests" then I don't care an inch if it is relatively slow. I'll just buy more servers.

3-5 years later - when I trash the servers - the next ones I buy will have ~24 cores, consume less power, and cost the same.

Well said Tony! I get so sick of people belching numbers and crowing. I much prefer the evidence, even if it didn't say Ruby was fast.

Sometimes it's like "My Dad would beat your Dad in a fight any day!" playground mentality.

In any case, I don't use my language of choice because it's fast, but because it gets the specific job I want done, done, and does it well. The fact that it is a pleasure to write (and think) in is a BIG bonus though.