Thoughts on Perl and Emacs, technology and writing

Benchmarking Moose Startup Overhead

I think Moose is fantastic, awesome, intuitive, helpful – except for one thing: its awful, unforgivable speed penalty. When writing GUI or CGI programs, I simply cannot wait eight to twelve seconds for a program to start. Ergo, none of my local library modules use Moose.

8 – 12 seconds for a program to start!? I don’t remember it being that quite bad when I used it.

Yikes, it’s more than 25 times slower! But is it really that bad? It’s adding between 0.3 and 0.35 seconds to the start time. That actually sounds quite reasonable. And when you consider that a typical large perl program will load a whole bunch of other modules which also add to the start time, maybe it will start to get lost in the noise.

Based on this test, rejecting Moose solely on start-up time overhead seems a little unreasonable. (There could be some more overhead if you actually try to use it mind you.)

test2.pl

Just in case I’d used Benchmark incorrectly, I tried something a bit more "traditional" (read hacky).

And I should say that 0.35s is quite noticeable, because a lot of CGI/command line programs finish their main task under 0.1-1s, so the Moose overhead is quite significant. Mouse is at about 0.1s, which is much less worse.

running time on the test that makes a single request against the server resulted in

8.79 real 5.10 user 0.32 sys

This is an application that is deployed and has users so is as close to “real world” as you can get. It is however a long-running daemon so I dont’ really care about the startup time since it happens once or worst case twice a release.

I’m trying to get some measure on the much larger application (~50K sloc) but it’s not working at the moment.

@Zbigniew – okay, firstly, have you got any examples? Secondly, do you know how much of that 8-12 seconds is Moose related and how much is loading other modules and connecting to the database (for example). And finally, yikes, 8-12 seconds? What is Moose doing with the other 7.65 – 11.65 seconds? Building an inheritance graph doesn’t sound *that* expensive.

@Steven – You’re right. I’m making the classic blunder of thinking that everyone else is writing similar code to me. Moose probably isn’t the best choice for a CGI script. Your qmail example is similarly reasonable.

For command-line scripts, I’d argue that a 0.35 second overhead is largely irrelevant. I do a lot of commandline work and most of my (even short) scripts spend at least a couple of seconds connecting to various databases and extracting data. Another 0.35 seconds would not bother me.

I’m not convinced by your stability argument either. Most of my stuff is regularly bounced for regular reasons, but if it is running for more than a minute then 0.35 seconds is not much. How frequently do you restart your daemons?

@Chris – excellent, a realworld example. But wait a minute, are you saying that pressing ctrl-c after starting is a reliable test? Even I am not generally quite that slapdash 😉 And for connecting to a remote server, how much of the overhead is down to Moose?

I have to ask a question. I know ‘no Moose’ basically then shuts Moose off so it can’t interfere with functions and such in things using it. But other than that is there any advantage to no Moose? would we gain performance by not using ‘no Moose’? does ‘namespace::clean’ (etc) have the exact same effect? Moose’s negative impact in startup reminds me of Java, and that startup impact is why I don’t use any Java apps.

I have similar reservations about using Java apps, but I probably shouldn’t have. Most significant apps that I use have a start-up time which is significantly greater that starting the JVM, or starting Perl + Moose due to connecting to databases, loading config into memory or that sort of stuff.

I don’t know if it is faster to omit no Moose but it should be fairly easy to benchmark. My guess would be yes, but not much.

Thanks for following-up 🙂 I am sorry I didn’t respond sooner; I have been going through the throes of a Windows XP -> 7 upgrade. 😐

Yeah, the problem is that simply including the Moose module without having it do anything doesn’t give you (or, I suspect, the maintainers of Moose) an idea of how it behaves when you actually *use* it.

Finally, I ran the program ten times for each object class (on a fairly new, fairly fast workstation with no load). The inside-out object program ran ten times in 11.1 seconds; the Moose object program took 46.2 seconds.

So depending on how you look at it, Moose added 3.5 seconds to each compile, or it tripled the compile time.

And that’s just a toy program. Several weeks ago I started to rewrite a major in-house GUI app in Moose. After I had a couple dozen GUI and database object classes implemented in Moose, the startup time became intolerable. Yes, it literally was 8-12 seconds per startup — that was no exaggeration — and I had only implemented one-half to one-third of the objects that the final app would need.

Moose has the potential to save a lot of programmer time and effort. But honestly, it also has the potential to take away as much as it adds. While I was writing this dopey little test script, I had to fight errors like:

Can you figure out what the problem is from that? I had a hell of a time. It turns out that when Moose attribute triggers fire for the *first* time, when the attribute has no value, the triggers are only passed the *new* value, not the new-and-old values. So the method signature barfs because it was expecting two arguments. Making the second argument optional fixed that. Moose::Manual::Attributes does not mention this.

Moose programs are almost impossible to debug, too, because you can’t step through the generated code.

I would love nothing more than for Moose to work. It looks beautiful. Writing Moose classes is a breeze, a joy, compared to the tedium of writing inside-out classes. But until Moose fixes its horrible startup cost, until Moose has decent error messages, until Moose programs are actually debuggable, Moose is simply a pretty experiment, not suitable for the real world.

And apologies your comment got stuck in unapproved purgatory. Out of the box, wordpress does that with comments with more than two links, which catches a lot of spam. I thought it had some learning capability where if I let previous comments from a user run it would let new ones through too.

Although I believe you (why would you make it up?), it does seem surprising. What is it doing with all that time? I’m going to check your working anyway of course just in case. Because it would be silly to say something without checking. And I obviously have to follow-up again.

> So the method signature barfs because it was expecting two arguments. Making the second argument optional fixed that. Moose::Manual::Attributes does not mention this.

How would you interpret this then (from Moose::Manual::Attributes under “Triggers”):

> The trigger is called after an attribute’s value is set. It is called as a method on the object, and receives the new and old values as its arguments. **If the attribute had not previously been set at all, then only the new value is passed.** This lets you distinguish between the case where the attribute had no value versus when it was “undef”.

big: I don’t see how that would help. 1) making the class immutable speeds up runtime creation of objects, not compile-time. 2) MooseX::Declare automatically makes classes immutable by invoking make_immutable behind the scenes.