Apple integrates LLVM compiler to boost WebKit JavaScript performance

Since compilers already make fast code, why not take advantage?

We looked last week at some of the things that can influence the speed of programming languages and how different approaches to running code can influence program performance. Some work described by Apple this week shows how techniques common in one language can be applied to another.

Modern browsers use a scheme called just-in-time compilation to convert JavaScript into executable code and achieve high performance when they do so. As the optimization process has become more advanced and performance improved, Apple's WebKit developers have realized that they're increasingly building the same capabilities as traditional compiler developers have already built. Feeling that's rather wasteful, they've decided to stop building their own and instead leverage the optimization capabilities of the LLVM compiler.

The work was described earlier this week in a blog post. The WebKit JavaScript engine previously had three different stages that achieved different performance levels for the scripts they ran. The first stage was an interpreter that ran script programs without compiling them. This allows scripts to run "instantly" without having to wait for a compiler.

If a JavaScript function is called more than six times, or loops more than 100 times, it gets fed into the next stage: a basic JIT compiler that produces executable code. This doesn't optimize heavily, but it removes some of the overheads that the interpreter has to produce code in the ballpark of ten times faster than the interpreter.

If the function is then called more than 66 times, or loops more than 1,000 times, it gets fed into the third stage, a JIT compiler that actually spends a relatively long time to optimize the code. The third stage output is about three times faster than the basic JIT, or 30 times faster than the interpreter. It would take too long to optimize every function with this compiler, which is why this staged approach is used.

Though that third stage is much faster than the interpreter, it still doesn't match a typical C or C++ compiler for low-level optimizations: figuring out which values to store into which registers, for example. The WebKit developers wanted to add a fourth stage to perform this kind of work, but while they thought that writing it from scratch would be "super fun," it'd also be a lot of work. A lot of work that existing C and C++ compilers already do. Hence, they decided to investigate the use of LLVM's code generation and optimization.

The LLVM-based fourth tier is called FTL, for Fourth Tier LLVM (and, of course, faster than light). It shares some portions with the third stage, since the third stage already does important work for handling JavaScript's dynamic nature but has a different code generating portion.

The result is a healthy performance boost. FTL produces code that is more than 40 times faster than the interpreter, with benchmarks taking about a third less time to run than the old three-tier system.

One of the downsides of the LLVM compiler is that it's, relatively speaking, quite slow. This isn't altogether surprising, as LLVM is designed to optimize well, rather than optimize quickly, and for ahead-of-time compilation of the kind used for C and C++ it's worth taking the extra time to do a good job. To handle that, WebKit only uses FTL for functions that are taking a long time to run (10 milliseconds) or being called an awful lot—to ensure that it's really worth using FTL—and then it runs FTL on a background thread. It doesn't have to wait for that thread to finish, as it can simply use the code from the third stage in the meantime.

FTL is currently available in WebKit nightly builds and enabled by default.

A decade ago, JavaScript was only ever an interpreted language, and it was quite slow because of it. Since then, it has risen in importance and become the focus of considerable development effort to improve its performance. The use of LLVM and the integration of a traditional compiler's optimizer show how features common to one set of languages can be used in others, making the distinctions between different kinds of language fuzzy.

It's unlikely that Apple will be the only browser developer to do this kind of thing. Browser developers like to crow about the performance of their JavaScript engines, and the likes of Microsoft and Google won't want to let Apple take the lead in this field.

153 Reader Comments

Wow quite clever how they handle the insane dynamic polymorphic nature by fusing profiled data from the previous optimization JIT into the LLVM and creating a cache of NOP statements at the ready. Quite a bit of thought has gone into this.

Wow quite clever how they handle the insane dynamic polymorphic nature by fusing profiled data from the previous optimization JIT into the LLVM and creating a cache of NOP statements at the ready. Quite a bit of thought has gone into this.

Good show apple

That's a commonly used JIT technique for well over a decade, nothing revolutionary there really.

[With GCC, they're very worried about non-free projects being able to leverage their work without contributing back. So, they made it hard for things besides GCC to leverage GCC's guts. The internal representations and the different phases are not meant to be used by other tools.

We don't want to make a program's entire AST available for parsingbecause that would make it easy to extend GCC with proprietaryprograms. If we had allowed this, we would have brought on the sameproblem that LLVM is causing: GCC would then change from a freecompiler into a platform for nonfree compilers.

This is explicitly one of the design considerations behind GCC. They do not pretend otherwise.

And hilariously, RMS calls it a "problem" that people are using LLVM to do useful and interesting things.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Ok, then help me design a compiler that can get around the following conundrum: As we all know operating on IEE-754 denormals is exceedingly expensive on the vast majority of all architectures (can't think of a single one where we don't get a performance hit for it actually, 2 orders of magnitudes aren't unheard of).

The JLS requires that FP operations still correctly return denormalized numbers. C++ standard on the other hand doesn't make any such requirements, which means it's allowed to flush denormal values to zero and get the nice performance benefit.

According to Agner Fog, since Intel's Ivy Bridge and AMD's Steamroller, there's no penalty for denormals (on SSE/AVX at least).

If you really want, you can break the Java standard a bit and add an option to your JVM to flush denormals to zero.

Interesting, he actually has the same for SB already. Surprising that both Intel and AMD consider the additional die space to handle denormals efficiently worth it.

Anyhow the point is that a language can certainly demand requirements that can cause slowdowns that cannot be optimized away by the compiler. So while a language may not be slow or fast, it can certainly demand things that have a performance overhead.

That's the point of the web. If your js requires JIT compilation and optimization with LLVM, you are doing it wrong.

Thanks for the 1998 perspective on things, but people expect more from the web here in the 21st century.

Oh yeah, you are right, in 2014, people expect their web browser to run Office suites and high end 3D games in full screen. Wait, I just checked the web and in 2014, web Office suites on the web still suck and nobody use them still and web games are still tic-tac-toes. Some quick projection tells me that in 2020 Office suites will still suck and that they are doing it wrong.Please point me to a rich and computation intensive application that run on the web AND don't suck in 2014.What you will find is demos that run on the latest chrome nightly build and are projected to be the future of the web.In 2050 it will be the same. The web applications will run smoother and smoother but application using frameworks designed to run applications will always perform better.

1) It would be nice if the programmer and user had some control over which scripts are optimized when. As a user I'd like to be able to tell my browser "go ahead and fully optimize every bit of JavaScript you see and cache the result indefinitely" because storage is super cheap for me and I'd only be paying the compiling cost once per update to each script. As a programmer I'd like to inform the interpreter that based on my profiling I've determined that part or all of my code should be optimized right away or not at all.

2) Languages sure do bring out the hate. Haters need to relax. If you find some code incomprehensible, blame the programmer, not the language. If you can understand the code but just don't like what it looks like, that's on you. If someone writes bad English that's not the language's fault either, nor is it English's fault if you don't like how it looks or sounds. There's nothing intrinsically better about "(cond (test) (true_action) (false_action))" or "if (test()) { true_action(); } else { false_action(); }" or "if test then true_action else false_action". Languages are tools and can be better or worse than each other at solving a problem but outside the context of that problem judging them is a matter of taste. Obviously intentionally difficult languages like Unlambda and Brainfuck don't count. Humans are very creative and can always make things more difficult for themselves if they want. But any language with the level of popularity of Fortran, Pascal, Perl, Python, Ruby, JavaScript, C#, or C/C++ is clearly The Right Tool for a lot of jobs. Lisp, Forth and Haskel ... maybe not so much? I don't know where to draw the line.

3) For my use cases, Google web apps are dramatically superior to their native equivalents (MS Office, LibreOffice, etc). Don't assume everyone works the same way you do and has the same preferences. For that matter, my wife was writing her novel in Google docs until it got too big and then she switched to Scrivener - which is a tool specifically designed for authors.

Heh, common theme for all three points: I like options, I like control and I don't like people giving each other instructions about what to think or how to solve their problems.

Interesting, he actually has the same for SB already. Surprising that both Intel and AMD consider the additional die space to handle denormals efficiently worth it.

SB handled denormal addition quickly but has a large penalty for denormal multiplication.

Most x86 applications do not run with FDTZ/DAZ. The problem is that, on x86, it's controlled by a CPU flag, which is a per-thread state.Compilers don't want to sprinkle their output with the necessary save/set/restore instructions.And a program may just call a library that needs proper denormal support.

So, in the end, the use case is that you either run your entire program with denormals (common case) or you run your entire programs with FDTZ/DAZ.

Quote:

Anyhow the point is that a language can certainly demand requirements that can cause slowdowns that cannot be optimized away by the compiler. So while a language may not be slow or fast, it can certainly demand things that have a performance overhead.

Yep. If you don't need full IEEE 754 compliance, you can optimize a number of things.That said, note the C99 and C++11 also demands the same compliance as Java. It's just that C/C++ compilers pay variable amount of respect to it.

[With GCC, they're very worried about non-free projects being able to leverage their work without contributing back. So, they made it hard for things besides GCC to leverage GCC's guts. The internal representations and the different phases are not meant to be used by other tools.

We don't want to make a program's entire AST available for parsingbecause that would make it easy to extend GCC with proprietaryprograms. If we had allowed this, we would have brought on the sameproblem that LLVM is causing: GCC would then change from a freecompiler into a platform for nonfree compilers.

This is explicitly one of the design considerations behind GCC. They do not pretend otherwise.

And hilariously, RMS calls it a "problem" that people are using LLVM to do useful and interesting things.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

But deliberately designing the product so that even the users you approve of can't do certain things at all, just because it might make things slightly easier for the folks you don't approve of... wait a minute, you're right, they're exactly the same.

Interesting, he actually has the same for SB already. Surprising that both Intel and AMD consider the additional die space to handle denormals efficiently worth it.

SB handled denormal addition quickly but has a large penalty for denormal multiplication.

Most x86 applications do not run with FDTZ/DAZ. The problem is that, on x86, it's controlled by a CPU flag, which is a per-thread state.Compilers don't want to sprinkle their output with the necessary save/set/restore instructions.And a program may just call a library that needs proper denormal support.

So, in the end, the use case is that you either run your entire program with denormals (common case) or you run your entire programs with FDTZ/DAZ.

Cannot say I'm much involved with numerical code these days, if I'd known that the latest Intel/AMD CPUs didn't have a large penalty for denormals any more I'd have picked another example.. a language that demands sequential consistency for multi-threaded code in all cases or Knuth knows what ;-)

I'm sure you're absolutely right with your assumptions, although I personally think the way that C/C++ handle this in practice is perfectly alright: The compiler leaves the flags alone and sets it to the default to follow IEEE-754, but if the programmer has benchmarked their application and thought the whole thing through they should just be able to call fesetenv and are then responsible for worrying about other libraries. In practice for 99% of all programs denormals aren't really something to worry about after all.

Yep. If you don't need full IEEE 754 compliance, you can optimize a number of things.That said, note the C99 and C++11 also demands the same compliance as Java. It's just that C/C++ compilers pay variable amount of respect to it.

My C++11 standard says on a cursory glance that it's not necessary to fully support IEE-754 - there's a flag though that tells a program whether it does or not. Although without fastmath options, etc. C compilers seem pretty tame these days in that regard.

Can't test Safari on this machine as it's only available on OSX.Judging from http://www.arewefastyet.com/ the numbers would be nothing to write home about, unless LLVM-JS-optimization hasn't found its way into Safari yet.

It hasn't. This is very new stuff. If you want it, I believe you need to build from upstream sources. I'm not sure it'll make it into Safari (or JavaScriptCore) before 10.10.

AreWeFastYet (AWFY) does not use Safari builds, it uses builds from specific changelists on http://trac.webkit.org of a stand-alone version of JavaScriptCore (e.g. the first build of JavaScriptCore on AWFY is between changelist 68285 and 168377, and the latest is 168970 -- the "rev" property when you hover over the dots). If you click on a dot, it will provide you a link to the changelist.

Okay, then the next question is whether it's enabled by the default build process, and what configuration settings AWFY uses. The next question is whether it's enabled by default at runtime, and how AWFY's runtime tests are configured. And I can go look into that stuff in a bit.

(Right now, I'm cloning the WebKit git repo onto one of my dev machines, so I can begin to play with this stuff. After that, I should know what to actually look for at AWFY.)

Interesting, he actually has the same for SB already. Surprising that both Intel and AMD consider the additional die space to handle denormals efficiently worth it.

I'm sure you're absolutely right with your assumptions, although I personally think the way that C/C++ handle this in practice is perfectly alright: The compiler leaves the flags alone and sets it to the default to follow IEEE-754, but if the programmer has benchmarked their application and thought the whole thing through they should just be able to call fesetenv and are then responsible for worrying about other libraries. In practice for 99% of all programs denormals aren't really something to worry about after all.

I'm not complaining about how it's done either.If most of my stuff, 32 bit floats have way too much precision already.

I was just offering a rational why AMD and Intel, in particular, has put so much effort into supporting denormals.

That said, it's still an incomplete rationale. Intel has always pushed very hard.While AMD's 3d!Now simply lacked denormals, PowerPC Altivec disables them by default, ARM's NEON flushes them to zero, Intel made SSE fully IEEE-754 compatible from the start.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Adobe doesn't claim itself to be a bastion of freedom. So no, not the same mindset at all. And RMS's mentality hurts even other open source, GPLed developers.

I actually do think that the person you're responding to has a valid point here.

When RMS talks about freedom, he means something pretty specific, and he's pretty upset when people fold the stuff he's talking about into the same category as "open source". He's extremely concerned with making sure everyone always has permission to edit any source code for anything they come in contact with. TiVo is an example of something he feels strongly should never have been allowed to happen (hence GPLv3).

If you're an open source developer who does not care about that, then from his point of view, hurting you is a feature, not a bug.

Interesting, he actually has the same for SB already. Surprising that both Intel and AMD consider the additional die space to handle denormals efficiently worth it.

I'm sure you're absolutely right with your assumptions, although I personally think the way that C/C++ handle this in practice is perfectly alright: The compiler leaves the flags alone and sets it to the default to follow IEEE-754, but if the programmer has benchmarked their application and thought the whole thing through they should just be able to call fesetenv and are then responsible for worrying about other libraries. In practice for 99% of all programs denormals aren't really something to worry about after all.

I'm not complaining about how it's done either.If most of my stuff, 32 bit floats have way too much precision already.

I was just offering a rational why AMD and Intel, in particular, has put so much effort into supporting denormals.

That said, it's still an incomplete rationale. Intel has always pushed very hard.While AMD's 3d!Now simply lacked denormals, PowerPC Altivec disables them by default, ARM's NEON flushes them to zero, Intel made SSE fully IEEE-754 compatible from the start.

Oh I'm happy to get corrected by a knowledgeable person on unquestioned old truths. Sure it may make absolutely no difference to me in my day job (heck I worked on compilers - ARM though - for a while and it still wouldn't have made a difference for me), I like to learn new things.. or relearn old things :-)

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Adobe doesn't claim itself to be a bastion of freedom. So no, not the same mindset at all. And RMS's mentality hurts even other open source, GPLed developers.

I actually do think that the person you're responding to has a valid point here.

When RMS talks about freedom, he means something pretty specific, and he's pretty upset when people fold the stuff he's talking about into the same category as "open source". He's extremely concerned with making sure everyone always has permission to edit any source code for anything they come in contact with. TiVo is an example of something he feels strongly should never have been allowed to happen (hence GPLv3).

If you're an open source developer who does not care about that, then from his point of view, hurting you is a feature, not a bug.

But it also hurts open source developers who do care about that. Deliberately making your software harder to use hurts everyone regardless of ideology.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Adobe doesn't claim itself to be a bastion of freedom. So no, not the same mindset at all. And RMS's mentality hurts even other open source, GPLed developers.

I think that you are conflating two different concepts of freedom here . RMS is advocating for the freedom to modify and share the source code of any software, and that freedom shall remain equal for everybody , hence people should be free to cooperate.

The other "freedom" you are talking about has more to do with liberal licenses where developers are more interested in close sourcing and turning proprietary their software for commercial purposes . I think this has more to do with economic liberalism and that's something RMS is certainly not advocating for .

Whatever or not people agree or disagrees with him is not relevant. But i can tell you that Apple can still make useful things without the need of close sourcing the code . RMS advocates against proprietary software, not against useful things .

Oh yeah, you are right, in 2014, people expect their web browser to run Office suites and high end 3D games in full screen. Wait, I just checked the web and in 2014, web Office suites on the web still suck and nobody use them still and web games are still tic-tac-toes.

Well, pretty much all of education is either there or soon to be there. My job is pretty close to 40 hrs/wk of Office suite work and over half of it is now happening in Google Docs rather than in Office. The collaboration aspect is much better and more secure if you don't happen to have some kind of SharePoint thing going and instead are stuck emailing documents all over the place.

And that model is taking over in education. That's why Apple is investing quite heavily in getting iWork in the browser (and doing a rather good job of it, I must say). If you want to know why Apple is investing so heavily in javascript optimizations at this level, look no further than iCloud. But all of the LMS are web/javascript based and many are quite advanced. All of our in-house tools are web/javascript based and we periodically move users from browser to browser based primarily on how well they run JS and allow us to expand our apps to do more advanced tasks. Java as a cross-platform language is long dead everywhere I look. It's all either Objective C/iOS or javascript now. I haven't seen an in-house Windows or Java app in probably 5 years.

Wow quite clever how they handle the insane dynamic polymorphic nature by fusing profiled data from the previous optimization JIT into the LLVM and creating a cache of NOP statements at the ready. Quite a bit of thought has gone into this.

Good show apple

That's a commonly used JIT technique for well over a decade, nothing revolutionary there really.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Adobe doesn't claim itself to be a bastion of freedom. So no, not the same mindset at all. And RMS's mentality hurts even other open source, GPLed developers.

I wonder if this is a showcase of how there can be a general conflict between copyright constants and good software design. Most developers would say that small and modular programs that easily work with other programs are preferable. In the real world however, there is a tendency towards bloat. And software often doesn't play nice with other software, and when it does, its often not through the most interoperable methods (such as inter-process communication methods like DBUS, and pipes), but instead through dynamic linking and source code libraries.

However, in the proprietary world copyright encourages bloated programs because you only get paid for what you sell, and you have no incentive to write programs that play nice with others because you are not selling the other software, and if you did that you just would be giving other companies a leg up.

In the free software/open source world developers need to use copyright methods such as the GPL to prevent leeching from the community. But the gpl only applies to direct code derivatives, so if the code is too decentralized and interoapable you have a problem. This encourages people to pull an RMS.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Adobe doesn't claim itself to be a bastion of freedom. So no, not the same mindset at all. And RMS's mentality hurts even other open source, GPLed developers.

FSF doesn't claim to be a bastion of freedom for people. FSF claims to be a bastion of freedom for software, and the most important guarantee of the freedom of software is that nobody is allowed to take free software and make it proprietary.

It doesn't really matter if you're a hardline ideologue on free software like RMS and the FSF (I'm certainly not), but there is nothing inconsistent or self-contradictory in the FSF position.

I wish this much effort was being put into making something like python fast. In my opinion python is a much better thought out language and I would vastly prefer webdevelopment if it meant I could use python instead of js.

Agreed! A recent GUI program we wrote in python using PyQt and PyQtGraph for plotting starting consuming over 90% CPU utilization. Took a day of optimizing to get that down to around 60%.

Adobe considers it a problem if people use pirated copies of creative suite to do useful and interesting things. Same mindset, different direction.

Adobe doesn't claim itself to be a bastion of freedom. So no, not the same mindset at all. And RMS's mentality hurts even other open source, GPLed developers.

FSF doesn't claim to be a bastion of freedom for people. FSF claims to be a bastion of freedom for software, and the most important guarantee of the freedom of software is that nobody is allowed to take free software and make it proprietary.

It doesn't really matter if you're a hardline ideologue on free software like RMS and the FSF (I'm certainly not), but there is nothing inconsistent or self-contradictory in the FSF position.

FWIW - here are the comments by RMS about why he refuses to allow emacs to be moved to llvm:

It's a minority position even on the emacs list, as you can see from the comments, but yeah, there is a certain logic to it. And if RMS wasn't so hardline we surely wouldn't have the GNU stuff in the first place.

Oh yeah, you are right, in 2014, people expect their web browser to run Office suites and high end 3D games in full screen. Wait, I just checked the web and in 2014, web Office suites on the web still suck and nobody use them still and web games are still tic-tac-toes.

Well, pretty much all of education is either there or soon to be there. My job is pretty close to 40 hrs/wk of Office suite work and over half of it is now happening in Google Docs rather than in Office. The collaboration aspect is much better and more secure if you don't happen to have some kind of SharePoint thing going and instead are stuck emailing documents all over the place.

And that model is taking over in education. That's why Apple is investing quite heavily in getting iWork in the browser (and doing a rather good job of it, I must say). If you want to know why Apple is investing so heavily in javascript optimizations at this level, look no further than iCloud. But all of the LMS are web/javascript based and many are quite advanced. All of our in-house tools are web/javascript based and we periodically move users from browser to browser based primarily on how well they run JS and allow us to expand our apps to do more advanced tasks. Java as a cross-platform language is long dead everywhere I look. It's all either Objective C/iOS or javascript now. I haven't seen an in-house Windows or Java app in probably 5 years.

I wish this much effort was being put into making something like python fast. In my opinion python is a much better thought out language and I would vastly prefer webdevelopment if it meant I could use python instead of js.

Python has more advanced capabilities than JS, but these capabilities can also lead to security problems. To integrate python in the browser, you would need to develop a highly secure python runtime that prevents the execution of harmful code (a sandbox). This is not a trivial task, but Google's Native Client (NaCl) shows that it can be done: "Native Client is an open-source technology that allows you to build web applications that seamlessly execute native compiled code inside the browser."

I wish this much effort was being put into making something like python fast. In my opinion python is a much better thought out language and I would vastly prefer webdevelopment if it meant I could use python instead of js.

Python has more advanced capabilities than JS, but these capabilities can also lead to security problems

Umn.. name one.

Obviously you wouldn't want to provide the whole python standard library because that clearly contains dangerous calls, but the standard library is not the language.

So with a compiled-code cache that spanned pages and invocations this would be pretty darn sweet. But I think that the developers are also using a somewhat wrong metric about when to invoke the high-end compiler. If it's in a separate thread, that thread should be running whenever there's significant idle time on whatever chunk of js is currently at the top of the usage list (that they're keeping to manage the JIT anyway) and not currently fully compiled.

If you can do this kind of stuff concurrently, on most machines you have effectively infinite resources. Heck, I spent several minutes reading this article and the comments, and in that time a concurrent compilation thread could have distilled the javascript down to something terrifyingly fast.

Sounds like a great engineering tradeoff… for desktop machines. For battery-powered devices, speculative compilation in a different thread sounds like a guarantee that you'll use more battery with a possibility that it'll ever speed up user interactions enough to be noticeable. Making this work right on iPhones — and other phone platforms' efforts to adopt similar logic — will be quite a bit more challenging, I'd think.

Whether or not this is a major coup for Apple, when I read thru the full blogpost about the work, there were several points at which I thought, “wow, if *I* were interested in résumé strength in compiler development, I'd want to have done THAT.”

For all the complexity here, Apple seems to have squeezed out about a 30% speed improvement, possibly by putting the advanced compilation thread on a separate CPU core so there's additional power used, limiting its application on mobiles. Potentially, future changes in the language definition will be harder to implement, and the codebase will be more expensive to maintain.

And yet, for the vast majority of today's billion-plus mobiles, the techniques here just are out of reach; might be for 5 or 10 years.

Assuming that Apple was smart about prioritizing their work on the low-hanging fruit, it seems that javascript isn't going to get a lot faster without a LOT more work, i.e., isn't going to get a lot faster. Seems the next speedup will be from introducing static substitutes—maybe LLVM opcodes?—that retain device independence while reducing the workload on the client.

I think the biggest issue with Python JITs and PyPy in particular is interfacing with C libs which is either slow, or doesn't work at all.

I don't think that's the case, at least any more. Quite a few packages use CFFI which was developed specifically for PyPy, but even ctypes seems to work fine.

When refering to "C libs" I was somewhat imprecise. I wasn't talking about ctypes but native extensions which is: a) the thing that PyPy hasn't nailed quite as well, b) probably solvable by bringing them all in under LLVM and c) the thing blocking interesting apps like Trac, Zope/Plone or OpenERP on PyPy.