Most major manufacturers recently rolled out the Android 5.0 Lollipop update to their respective flagships and we were more then eager to test how their performance has changed now that Lollipop is here. The speed comparison will be done on our usual battery of synthetic benchmarks.

So, does Lollipop improve performance or make it worse? Read on to find out.

Our expectations were high as Android 5.0 made the jump to a new runtime, ART, which replaces the Dalvik. Dalvik uses Just-in-time compilation – after the app is already running, so it has to be done quickly, leaving little time for optimization.

ART meanwhile does ahead-of-time compilation – that “Optimizing apps” screen you get at first boot. You may have noticed that it takes longer than before, the reason being the phone is compiling the apps and spending time to optimize them properly.

This requires a word of warning too – as ART is a new runtime, some apps suffer from some issues, but with multiple benchmarks a issue with a single test shouldn’t be a problem overall.

Ready, set, go! The first lap is around the Geekbench 3 ring and the Galaxy S5 keeps in top position with a small increase in score. The HTC One (M8) and Nexus 5 got much bigger boosts, while the LG G3 has actually gotten worse.

GeekBench 3

Higher is better

Samsung Galaxy S5 (Lollipop)3120

Samsung Galaxy S5 (KitKat)3011

HTC One (M8) (Lollipop)2923

Nexus 5 (Lollipop)2743

LG G3 (KitKat)2563

Nexus 5 (KitKat)2453

LG G3 (Lollipop)2370

HTC One (M8) (KitKat)2367

We look to AnTuTu 5 for overall performance, where all phones got a small boost. The Nexus 5 in particular got a massive boost.

AnTuTu 5

Higher is better

HTC One (M8) (Lollipop)45530

Samsung Galaxy S5 (Lollipop)45348

HTC One (M8) (KitKat)44020

Samsung Galaxy S5 (KitKat)43164

LG G3 (Lollipop)42038

LG G3 (KitKat)39905

Nexus 5 (Lollipop)37774

Nexus 5 (KitKat)29500

Basemark OS II is another full-system benchmark and it shows improvements across the board ranging from small (One (M8)) to big (LG G3).

Basemark OS II

Higher is better

Nexus 5 (KitKat)1351

LG G3 (Lollipop)1327

HTC One (M8) (Lollipop)1202

Samsung Galaxy S5 (Lollipop)1147

HTC One (M8) (KitKat)1126

LG G3 (KitKat)1126

Samsung Galaxy S5 (KitKat)1082

Nexus 5 (Lollipop)948

Single and multi-core performance is a bit of a mixed bag, but scores largely remained unchanged. Besides CPU performance, the benchmark also tests the internal storage, the graphics and web performance. It’s not clear which of these contributed to the performance bump as Lollipop did little for web performance.

Basemark OS II (single-core)

Higher is better

Samsung Galaxy S5 (Lollipop)2510

HTC One (M8) (KitKat)2428

HTC One (M8) (Lollipop)2426

Samsung Galaxy S5 (KitKat)2415

LG G3 (Lollipop)2267

LG G3 (KitKat)2213

Basemark OS II (multi-core)

Higher is better

Samsung Galaxy S5 (KitKat)10063

LG G3 (Lollipop)9975

HTC One (M8) (KitKat)9860

HTC One (M8) (Lollipop)9827

Samsung Galaxy S5 (Lollipop)9646

LG G3 (KitKat)9611

Going to GFX benchmark for a look at 3D performance we see that very little has changed. The new Android version brings the Android Extension Pack, which promises PC-level graphics. These versions of the benchmark don’t support the new APIs and most games don’t either (though Unreal did demo the tech on the Tegra K1).

GFX 2.7 T-Rex (onscreen)

Higher is better

HTC One (M8) (KitKat)30.1

HTC One (M8) (Lollipop)30

Samsung Galaxy S5 (KitKat)28.1

Samsung Galaxy S5 (Lollipop)28

Nexus 5 (KitKat)24

Nexus 5 (Lollipop)24

LG G3 (KitKat)20.5

LG G3 (Lollipop)20

GFX 3.0 Manhattan (onscreen)

Higher is better

HTC One (M8) (Lollipop)13

Samsung Galaxy S5 (Lollipop)12

HTC One (M8) (KitKat)11.9

Samsung Galaxy S5 (KitKat)11.7

Nexus 5 (Lollipop)10

LG G3 (Lollipop)7.7

LG G3 (KitKat)7.2

Basemark X didn’t see much of a change in performance either, aside from the LG G3, which slipped back a bit.

Basemark X

Higher is better

HTC One (M8) (KitKat)12396

HTC One (M8) (Lollipop)12257

Samsung Galaxy S5 (Lollipop)11798

Samsung Galaxy S5 (KitKat)11744

Nexus 5 (KitKat)11638

LG G3 (KitKat)11552

LG G3 (Lollipop)10580

Nexus 5 (Lollipop)9786

For web browsing it seems that Lollipop brought some regressions in Kraken 1.1. The Nexus 5 slowed down significantly, the LG G3 also took a hit. The HTC One (M8) is the big winner here with a noticeably jump forward in times.

Kraken 1.1

Lower is better

Samsung Galaxy S5 (Lollipop)5968

Samsung Galaxy S5 (KitKat)6043

LG G3 (KitKat)6987

HTC One (M8) (Lollipop)7023

Nexus 5 (KitKat)7148

LG G3 (Lollipop)7632

Nexus 5 (Lollipop)9524

HTC One (M8) (KitKat)10296

BrowserMark 2.1 on the other hand saw some gains, big ones for the Galaxy S5. The LG G3 stood in place, posting a virtually identical result.

BrowserMark 2.1

Higher is better

Samsung Galaxy S5 (Lollipop)2066

LG G3 (KitKat)1474

HTC One (M8) (Lollipop)1460

LG G3 (Lollipop)1453

Nexus 5 (Lollipop)1440

Samsung Galaxy S5 (KitKat)1398

Nexus 5 (KitKat)1286

HTC One (M8) (KitKat)1069

Android 5.0 Lollipop carried some promise of speed and while there are some improvements, this is certainly not like when Android 2.2 Froyo changed the Just-in-time compilation (JIT) engine to Dalvik back in the day.

Of course, benchmarks poorly capture some of the differences between the two runtimes. JIT typically compiles only the most intensive code – like a benchmark – because this is done in real-time while the app is running, compiling the whole thing will slow down the app. Also the compiled code is cached but not kept permanently so apps may have to be re-JITed.

Lollipop also promised better battery life, we’re in the process of testing that too (see you again next week).