Testing mobile browser compatibility — the beginning

About a month ago the software department
of Vodafone Internet Services, based in Düsseldorf, Germany, asked for my help in creating
mobile widgets according to the W3C
Widgets specification. In particular, they’d noticed there are differences between
browsers even on mobile phones (imagine my surprise), and decided they needed advice from
a specialist (that would be me).

Better still, it quickly turned out that they were willing to pay me for doing
serious mobile browser compatibility tests and publishing them on this site.
The payment thingy is quite unusual, I can tell you (though not entirely unique).

This is easily the best job offer I’ve gotten in my entire freelance career, so I hurried to
accept it. Meanwhile I’ve done mobile tests for five days; enough to offer some
guidance for setting up a doctrine for mobile browser testing.

As far as I’m concerned you can look over my
shoulder while I’m working, but please PLEASE remember that everything I say
may change radically without notice after I’ve tested the same browsers on other devices.

Right now I’ve only done a few tests of functionality that’s basic to the
mobile experience, and even these basic tests will likely have to be expanded.
Besides, right now getting a general feeling for mobile testing and its manifold problems
is more important than running lots and lots of tests.

If you’re interested in helping me you can
follow me on Twitter, which is
shaping up to be a fantasically valuable aid.

I report oddities and bugs within minutes of finding them, and I’ve received quite a bit of
useful feedback already. In fact, it’s
likely that these tests won’t ever succeed without cloudsourcing some of the more
complex problems.

(Please remember I do most of my tests during European office hours.)

Sheer numbers

The first and most obvious problem in mobile browser testing is the sheer numbers.
The Vodafone master list of devices that must support their widgets contains
43 phone types, and those are only the modern ones that have a reasonable chance of containing a
modern browser.

For the moment I’m considering every single browser on every single device unique;
so if I test five Nokia phones that all have WebKit, I’m testing five different WebKit
versions and the resulting compatibility tables will show five entries.

Sure, I assume that these five WebKits will resemble each other, and I expect to be able to
eventually merge these five entries into one. I’m not going to hurry that process, though;
one of the points of this research is to find out if there are differences between these five
devices. There could be, and if there are Vodafone and I (and, in fact, the world at large)
have to know about them.

Fortunately, the Vodafone team has quite a few phones, and although it’s absolutely impossible to test everything
on every phone, it does allow me the chance to validate my conclusions time and again.

The browsers

How many browsers are there in the mobile space? Right now I think
the correct answer is five, but I may be wrong. They are:

NetFront, which is supposed to be very popular in Asia. Right now I assume it comes
in flavours, but there’s so little information about this browser that I just don’t know.

IE Mobile. The current default is based on IE4.
Version 6 will be released pretty soon, and it is based on the IE6 rendering engine.
I assume it comes in flavours.

the Blackberry browser, although I heard rumours Blackberry is going to switch to
WebKit. It don’t think it comes in flavours, as there’s only one Blackberry vendor, but I assume it
has versions instead.

There’s also Fennec of the Mozilla Project, but as far as I know
it’s in early alpha, and I was even unable to find a formal homepage for it.
It’s not yet the default browser on any mobile device; as soon as it is I’ll include it in my tests.

If you know of more mobile browsers, please leave a comment.

Opera and WebKit form the vanguard of the mobile space, IE, NetFront and Blackberry the rearguard.
We’ll leave it at that for the moment; I will provide a more detailed breakdown when I have more data.

Test pages

I’m creating new test pages for my mobile tests. The main reason is readability;
my current test pages are optimised for screen, and the events tests, especially, require
ample amounts of space to see all event reports. Mobile devices just don’t have that space
available. Besides, the event test pages refuse to work in Nokia WebKit (two versions tested).

After three or four tests I found that the version of IE Mobile I’m using doesn’t
support the event object at all and requires you to use inline events. document.onclick,
for instance, is not supported. (Oh, and createTextNode() seems to be missing, too.)

So theoretically, if I want to give IE Mobile
a fair chance to compete, I’d have to write tests that use inline event handlers only
and don’t depend on the event object.

Right now I’m not giving IE Mobile a fair chance. Obviously, I will validate my conclusions
by an IE Mobile test on a completely different phone, but if that IE version has the same problems,
I’m going to quietly forget about it. I’m just not going to waste my time on a browser
that forces me to break one of the core rules of unobtrusive JavaScript.

By the way, don’t blame the regular IE team for this fantastically incapable browser.
IE Mobile is created by a completely different team.

But exactly which IE Mobile was I testing? That turns out to be a tricky question.

Identification

Identifying exactly which browser is running on a certain phone is going to be a major problem.
Mobile browsers, being chromeless, are usually impossible to distinguish by sight, so I
need a browser detect script to help me out. Unfortunately, writing such a script is not as
easy as it seems.

Right now browser vendors are running around like headless chicken strewing identification strings like
chaff, making sure they’re totally incompatible with desktop identification strings, other mobile
identification strings, and occasionally even other identification strings sent by the same browser.

Opera is rumoured to be pulling its act together,
but I haven’t yet personally verified that.

In any case, distrust any market share stats of mobile browsers you see. The detection script
is quite likely to be totally wrong.

IE

When I started writing this entry I was convinced that the IE Mobile I was testing
was the brand-new IE Mobile 6. Why? Because it identified itself as IE6. Silly me. I’m
not going to make that mistake any more.

When I saw
this IE Mobile 6 promo video I started to doubt my identification.
The IE Mobile I test doesn’t have all these new features, even though it runs on a
touchscreen phone. Besides, IE Mobile 6 has not yet been released (I think).

So the mobile browser identifying itself as IE Mobile 6 is in fact the earlier IE Mobile that
was branched off from IE4. Still, it supports basic Ajax, something that IE4 didn’t.

Now if the old IE Mobile identifies itself as IE6, how will the new IE Mobile, officially
announced as version 6, identify itself? Also as IE6? I wouldn’t put it beyond
Microsoft.

To make things worse the old IE Mobile also has IEMobile 7.11 in its identification
string. No doubt the new IE Mobile will announce itself as IEMobile 8.01 at the
very least.

WebKit

WebKit version numbers are mostly useless. The most common mobile one I encountered is 422, which
is distinctly lower than Safari 3.0’s 522. The funny thing is that WebKit on Nokia supports
:only-child and Safari 3.0 doesn’t.

Therefore it seems likely that Nokia has branched off its browser from WebKit 422 and then
continued to work on it by itself, importing (apparently) certain newer WebKit modules but not
updating to an entirely new version. They didn’t bother to update the identification string,
either.

Worse, many other mobile device manufacturers are doing the same. WebKit/422 on Nokia is likely
to be different from WebKit/422 on Motorola. I’m inches away from declaring the WebKit version
number completely worthless.

By the way, the navigator.vendor of all mobile WebKits I’ve seen so far is Apple!
This is pure misinformation (except on the iPhone), and I assume it’s a mistake of the WebKit
core team who’ve accidentally left this value in place when they received updates from Apple.
Please make this property empty; no information is better than false information.

Opera

Although modern Opera’s Mobile or Mini have Mobile or Mini in their identification
string, the particular versions I’m using don’t. Fortunately Opera Mobile version numbering
follows the Opera desktop one, so the knowledge that I’m testing version 9.5 helps me instead of hindering me.

As to the Mini, it took me and an Opera employee
quite some time to identify it as version 2; distinctly antiquated. Besides, when I sent it over to
Steve Souders’s UA String Parser, which
uses the HTTP User-Agent string instead of JavaScript’s navigator.userAgent,
this browser identified itself as NetFront! The string was totally, and I mean totally, different.

You can’t make this up.

New classes of incompatibilities

It seems that Nokia devices refuse to send key codes to the browsers.
Although all key events are properly fired, keyCode and friends, which are supposed to contain
the key number, stubbornly remain 0. This is not the browsers’ fault; if the device withholds
this information there’s nothing they can do.

Until now I’ve seen this Nokia bug only on devices with numerical keyboards. Maybe touch devices
or devices with qwerty keyboards will behave properly. One more thing to test.

This is an example of an entirely new class of problems: device compatibility errors.
Since I’ve only discovered them just now I’m not yet sure how to treat them, except that we
should be very careful to distinguish them from browser compatibility errors.

There’s also a new class of browser compatibility errors: when browsers are unable to make full
use of the device’s capabilities. An example is an orientation change from portrait to landscape or vice versa:
the device may support this fine, but if the browser doesn’t, nothing happens. On the HTC Diamond Opera supports
orientation change; IE Mobile doesn’t.

The beginning

This is only the beginning of our research into the mobile sphere and our discovery of exciting new
classes of incompatibilities. Again, if you’re interested in helping me out and/or have experience in mobile browser
compatibility problems, follow me on Twitter. I’ll need
the help.

I'm happy someone like you landed this gig. This information has been needed for a long time by the industry at large. And since you're so open with your findings, this can only make the job of the web developer (me!) easier.

Not sure if this is relevant to your tests but I've observed chromeless Webclips on the iPhone and iPod touch exhibiting behavior that differs from the same page displayed in MobileSafari. The one example that comes to mind is onOrientationChange, the event doesn't fire in a chromeless webclip.

Shaun re chromeless Safari windows: isn't there a setting you can use to fix a Safari window in one orientation? If that's so, it's not a bug but a feature.

Wolf re Blackberry: Thanks! Useful info.

David: Vodafone doesn't have a Playstation Portable, so I guess I won't test it in the near future. Thanks for the Skyfire tip; it uses the same client-server scheme as Opera Mini. Will have to test it eventually.

Alberto: Phone in SonyEricsson I tested is Opera Mini, but I'll test a modern SE next week, I hope.

BlackBerry browsers (pre Storm): Terrible. Especially as they are shipped with various default settings! The exact same device is pre-configured differently, depending on vendor and branding. With/without support for various things like js, css "table-support".

With IEMobile it seems to me that things also depend alot on the windows version itself (CE, 5, 6, ...) and the actual device. The newer candidates seem to handle things "quite ok".

I'm building a mobile app at the moment and I'm stuck at the font sizes part.

My current solution is to use percentages, seems to be the safest bet. Most mobile browsers override seem to override your font sizes with their settings; so I'm setting some "good defaults" and make sure my design works properly with a good range of font sizes.

Android based mobile browser does a good job at rendering complex web pages. I had a chance to use one Android powered device last weekend, the only one in Down Under marked so far (pls correct me if I am wrong), great toy! Good on you PPK.