Archive

In the last post we discussed how digital multiplayer games have evolved, various strategies for getting different game clients to keep their games in sync, and why the core HTTP request/response architecture of the web was fundamentally flawed when it came to multiplayer games. In this post, we’ll dig into the HTML5 solution to this problem and how to use it in desktop or mobile web applications.

WebSockets are a new technology that was written in response to JavaScript developers using AJAX calls to keep checking back with the server to see if anything new had come in. It was clear that the current design was not meeting the needs of modern Web 2.0 applications, which wanted to leave a connection with the server open for as long as the web page was up so that any new information from the server could instantly be read by the JavaScript running on the client and displayed.

At the lowest levels, WebSockets do piggyback on top of the World Wide Web infrastructure (hence the name). They follow a URL scheme, similar to the ubiquitous HTTP definition. Since the underlying protocol is different from HTTP, however, the URL scheme begins instead with a new label: ‘ws‘ (for unencrypted streams) and ‘wss‘ (for encrypted streams). For example:

ws://yourserver.com/yourwebsocketresource

If you’re running your own web servers, you’ll need to have a program running on the backend to handle WebSocket requests such as these. Apache’s ActiveMQ project is one example of WebSocket-aware server software. A full discussion of the pros and cons of various server configurations and software packages to handle WebSockets is beyond the scope of this post.

WebSocket technology is new enough that browser support when running your client HTML5 application should not be assumed. To assist in this, there are several JavaScript libraries that wrap the WebSocket calls in their own API and thus can downgrade the connection to an HTTP polling connection if the client application is running on a browser that does not support WebSockets, and we’ll dig into one of those wrappers momentarily. In the HTML5 mobile space, WebSockets were added to Mobile Safari in iOS 4.2 (though the WebSocket spec supported is an out-of-date specification, so make sure your servers are compatible with the old spec if you want to support iOS devices), but as of this writing are not supported on the default Android browser. WebSockets are supported in the newly-announced Chrome for Android beta, which only runs on Android 4.0.

If you are running in an environment that supports WebSockets, interacting with one on the client side is delightfully simple. Read more…

This is a fairly specialized topic, but it’s also an easy one, so let’s devote a quick blog post to it for the sake of thoroughness. Maybe it will help some of you HTML5 web searchers out there. (Welcome, web searchers!)

Some of you may be wondering if an HTML5 app running on the iPhone has access to the devices latitude and longitude, or “geolocation.” It does! One of the new HTML5 APIs is called the Geolocation API, and it gives your JavaScript access to the device’s geolocation information just as a native app has access to it.

Fun fact #1: when you fetch the iPhone’s geolocation info, the little compass icon in the status bar appears, just as it does for a native app, to remind the user that your application is tracking where they are.

Fun fact #2: when you attempt to access the device’s geolocation info, the user will get a popup asking if they want to grant permission to your app to…you know…track where they are. They may say “no.” A good app will be prepared for rejection.

Fun fact #3: many modern desktop browsers also support the geolocation API! In my experience, the values a desktop computer comes up with as its best guess for where you are is a joke though.

Get on with it

Final fun fact: You know how when you’re using Google Maps on your phone to drop a pin on the map showing where you are and that pin moves back and forth a few times before it finally settles down? You’re going to get that, too. When you begin querying the iPhone for its location, it will take a little while before the iPhone can come up with a final value for you, so be prepared for the location to change on you even if the user is sitting still.

So how do you fetch the location of the device, assuming your user allows you access to this private information? The call is pretty straightforward:

The coordinates that are returned are (on the iPhone at least) floating point numbers that go down many, many decimal places. In practice, I’ve found those decimal places give you a pretty decent idea of what house you’re at, but not enough resolution to tell you what room of the house you’re in. What you do with that information is up to you! There are several tools available to help you put a latitude/longitude value on a JavaScript map, for example, but that reaches into general JavaScript programming and beyond the scope of this blog, at least for now.

Or you could help poor Spirit find the nearest Starbuck’s. Goodness knows it has earned a latte.

If you’ve spent any time with an iOS device like an iPhone, you’ve discovered that one of the coolest behaviors of the device is that it responds to how you are holding it. If you hold it vertically, the screen orients itself in portrait mode, and if you hold it horizontally, the screen rotates to landscape mode. For various reasons, many native apps lock this behavior down so that the screen does not orient no matter how the user turns the device. Examples include the home screen on the iPhone itself (though not the iPad) or many native games where the the UI has been carefully crafted for a specific screen shape. Some native apps, however, do adjust their interface and behavior depending on how the user is holding the device. The humble iPhone calculator is a great example of this type of app behavior (if you haven’t noticed, try launching the calculator while holding your iPhone vertically, then watch what happens when you turn the iPhone horizontal). As an HTML5 developer probably knows, Safari is another app that reacts to the orientation of the device, by default rotating the web page it is pointed to so that the display is always pointed “up” and adjusting the zoom level of the screen to the new aspect ratio.

First the good news

For HTML5 developers writing apps on top of mobile Safari, I have good news and bad news. In the good news department, somewhat to my surprise, Apple grants your HTML5 app access to the current orientation of the device. This allows me to build an app that can determine whether the device is in portrait or landscape mode before I draw to the canvas and render a different UI depending on the which mode the device is currently in. See?

Landscape mode is red!

Portrait mode is blue!

You can get at this information in a couple of different ways. You can query the current orientation of the device by reading

window.orientation

which will return an integer value of either 0, 90, or -90 (in number form, not a string or object) depending on how the user is holding the device. In theory, iPhone may return a value of 180 someday as well, as the iPad does today, but in practice a value of 180 is not supported on the iPhone in iOS5.

Alternatively, you can register a callback function which will provide direct access to the device’s accelerometer values:

If you register a callback function, you get much more fine-tuned information about how the user is holding their device, but be ready for a lot of callback events, similar to the deluge of data you get with touchmove callbacks.

But there’s some bad news

So that’s the good news: your application or game can be aware of the device’s orientation, even at the HTML5 layer! Here’s the bad news: your application or game MUST be aware of the device’s orientation. Yes, I’m afraid if your HTML5 app is using a full-screen canvas to render your UI within the canvas element, you can’t really get away with ignoring the orientation of the device. Here’s why: unlike a native application, an HTML5 application can’t stop Safari from responding to the orientation of the device. If the user turns their device from portrait to landscape mode or vice versa, Safari will run its little rotation animation and the screen will rotate in response to the new orientation. It is up to your application how to behave in response to this behavior, but if you do nothing, your canvas element that fit so well at 320×460 will no longer match the screen’s new dimensions of at 480×300. This behavior takes place whether your app is running in standalone mode or embedded within the full Safari UI. (Standalone mode was discussed more in the previous post.)

Don't let the Dark Side dominate your destiny. It does terrible things to your family relations.

It gets worse. A simple rotation of your UI in response to the orientation changing–in effect, trying to “undo” the rotation so your UI is constant no matter the orientation of the device–is not sufficient. There are two problems in trying to take the relatively easy way out (otherwise known, as Master Yoda tells us, as the Dark Side of the Force).

First, Safari is going to do that rotation animation whether you like it or not. If you re-orient your UI back to its original position on the physical device after it has rotated, to the user it will look like the screen has rotated one way then “snapped” back to where it was. It looks bad, trust me, I tried it.

Second, when the screen rotates, it’s not just a matter of swapping the X and Y values to pretend it didn’t happen, because those don’t take into account the additional iPhone UI elements your HTML5 app is operating within which _will_ rotate along with the device. In standalone mode, you have the 20 pixel header bar which moves from the top of the vertical screen to the top of the horizontal screen. This is why the dimensions of a standalone app in portrait mode are 320×460 but in landscape mode become 480×300. When operating with the Safari bar on the bottom of the screen, it just gets worse.

Trust your feelings

What to do instead? Embrace the Light Side! Having an app that gives the user a different UI depending on whether they’re holding the iPhone in portrait or landscape mode is really cool. It’s a mode we as developers haven’t begun to properly explore yet, and it’s high time we’ve started to. How fortunate we are as HTML5 developers to have access to this information! We should use it and write our applications to leverage it. Your users will think it’s the coolest thing ever and love using your app on their phone.

There you have it. Orientation on the iPhone in HTML5. Learn it, love it. You kind of have to. And let me know what you build with it!

I’m listening to music as I type this entry. I love music. It adds a level of depth and immersion to an experience that is difficult if not impossible to achieve otherwise. Movies figured this out very early on. Games did too, though it took a while for technology to catch up. Along with emotive music, sound effects help capture the experience of deeper interaction with your application. Having virtual buttons click when you press them, or even cartoonish sound effects when your flash sprite jumps up and down on the screen helps give the user of the app or player of the game closer ties to the virtual system they are interacting with.

First, without preloading, the first time your app tries to play a sound the app animations are going to stutter as the JavaScript works on loading up the sound rather than refreshing the screen (remember, JavaScript isn’t truly multithreaded…alas). The whole point of preloading is to remove that stutter by having any audio already loaded and ready to play the instant it is needed when the app was first loading up. The iPhone ignores any preload requests, however (either in the HTML “audio” element or via the JavaScript Audio.load() call), so the first time you play a sound you are stuck with that hit.

Second, unlike the modern desktop browser, iOS Safari will only play a single sound at a time. On the desktop, you can load two different audio elements (say, background music and a sound effect) with multiple Audio objects in your JavaScript, and if you call play() on each object, the browser will play both audio tracks simultaneously. The iPhone does not support this. Instead, when your app calls play() on the second audio object, the first audio stream is interrupted and the second stream plays. If you call play() on the first audio stream again, the track continues from where it was interrupted by the second audio stream.

If you know the enemy and know yourself you need not fear the results of a hundred battles. – Sun Tsu

Armed with the knowledge of the issues we are facing, all is not entirely lost. If you’re writing a game, you can still have sound effects, but they must be short. Short audio tracks will load faster the first time they are played and thus won’t impact your game quite as much as they will if the audio is long. Different sound effects are also unlikely to interfere with each other if any particular effect only takes a fraction of a second to play. Music might even still be possible as well, but you won’t be able to write atmospheric music that is intended to be running in the background for the entire game session, as any sound effects will interrupt your music track. You might, however, still be able to play some music to set the tone during an introductory or victory screen for your game. Look for opportunities to play music when there will not be any other sounds from the game that will interfere with that music.

I was really looking forward to a Bones pic for this post. So I’m going to put one up anyway, because I’m stubborn like that.

The latest update of the HTML5 iPhone test app introduces touch events to the app. Yes sir, that’s right, the user now has an interface. Of sorts, anyway. The purpose of this update, and the reason why I thought it was going to allow me to blast the app into oblivion, was to add a new goblin token every time the user touched the screen and start sliding that token back and forth with all the tokens that have already been added. With the amount of JavaScript processing going on to move and render each individual token, I wondered if I’d be able to get 20 or so tokens moving before my iPhone 3GS started to show some serious stutter. Yeah, that didn’t happen.

THIS IS CANVAS!

At one hudred tokens, I finally got the iPhone rendering to drop below 50 FPS. At three hundred tokens, all animated to slide back and forth with every heartbeat tick, the FPS on the iPhone drops to 24FPS, but honestly if it wasn’t for the number printed at the top of the screen, I don’t think I would have realized it.

Now is probably also a good time to point out that on my desktop, I haven’t seen the slightest dip in FPS no matter how many tokens I add to the screen. You desktop HTML5 developers are in really, really good shape.

The extra code that I mentioned in the last update to make this piece a snap to add worked swimmingly. Thanks to the wonders of OO programming, pretty much all I needed to add to get hundreds of tokens rather than a single token was the following:

As an aside or three, in the process of adding the touch event, I learned a few tricks to getting an iPhone web application to behave more like a native application.

Trick the first: as soon as the app is loaded in the browser, call window.scrollTo to move the location bar out of the way. This both makes the screen look more like a native app with less of the browser UI in the way and provides your app with more precious screen space. Like so:

if(!window.navigator.standalone){
window.scrollTo(0, 1);
}

Trick the second: did you catch that test for whether navigator was in standalone mode in there? If a user bookmarks your app and saves it to their home screen, several nifty features open up. 1) You can specify an icon and loading image for your app. 2) You don’t have any of the Safari UI in your way, granting you even more valuable screen real estate. 3) You can even cache all of your resources on the device so that even if the device goes offline, a user can still run your “web” app.

Trick the third: use event.preventDefault() to keep Safari from panning up and down as the user “drags” the canvas element up and down with their finger. Again, this allows your web app to behave more like a native app and gives you the opportunity to support drag events from the user across your canvas. Note: the only way I was able to disable the standard panning behavior when the app was running both within the browser and in standalone mode was to add a preventDefault() trigger to the body element in the HTML page:

<body ontouchmove="function(e){e.preventDefault();}">

At this point the web app is running strong and feeling an awful lot like a native application. I wasn’t sure when I started this project that this would be the result, and to be honest I’m thrilled with the results so far. So come on you HTML5 developers, lets see some apps for the iPhone!

As you can see, the test app has been updated to support a wee bit of image rendering for today’s iteration. Things you can’t see are A) the image is animated to slide back and forth across the screen, and B) there’s probably three times as much JavaScript code now under the hood as I add a whole bunch of object oriented design to the implementation of the app. This will help make tomorrow’s update super easy (or so he says now…).

The FPS numbers are still hearteningly very, very high. However, I will admit that the human eye can detect a bit of stuttering every couple of seconds or so as the image slides back and forth across the screen. This is true on the desktop as well, so it seems to be more of an HTML5 engine thing than an iPhone platform thing. We will see as things progress whether it ends up being too distracting for the player or if it ends up being unnoticeable in the end.

Plan for tomorrow? Kill the app dead and dance on its sticky bones. More soon!

The first question we need to answer when exploring HTML5 development on a mobile device is whether the hardware is going to be powerful enough to create the experience we want our users to have in an interpreted language like JavaScript. Towards that end, the first test app I built was a very simple JS application that did the least I felt I could get away with:

2) Started a heartbeat timer that would optimally go off every 10 ms with setInterval()

3) Every time the heartbeat fired, change a random RGB value one point either up or down (also randomly)

4) Repaint the screen with the new value to show animation working and display the number of times the heartbeat fired in the past second (Frames Per Second)

In other words, I wanted to measure the FPS I could crank out of the iPhone without doing anything else. If the FPS was in the 10-20 range, the type of animations I’d be able to support in a game would be much more limited than if the FPS was in the 40-60 range. Here’s what I got on my iPhone 3GS:

Yeah. That'll do.

83 frames per second. Which is about 5FPS slower than my Chrome desktop can put out. And conventional wisdom for PC game development is that anything beyond 60FPS hits the diminishing returns curve pretty hard.

Of course, this is with ABSOLUTELY NOTHING else the app has to handle, so it’s not time to pop any champagne corks yet. But a critical first hurdle has been passed. Hooray!