~ On Programming and Language Design

Why I have my own Android game framework

All I wanted to do was to make a game for my Android tablet, so how come I ended up writing a new C++ framework? What happened? Between the standard Android SDK and the numerous frameworks, it seems illogical to build something new. A serious case of “not invented here syndrome” perhaps? No, I didn’t want to, and even tried to avoid it, but couldn’t. Here is an approximate recollection of what happened over the past three weeks.

New to Android

I have an Android tablet sitting by my desk. It really isn’t used much other than to occasionally stream music. Since I had a bit of time available I thought I’d program something for it. Before I even decided to write a game I just jumped into the Google samples. It didn’t take long at all to get a basic “Hello, World.” program working and to understand a bit about Android’s architecture.

The tutorial section then offered an introduction to OpenGL programming, so I thought I’d move on to there. Thankfully this was for OpenGL ES 2.0 which is using the new GL approach (no fixed pipeline), so it’s a modern tutorial. I liked it, but I hit a painful stumbling block: Java.

I’m not really complaining about Java per-se here, but how it is integrated with OpenGL. OpenGL wants native memory, but Java and native memory don’t mix well. You’re left using this convoluted ByteBuffer system. It pained me to write this code knowing how absolutely trivial it is in C++. I suspect the wrapping and conversion is also quite inefficient (given the large amount of data involved this would likely become an issue). Nonetheless I decided to press on. I figured with a decent abstraction it could probably work okay.

I moved on to the next demo of using fragments for dialogs — instead of the method introduced earlier in the tutorials. I got this setup and compiled quickly enough but it didn’t work on my tablet. To find out why I learned about the logging facility (and the debugger, which no longer appears supported, or at least never works). Turns out the recommended fragment approach is not supported on my device. If I wish to use it I’d need to install the support library.

It isn’t enough just to install the support library however, you need to change all the class names to use the support version instead. At this point I need to mention that I consider it a significant failing of Java not to allow any kind of global typedef’s. It’d avoid the hassle of this prefixed package name nonsense. Anyway, I set this up, compile, package, install, and it works fine.

Something catches my eye at that packaging stage however. This added support library makes that “dex” step take a long time — about 5 times longer than the entire rest of my build (still only a few seconds, but enough to distract me). It it only happened once it’d be fine, but it runs every time I do a build! I assumed I did something wrong so checked online. Sure enough other people have the same problem, and it relates to the number of Java libraries I use. So as my project grows I could look forward to longer and longer packaging times. Long compilation times for incremental changes are a huge hit in programmer productivity. Google really should be ashamed of this “dex” bottleneck, especially knowing there is no good reason it couldn’t be incremental as well.

From Java to C++

I didn’t give up yet however. I figured that if the install took a long time likely there was a way to test without having to repackage and install all the time. These are after all just a bunch of Java classes and incremental build and run is not uncommon to this language. Alas, for Android development I could not find any such option. You could test all your generic code separately, but to get any of the Android framework you have to install.

What I really wanted was a desktop development approach. I would write, debug, and test the code primarily on a host machine like Linux. This thought led me to libGDX, which does exactly this. But at the same point I started to consider writing in C++ instead. Looking at the desktop and mobile markets C++ actually seems the more portable solution (compared to Java). It deals with OpenGL memory a lot better and doesn’t have as many problems of efficiency (parts of libGDX are also written in C++ reinforcing that this is an issue). I also thought that if I were to do a game I wouldn’t be using the Android UI layer anyway (as very few of the Android games do).

So now I was looking for the C++ equivalent of libGDX. There is a C++ port, but it isn’t very far along it seems. I also found something called GamePlay. I liked this enough to get the samples running on Linux and to contribute some patches. The more I used it however the more I saw it was an actual game engine instead of a framework. That is, it is specifically targeting particular types of games rather than being a completely open system. I was uncertain whether this was the best approach, but I persisted since it really offered a lot of features, and it did appear I could avoid most of the engine.

I hit a critical showstopper however: the frame rate for even simple scenes was extremely low on my tablet. I’m glad I did the Java approach first here; I had already coded a simple scene and saw how fast it could function on my tablet. GamePlay came nowhere close for an even simpler scene. I spent several days debugging this and trawling through forums. I even wrote a new pure NDK OpenGL demo to see if I could reproduce the problem. I determined it has nothing to do with the OpenGL parts, but never could pinpoint where in the GamePlay framework something was going wrong. Lacking any progress and facing a showstopper defect I really had no choice but to abandon this framework.

Onwards

I mentioned that I coded a native demo using the NDK as a test. I also worked on the Linux game loop in the GamePlay library. Thi led me to a new insight: the amount of coding for basic cross-platform game engine is actually quite limited. I didn’t want a layer between me and OpenGL. Common libraries could handle the image loading. I also figured I could get libraries to do the Font loading and text display. Audio I already know how to do cross-platform and saw the Android has a reasonable API.

A New Game Framework

I finally decided to drop everything and proceeded to create a new game framework. I tackled a basic game loop first, just setting up windows and getting a simple GL command to work. On Linux I started with EGL first (as I learned that from my native android app), but had to abandon that and move to glX instead — connected to xlib which provides keyboard and mouse support. I then ported that simple test to Android by using EGL and then translating touch commands appropriately. From there I added libPNG and then FreeType font loading support. Along the way I needed a matrix library — I found GLM for that. The only significant portion I’m missing is audio, which I don’t think will be hard. Networking looks like it just uses normal C sockets, so that should also be straight forward, although I’ll likely just cross-compile an abstraction layer; I’m just not sure which one.

I stress that this is a framework and not an an engine. It is simply a bunch of glue holding together other libraries which already exist. I’m not against game engines, but they are limited in focus. By using an engine you are bound to making only the type of game it supports. In contrast a framework merely offers several components which you need to pull together yourself. It is also easier to port to new platforms: independent modules are easier to adapt, interface, and rewrite for new targets.

So now I have everything in place and can focus on my actual game. I definitely find that overall this approach will save me a lot of time and save me from a lot of headaches. I get to code and test directly on Linux, and only occasionally install on the tablet to test. It also solves the incremental build problem: the time from a code change, to compiling and the executing is about 1-2 seconds.

I do intend on releasing this all under an appropriate free license. At the moment my focus is on the game, and the framework is still in a great state of flux.

I have a bit of trouble understanding why you chose not to go with Java and use libGdx with it. Was the main problem really with Java in the first place? It sounds through most of the blog entry that despite its various issues you were okay with using it as long as you could make it work warts and all

My original choice for Java was motivated solely because that is the primary SDK for Android. Any other platform and C++ would likely have been my primarry choice. So by the time I found libGDX I was probably already leaning towards C++. I knew what my game would need, and I could already envision it better in C++ than Java. The issues I encountered with the Java toolchain didn’t make the decision for me, but simply pushed me in that direction.

“… I consider it a significant failing of Java not to allow any kind of global typedef’s.”

I having trouble understanding what you mean. Would global typedefs make your code easier to write and read? Or would it make your code run faster? Cannot you just import classes to get rid of the prefixes?

I would make the code clearer, thus easier to write and maintain. Consider that if you wish to use the DialogFragment from the support package you need to import “android.support.v4.app.DialogFragment”. Note only is that long, the package names are quite arbitrary, especially the “v4″ part.

A global typedef would let you simply do “typedef android.support.v4.app.DialogFragment DialogFragment” and you could then use “DialogFragment” throughout your code. This also has the advantage that if you wish to then compile without using the support library you can simply alter the typedef instead of changing all occurences in the code.

I wonder why the Java compiler can’t just infer types and do the imports behind the scenes by itself. Of course conflicts would generate an error. Also it occurs to me that typedef would let you cleanly use two classes with the same name in one Java file.

Nonetheless I classify this sort of thing in the relatively minor annoyance category rather than significant failure. In that respect i agree with the original comment.

Finally I imagine it would not be hard to bolt on your own macro preprocessing step to do this kind of thing.

I’m not sure it’s just a minor annoyance in the Android SDK. Once you start using the support packages virtually every class is 5-7 levels deep in a package hierarchy. I also find the classes to be scattered through a variety of packages, thus * imports don’t help much either.