Thursday, August 23, 2012

Thanks for the Memories

(This tip and anecdote is specifically about Android, but the same technique applies to every platform I've ever worked on, which at this point is quite a few. In fact, it's a technique that Romain and I have stressed in our book and in most graphics/GUI talks that we've given over the years, such as this one from Devoxx in 2008) .
I was debugging an application recently (names withheld to protect the completely and utterly guilty) and discovered that the source of a serious performance bottleneck was simply the size of the bitmaps involved.

The application's job is to display lots of pictures, so using bitmaps is a given. And the size of the bitmaps being loaded and displayed is signficant, so there were going to be issues around memory and performance anyway. But it was the way in which the application was treating the source and destination sizes that was at the root of the problem.

In particular, the application was loading each image into a bitmap of size X x Y. Meanwhile, they wanted to draw that bitmap into a destination rectangle half that size, .5X x .5Y. This is easy to do; you just call the appropriate Canvas.drawBitmap() method with the relevant source/destination rectangles, or specify a scale on the Canvas object, and we'll take care of the details.

Simple.

But.

Wrong.

That is: Scaling on the fly is easy, and it works. But asking us to do your work for you on every frame in which you draw that bitmap might cost you significantly in performance and memory when there's a very easy way for you to do this once and simplify all future operations with that bitmap.

Here's the right thing to do: pre-scale the bitmap to exactly the size you need. Then when you need to copy it into the destination, you call Canvas.drawBitmap(left, top, Paint) (the version that doesn't take a dest rectangle) and then all that Android needs to do is copy the bitmap. Much simpler. And what's more: it requires potentially far less memory than downscaling to a smaller size.

This is probably (hopefully? please?) obvious when you're running in a software-rendering situation (e.g., all releases prior to 3.0, or any app targeting pre-4.0 releases and not specifying hardwareAccelerated="true" in the manifest); having the framework scale the image every time it's drawn means going through a much slower path than a simple 1-to-1 copy entails.

But what about on GPUs, with our wonderful new hardware-accelerated world of Android apps as of version 3.0+? Aren't GPUs supposed to be faster at stuff like this? What are we paying them for, anyway?

Yes.

But.

Here's the problem: the actual scaling operation is quite cheap on a GPU, even negligible. But that's not all that you should be concerned about as a mobile developer. Mobile developers should always worry about memory. You should profile your application. You should think about memory consumption at night when you can't sleep. You should bring it up on first dates*, and fester on it while on vacation.

If you're displaying several images per frame, you want to be very aware of how much memory those bitmaps are soaking up. This is true for the bitmaps in CPU memory, but also true for bitmaps that we upload to textures. Just because it's cheap for a GPU to scale a large texture into into a small space on the GPU doesn't mean it's fast to upload it into texture memory, or cheap for the GPU to have that large image sitting around in memory. Memory is a constrained resource and should be treated as such. I'm sure your date will tell you as much (possibly as they leave the date in search of more interesting prospects).

If you're going to scale from a 2k x 2k image into a 32x32 icon, wouldn't it make more sense to pre-scale it once, chuck the original one, and thenceforth deal with only the smaller version instead?

Of course, if you really need full-size images, then go ahead and do what you need to do. And if you're animating an image's size (such as zooming in on it), then pre-scaling to each intermediate size probably doesn't make much sense. But if you know that you're going to be using a smaller version for a while, then you should probably pre-scale to that size rather than drag around the memory and performance baggage associated with the original version, no matter what the hardware acceleration situation on the target device is like.

* The advantage of discussing memory consumption on first dates is not only that it will help you keep it in mind at all times, but also that this will inevitably lead to more first dates on which you can continue discussing it. Or it will at least result in less second dates.

Then I'm checking them with justDecodeBounds. Then I compute a sample size with which to downsample them. Then I use Bitmap.createScapedBitmap() to write the scaled bitmap out to disk, and recycle both the downsampled and the downscaled versions when I'm done. Is there a more memory efficient way of doing this?

If a view is around that's looking for that image, the view will load it (in the background, given disk i/o can cause latency) and cache using https://github.com/chrisbanes/Android-BitmapMemoryCache

Actually, I'm running a couple of different transformations, using a fixed thread pool, the size of which is the smaller of 2 or the number of cores in the system. I'm thinking of changing this to be based on the heap size, rather than the number of cores.

Thank you for posting. I totally agree with you, and undoubtedly your suggestion will help many people who haven't done a lot of imaging/graphics.

I have a question or two...

To scale BigImage down to SmallImage, is the best and most efficient way just to create offscreen bitmaps and use the source-rect/dest-rect version of Canvas.drawBitmap() from Big to Small? (After which you set BigImage = null and use SmallImage.)

Or are there more efficient ways to scale down/minify an image in Android?

I'm assuming that grabbing BigImage's raw pixels and manually scaling it down in Java code would be slower than Canvas.drawBitmap(), correct?

But aside from this, I was wondering if there are there any undocumented asynchronous graphics/imaging background services in Android we can use to perform such tasks? (I'm thinking that such a service could conceivably have highly-optimized native code to take advantage of multiple cores, threads, etc. and therefore be more efficient than Canvas.drawImage().)

About Me

I'm a software geek, working at Google, making Android graphics and animation more excellent. In previous lives I've worked at Sun on the JDK, at Adobe on Flex, and various other places in Silicon Valley, always working on graphics software.

In my copious spare time, I write. I write humor on Enough About You... and technical articles on CodeDependent. I co-wrote the book Filthy Rich Clients with Romain Guy, wrote another programming book Flex 4 Fun about Flex graphics and animation, and wrote the humor book When I am King.... and the long-anticipated sequel, When I am King... II. Like women and childbirth, I eventually forget the pain of the process of writing a book, and will probably make the mistake of writing another one eventually. As soon as the scars from the last one heal.

I also have developed a strange and disturbing attraction to the microphone. Any microphone. You may find me giving a technical talk at a developer conference or user group, or doing some standup or improv in a comedy show. I've also been seen in videos ("You may know me from such hits as DevBytes..."), either work-related or posted on my comedy blog and YouTube channel.

None of what I write in my blogs, on Google+, or anywhere else has anything to do with my employer; they're just my thoughts, my jokes, my mistakes.