Google is working on a new Android camera API, supports Camera RAW

Could a new API shore up Android's biggest weak spot?

The Nexus 5 camera was a huge disappointment, especially after comments from high-ranking Googler Vic Gundotra stating that "we are committed to making Nexus phones insanely great cameras. Just you wait and see."

That was nine months ago. We waited and saw, and what showed up on the Nexus 5 wasn't very good. There may be an explanation for this, though. According to commits in the public Android source code, which were first spotted by Josh Brown on Google+, Google is working on a new camera API for Android. Work on the new API started in December 2012, which would make it seem targeted for KitKat, but about a month before the new OS's release, the API was pulled from Android's framework code. The commit that removed the API from the release Android code is here, with the comment saying:

DO NOT MERGE: Hide new camera API.
Not yet ready.
Bug: 11141002

This commit was pushed on October 11, about a month before the release of KitKat. A month before release was probably "feature freeze" time, where work on new features stops and everyone focuses on fixing bugs in time for release. The camera revamp didn't make it and was replaced with the original camera API.

The really good stuff is in the initial commit, which contains tons of documentation about the new camera setup. There's a new API class called "Android.hardware.photography" (the current camera functionality lives under "android.hardware.camera"), and with the fancier name comes fancier capabilities:

Full-capability devices allow for per-frame control of capture hardware and post-processing parameters at high frame rates. They also provide output data at high resolution in uncompressed formats, in addition to compressed JPEG output.

The new camera API has a backward-compatibility mode for older devices, but "full-capability" devices now have access to a few new picture formats. The only new image format listed that isn't present in Jelly Bean seems to be support for camera RAW:

General RAW camera sensor image format, usually representing a single-channel Bayer-mosaic image. Each pixel color sample is stored with 16 bits of precision.

The layout of the color mosaic, the maximum and minimum encoding values of the RAW pixel data, the color space of the image, and all other needed information to interpret a RAW sensor image must be queried from the {@link android.hardware.photography.CameraDevice} which produced the image.

Smartphone cameras normally output JPEG files, which are compressed, mostly finalized images. RAW is minimally compressed and unprocessed, so shooting in RAW gives the photographer much more flexibility after the picture is shot. Programs like Photoshop can do much more with a RAW file than with a JPEG.

Camera RAW is not totally unheard of on a mobile phone; Nokia's upcoming Lumia 1520 will be able to shoot RAW, for instance. Besides making Photoshoppers very happy, the RAW file could be passed to an even more powerful on-board photo editor, which Google seems very keen on building into Android and Google+.

The new API also supports face detection. This feature includes bounding boxes around faces and center coordinates for the eyes and mouth. In addition to the face-focus capabilities, the system can assign unique IDs to each face (provided they stay on screen) so developers could do things like assign silly hats to multiple faces in a video feed. While you may have seen face detection on some Android devices, those were all solutions built by Android OEMs.

There's support for burst mode, too—another feature that you would swear was already included in Android but isn't. On Nexus devices, the only "burst mode" involves the user pressing the shutter button really fast.

The camera device is removable and has been disconnected from the Android device, or the camera service has shut down the connection due to a higher-priority access request for the camera device.

The strangest new feature is probably support for a removable camera. We can't recall a single Android device of any kind that has had a removable camera, so feel free to leave your suggestions in the comments.

The most important possible improvement that wouldn't be visible in the source code: image quality. Android cameras arguably lag behind the iPhone in quality, so this new API may be Google's solution to that problem. Android's subpar image quality seems to be an across-the-board problem, so maybe the issue really is as low-level as the camera API. There's no way to be sure, though, until we get finished software and devices in our hands. With documentation using phrases like "substantially improved capabilities" and "fine-grain control" it certainly sounds like Google is out to fix Android's digital-imaging woes.

Google bought Nik Software recently. They specialize in image processing software which plugs into Lightroom, Aperture and Photoshop, and released Snapseed for iOS and Android. My guess is that they'll look to incorporate Nik software into Android.

> The strangest new feature is probably support for a removable camera. We can't recall a single Android device of any kind that has had a removable camera, so feel free to leave your suggestions in the comments.

I have to wonder how much help RAW really will be for a small sensor camera. There are a couple of advantages to saving RAW images and post-processing. One is that DSLR-grade sensors have more dynamic range than what can be easily saved in a 8-bit JPG file, so post-processing can allow you to recover details that may have otherwise gotten clipped in the highlights or lost in the shadow noise. Small sensor chips don't have nearly the same dynamic range, so you don't have the information to start with.

The other main advantage of RAW, which would pertain for a phone camera, is that the white-balance corrections aren't baked in (as they would be for JPGs). That makes color-correcting for different lighting a lot easier. If you don't care about that sort of thing (and I imagine most people just keep white-balance set to 'auto' and forget about it), RAW doesn't really buy very much.

Google bought Nik Software recently. They specialize in image processing software which plugs into Lightroom, Aperture and Photoshop, and released Snapseed for iOS and Android. My guess is that they'll look to incorporate Nik software into Android.

Snapseed was on iOS (not sure about Android) long before Google bought it.

Removable doesn't necessarily imply that a camera is a modular component of a phone or tablet. I think it might simply mean that a camera that is available to the system, via a wireless connection, can be either disabled, or go offline, due to something like being outside of wireless range or the host device having a higher priority.

So for example if a camera in a smart-watch, or Google Glass, can be remotely operated by say an Android phone, then the camera is removable in the sense that it won't always be available to the phone.

One of my coworkers is a fairly serious amateur photographer, and is looking forward to using a Nexus 7 as a remote display.

Perhaps the "removable camera" is related to such? An improved interface (either from Google or manufacturers) for remote shutter, color manipulation, light balance, or whatever it is photographers do when staging a picture?

I have to wonder how much help RAW really will be for a small sensor camera. There are a couple of advantages to saving RAW images and post-processing. One is that DSLR-grade sensors have more dynamic range than what can be easily saved in a 8-bit JPG file, so post-processing can allow you to recover details that may have otherwise gotten clipped in the highlights or lost in the shadow noise. Small sensor chips don't have nearly the same dynamic range, so you don't have the information to start with.

The other main advantage of RAW, which would pertain for a phone camera, is that the white-balance corrections aren't baked in (as they would be for JPGs). That makes color-correcting for different lighting a lot easier. If you don't care about that sort of thing (and I imagine most people just keep white-balance set to 'auto' and forget about it), RAW doesn't really buy very much.

One major advantage of RAW available at the API level would be for anyone who isn't entirely thrilled by the default software's (usually mediocre) munge-to-JPEG function.

As you say, RAW shots from some teeny little sensor arent' going to be anything like RAW shots from a DSLR when you load them into lightroom or whatever and start futzing around; but if 3rd-party photo apps can get RAWs directly off the sensor, it'll presumably make life a lot easier for (quite possibly more competent) outfits to plug in their bake-to-JPEG mechanisms of choice, in place of Google or OEM ones.

You'd have to be slightly obsessive (or using Android-based firmware on something that is more camera than phone), to care about getting and storing the RAWs of your happy-snaps; but it would be much less of a surprise if Adobe could pop out a camera app that grabs the RAW api data and does a better job by default, with options to twiddle, of spitting out jpegs...

One can only hope that they are also looking at the firmware of their Nexus devices as part of this. The focus delay on the NExus 5 isn't quite as awful as some are making out, but it's especially jarring given the speed of the device in all other aspects.

Or possibly multi-positional cameras like the one in the new Cyanogenmod phone. Since the camera physically rotates front-to-back and vice versa, I imagine when it's in the other position you've essentially removed the camera from the previous position.

All in all it's good that this is being worked on. EVERY single article on a new Android phone pans the camera compared to the iPhone, even if every other single aspect is better than a similar product from Apple. For many people it's a dealbreaker, and I imagine for anyone coming from Apple to Android, it would be a bit jarring to have pictures suddenly look noticeably worse.

I take it nice new software would still leave a lot in the hands of the hardware manufacturer? (i.e. HTC good, Sony not bad either, Samsung a bit iffy?)

It's probably more accurate to say "Sony is generally good, HTC generally average except for low light, and Samsung goes from great to terrible" but yes, there would still be a lot in the hands of the OEMs.

It might give OEMs that care, though, more freedom to make better software. Apple's cameras aren't, in hardware terms, any better than what Sony or Samsung ship, but Apple sweats the details better. Assuming the OEM cares,

Unless the sensor is very high quality IMHO there will be very little benefit in shooting RAW.

As noted in the article, it gives photographers more flexibility and control over their photos regardless of sensor quality - producing (potentially) better photos even with the same sensor. This would be especially useful to things such as snapseed or Google+ auto awesome, allowing these programs more control over the final image.

For your average snapshot going straight to facebook or instagram, it probably won't make much difference - though if you had the space to work with the RAW, even temporarily, you could do the common 'hipster' filters easier and better pre-JPEG, along with auto-improve tools having more data to work with and likely giving better results.

Being able to better fiddle with white balance and under-exposure post shot would be a great boon for 'keeper' photos, especially if you only got one chance at the shot (pets or children!) and you don't have your DSLR with you, or don't own one.

You also have the option of going to something other than jpeg from RAW - JPEG2000 springs to mind.

One use for me personally would be improving the shots from a cheap light-box + 6500k lamps - I sometimes need to take photos of home-made jewellery, or other small stuff (sometimes it's going on ebay or the like, or just showing it off to family) and I can't justify the cost of a DSLR for that use case (and don't do enough photography otherwise). My HTC one beats the pants off my old point-n-shoot - but even fiddling with the white balance + exposure hard in CS6 (academic copy) or aperture, it's hard to get good shots out of JPEG, as there's only so much you can do.

Being able to postprocess RAW would drastically improve that - I was totally shocked at how much you can improve mediocre RAW shots when I did get to use a DSLR for a while, so even with a small sensor I think it would be a big improvement, even if the space requirements keep it as a somewhat niche use case.

If that function (along with burst mode!) came to say, android 4.5, I would totally convert my HTC one to a GPE version to get it quicker.

> The strangest new feature is probably support for a removable camera. We can't recall a single Android device of any kind that has had a removable camera, so feel free to leave your suggestions in the comments.

Project ARA.

Bingo, I was going to mention that too. This is required for that to be swappable. Otherwise how does the OS know the hardware state changed? The Linux layer probably would of course, but not the Android components above it.

Unless the sensor is very high quality IMHO there will be very little benefit in shooting RAW.

As noted in the article, it gives photographers more flexibility and control over their photos regardless of sensor quality - producing (potentially) better photos even with the same sensor. This would be especially useful to things such as snapseed or Google+ auto awesome, allowing these programs more control over the final image.

Exactly. The phone has a weak processor and must be somewhat optimized for speed - this is even true on a DSLR. RAW allows you to all of the compression, noise removal and color balance in post - where it doesn't matter if it takes 30 seconds to process and the quality is much higher.

The Nexus 5 camera was a huge disappointment, especially after comments from high-ranking Googler Vic Gundotra stating that "we are committed to making Nexus phones insanely great cameras. Just you wait and see."

You can say that again. I bought the Nexus 5 as soon as it was released (i.e. before any reviews were out) partly based on this. I knew the specs were only 8MP and I wasn't really expecting an insanely great camera, but I figured as long as it was at least on-par with my thirty month old Galaxy S2, I would be happy enough.

It isn't even close. I don't mind the lack of settings and cool features, so much as that the photos are pretty dire outside of HDR+ mode, and you can't use zoom within HDR+ mode. Taking quick "take it or miss it" photos doesn't seem possible (the whole point of a camera phone, IMHO). And the controls are fiddly.

I am hoping that it is all a software problem and that the hardware is fine... and that I will see improved software soon (regardless of whether it uses a new API).

Oof, just imagine your monthly bill if you switch to RAW output, but left your phone's settings to auto-upload your images to G+.

Fortunately most carriers at least here in the US don't charge data overages anymore. They'll just throttle you down to 2g.

Huh? Unless you are grandfathered into an old "unlimited" plan, or are using a pre-pay plan from a smaller provider, virtually every single carrier in US charges overages. At least in my recent research.

Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in Android OS and Google products. He is always on the hunt for a new gadget and loves to rip things apart to see how they work.