RAW output information

I'm curious in the raw output option from the raspistill utlilty. I found a discussion about possible formats and a recent update that the function is now working, but I haven't found documentation about the output. Is this the Bayer output from the sensor? What is the bit depth?

AFAIK there is no documentation on this anywhere. Brand new and unexplored territory... you can be the first!

I've been looking forward to trying this once I get back to my Pi. But meanwhile, if you'd like to post a .jpg somewhere captured with the --raw option, I'd be happy to have a look at it! Might have to post it compressed as a zip, to avoid the raw-as-metadata getting munged by intermediate servers.

You can examine the structure of this file with programs like exiftool which has a special hex dump mode, as explained here. Using this tool you can see that the raw data (exiftool calls it "Unknown trailer") has been added to the end of the file, immediately after the JPEG End_Of_Image marker 0xFF 0xD9, extending for 6404097 bytes and starting with "@BRCM" and including the string "ov5647 version 0.1" and a bunch of 0x00 bytes. A little further on is the ASCII string "2592x1944" which is the full pixel resolution of the sensor. If the raw data block represents the entire sensor (not a 1920x1080 crop), that would be about 1.27 bytes per pixel, or just about 10 bits per pixel if it is a packed structure, pre-deBayer. The ov5647 chip has a 10-bit ADC, so that looks like it. Looking at the dump of another example, the raw2.jpg file above, the data block after the EOI marker is exactly the same size, 6404097 bytes so the RAW data has a constant size, as expected.

exiftool-hexdump.png (20.47 KiB) Viewed 10129 times

Last edited by jbeale on Sat May 25, 2013 6:05 am, edited 1 time in total.

Looking at the raw data block in a hex editor, after an initial block of 0x00 bytes I see what look like groups of 5 bytes; four bytes near 0x03 or 0x04 in value, then one larger say 0x55 or 0x4e. This pattern continues through the entire 6 MB block of raw data. The capture is an outdoor photo at night so the (jpeg) image is almost completely black. Now, if we had Red-Green-Blue values I would expect a 3-byte pattern. But if we have packed 10-bits per pixel, then groups of 8 bits are not the best representation. We see a repeating pattern of 5 bytes x 8 bits/byte = 40 bits. So at 10 bpp, we are seeing groups of 4 pixels? Bayer patterns are something like RGBG, so they do naturally form groups of four, although I'm not sure about the exact mapping in this case. Maybe it is some more abstracted representation, like one average intensity for each group of four pixels, and then one "delta" per pixel relative to that average. That encoding would offer an advantage if you were to later compress the raw data, although it seems to be stored here uncompressed given the strong patterns and conserved data block size.OK, maybe it is simple: first four bytes carry the high-order 8 bits of four pixels, and then the last two bits (LSBs) from each 10-bit pixel are packed together forming the 8 bits of the fifth byte. That would give us the pattern we see, and I now suspect that is the actual representation.

When you generate a "jpeg with raw" file using the '--raw' option to raspistill, the raw data is literally tacked onto the end of a standard JPEG file. This raw data block is always 6404096 bytes in size. The JPEG data varies in size with the image, but raw data block size is fixed, so you can obtain just the raw data by extracting the last 6404096 bytes of the file.

Of that raw data block, the first 32768 bytes is header data, starting with the string "BRCM" and ending with a lot of 0x00 bytes.

After the header is 6371328 bytes of (mostly) image pixel data, where each set of 5 bytes encodes four 10-bit pixels. The first 4 bytes are the high-order bits of 4 pixels, and the 5th byte contains the low-order two bits of each of the four pixels, packed together. The pixels have a Bayer pattern, so a de-Bayer process must be run to obtain R-G-B data. I haven't done the decode step yet, but there are only a few standard Bayer patterns.

The pixel data is arranged in the file in rows, with 3264 bytes per row. With 2592 image pixels across on the sensor, and 5/4 bytes per pixel, you might expect only 3240 bytes per row, but there are 24 extra bytes at the end of each row which encode some other, unknown information. Some of that may be masked-off ("dark") edge pixels used for calibrating the black level. (EDIT: I previously claimed 28 extra bytes per row but the above numbers now believed correct.)Now, with 3264 bytes per row x 1944 rows we have 6345216 bytes, that is the full 2592x1944 image area. The file data continues with 26112 additional bytes after that, the purpose of which I do not know.

This is "mostly" accurate, some offsets might be off-by-1. I believe the bytes per row is true, because when I view an un-debayered linear raw greyscale image with that stride, I see a true image appear without any diagonal offset per row.

RPi-FromRaw-Small.JPG (33.27 KiB) Viewed 9981 times

...even before debayer, a raw greyscale view shows that the image structure is there. The above greyscale image was generated by the 8-bit version of ImageMagick with

Note the offset of image data from the start of the JPEG file varies with each file, and had to be manually calculated. In this case it was (TotalFileLength:11112983 - RawBlockSize:6404096) + Header:32768 = 4741655

Last edited by jbeale on Sun May 26, 2013 6:02 am, edited 7 times in total.

jbeale wrote:The pixel data is arranged in the file in rows, with 3268 bytes per row. With 2592 image pixels across on the sensor, and 5/4 bytes per pixel, you might expect only 3240 bytes per row, but there are 28 extra bytes at the end of each row which encode some other, unknown information. Some of that may be masked-off ("dark") edge pixels used for calibrating the black level.

This is "mostly" accurate. I believe the bytes per row is true, because when I view an un-debayered linear raw greyscale image with that stride, I see a true image appear without any diagonal offset per row.

According to the product brief (see http://www.ovt.com/download_document.php?type=part&partid=171), the OV5647 has an image width of 3673.6 microns and a pixel pitch of 1.4 microns. This suggests that the image sensor is 2624 pixels wide. That would be 32 pixels beyond the 2592 required for a QSXGA image. I see that doesn't match your numbers, so I'm not sure the product brief information will help decipher this.

oops, someone already read the initial version of the post which was 4 bytes off per row, I just did an edit to fix it... anyway I'm just tweaking numbers until they look right, and then assuming they are.

You shouldn't need to calculate the offset in the JPEG manually - its stored as official JPEG metadata so there should be a offset defined in the file somewhere. Just need someone who knows how metadata is arranged in JPEGs

Volunteer at the Raspberry Pi Foundation, helper at Picademy September, October, November 2014.

jamesh wrote:You shouldn't need to calculate the offset in the JPEG manually - its stored as official JPEG metadata so there should be a offset defined in the file somewhere. Just need someone who knows how metadata is arranged in JPEGs

jamesh wrote:You shouldn't need to calculate the offset in the JPEG manually - its stored as official JPEG metadata so there should be a offset defined in the file somewhere. Just need someone who knows how metadata is arranged in JPEGs

I'm doing things fast-and-dirty right now. No doubt the likes of the 'exiftool' author could do things better; I have sent him an email. And if anyone else without a camera wants a look at some files (taken through a non-clean, double paned window; sorry!) here are a few:

A bit more progress. Testing code emits 16pbb PGM ASCII (hardwired for now to my test JPEG filename, for coders only!) easily converted to .png etc, but still no debayer yet. Still before debayer, but raw file seems to capture more detail as in crop examples of default JPEG and RAW images below:

#define LINELEN 256 // how long a string we need to hold the filename#define RAWBLOCKSIZE 6404096#define HEADERSIZE 32768#define ROWSIZE 3264 // number of bytes per row of pixels, including 24 'other' bytes at end#define IDSIZE 4 // number of bytes in raw header ID string#define HPIXELS 2592 // number of horizontal pixels on OV5647 sensor#define VPIXELS 1944 // number of vertical pixels on OV5647 sensor

// for one file, (TotalFileLength:11112983 - RawBlockSize:6404096) + Header:32768 = 4741655 // The pixel data is arranged in the file in rows, with 3264 bytes per row. // with 3264 bytes per row x 1944 rows we have 6345216 bytes, that is the full 2592x1944 image area.

Great stuff so far. The higher fidelity of output is another good sign and I'm really looking forward to getting a file format which can be imported into image editing software (such as Lightroom) for non-destructive editing of the raw files. Then maybe if we can get manual exposure control (already mentioned elsewhere) we'll really have the opportunity to open up a lot of avenues for exploration. Keep up the good work

Great stuff so far. The higher fidelity of output is another good sign and I'm really looking forward to getting a file format which can be imported into image editing software (such as Lightroom) for non-destructive editing of the raw files. Then maybe if we can get manual exposure control (already mentioned elsewhere) we'll really have the opportunity to open up a lot of avenues for exploration. Keep up the good work

+1I should receive a camera soon. Then I'll start a thread on manual exposure, see how far we come with the present feature set as per the suggestion of jamesh. I'm very curious to see how much IQ can be squeezed out of the little buggers.

Well, I managed to write code to convert the RPi "JPEG+RAW" file into Adobe DNG format so the raw data can be opened normally in UFRaw, Picasa etc. But I've got some things majorly incorrect given the colors have gone towards Andy Warhol (although bizarrely, a patch of grass is almost normal). I guess I can't just "guess" a color matrix, as well as the Bayer/CFA pattern Will try again later.

Great stuff so far. The higher fidelity of output is another good sign and I'm really looking forward to getting a file format which can be imported into image editing software (such as Lightroom) for non-destructive editing of the raw files. Then maybe if we can get manual exposure control (already mentioned elsewhere) we'll really have the opportunity to open up a lot of avenues for exploration. Keep up the good work

Remember that the incoming data has gone through many stages of processing to get to JPG from the original RAW data - lens shading, black level correction, debayering, denoise, sharpening, defective pixel correction, colour correction to list just a few. They are all there for a reason. You do lose some pixel fidelity to make a picture that actually looks right, doesn't have too much noise etc.

Volunteer at the Raspberry Pi Foundation, helper at Picademy September, October, November 2014.

jamesh wrote:You shouldn't need to calculate the offset in the JPEG manually - its stored as official JPEG metadata so there should be a offset defined in the file somewhere. Just need someone who knows how metadata is arranged in JPEGs

The easiest way is to take the file size and subtract the raw data block size as jbeale suggested. There is no offset stored in the JPEG that points to the EOI -- you need to parse through the entire JPEG image to be able to find the EOI.

jamesh wrote:Remember that the incoming data has gone through many stages of processing to get to JPG from the original RAW data - lens shading, black level correction, debayering, denoise, sharpening, defective pixel correction, colour correction to list just a few. They are all there for a reason. You do lose some pixel fidelity to make a picture that actually looks right, doesn't have too much noise etc.

Yes, but shooting in RAW format and making those adjustments yourself is the best way to get a higher degree of freedom from your images. If you're capturing in JPG then any changes you make to the image in post-processing (whether that's a crop, a brightness/exposure adjustment, a white balance adjustment etc) will degrade the quality of your output (because saving the image a second time after your adjustments have been applied means you get a 2nd generation JPG). However, with RAW files you can make any number of adjustments you want which are non-destructive, so when you save your output as a JPG you're saving a 1st generation JPG.

edit: RAW also (generally) provides a higher degree of dynamic range in the data, so you have more room for highlight/shadow recovery, for example, than you do with JPG files. This means in many cases you can 'save' an over-/underexposed image when shooting in RAW that you couldn't have done with JPG because the data has been dropped or severely reduced in the JPG encoding process.

amp88 wrote:If you're capturing in JPG then any changes you make to the image in post-processing (whether that's a crop, a brightness/exposure adjustment, a white balance adjustment etc) will degrade the quality of your output (because saving the image a second time after your adjustments have been applied means you get a 2nd generation JPG).

Besides usually the jpeg engine in cameras (even the €6000 flagship DSLRs) is not as good as a PC. Shooting a jpeg image with such a camera gives noticeably less image detail than when you shoot RAW and extract a jpeg on the computer; I heard people say this is due to floating point calculations.

Seeing the difference between the 1080p-based image from the early firmware that was scaled by the camera to 2952x1944 and the native 2952x1944 image from the new firmware makes me think that the little PiCamera particularly does not shine in this regard.

amp88 wrote:If you're capturing in JPG then any changes you make to the image in post-processing (whether that's a crop, a brightness/exposure adjustment, a white balance adjustment etc) will degrade the quality of your output (because saving the image a second time after your adjustments have been applied means you get a 2nd generation JPG).

Besides usually the jpeg engine in cameras (even the €6000 flagship DSLRs) is not as good as a PC. Shooting a jpeg image with such a camera gives noticeably less image detail than when you shoot RAW and extract a jpeg on the computer; I heard people say this is due to floating point calculations.

Seeing the difference between the 1080p-based image from the early firmware that was scaled by the camera to 2952x1944 and the native 2952x1944 image from the new firmware makes me think that the little PiCamera particularly does not shine in this regard.

You can change the JPEG quality you know...set it to 100, see if that improves things.

I think people are going to find out the long way how hard it is to make good images from the very RAW data coming out of the camera. Remember, some top notch experts did the camera tuning on the Raspi (although it could do with more)- this stuff doesn't happen overnight.

That said, I'm sure someone will spend the time on it.

Volunteer at the Raspberry Pi Foundation, helper at Picademy September, October, November 2014.

jamesh wrote:You can change the JPEG quality you know...set it to 100, see if that improves things.

I think people are going to find out the long way how hard it is to make good images from the very RAW data coming out of the camera. Remember, some top notch experts did the camera tuning on the Raspi (although it could do with more)- this stuff doesn't happen overnight.

That said, I'm sure someone will spend the time on it.

Setting the JPG quality higher does nothing really in comparison to having RAW files to work with. As said above, you get much more than just slightly higher quality files when you have access to the RAW data. Many/most serious photographers only shoot in RAW when using DSLRs or other cameras which support that output format for a wide number of reasons. I think we all realise that the camera board isn't intended to be used in the same way as a DSLR, but there are still some of us who would like to push it and see just how far it could go. RAW file output (which can be imported for use in software such as Lightroom/Photoshop) and manual exposure control are 2 of the biggest features which would put us a lot closer to that goal. Simply specifying higher JPG quality or choosing a different auto exposure mode (such as night or beach) does nothing to help in either of these regards, sadly.

However, I think we all also appreciate that these features are really on the fringe and that the overwhelmingly vast majority of Raspberry Pi users probably wouldn't care for them, so they don't deserve a particularly high priority in relation to other aspects.

Yes, and probably... finding out small bits at a time. Earlier I had assumed the Bayer filter on the sensor was R-G-G-B, but at the moment it seems closer to C-Y-Y-M (cyan, yellow x2, magenta) ...or maybe it's just a clipping effect. It will be a while before I can do some better color charts, and make an effort on the Raw->XYZ->RGB color transform. Presumably the tuning experts did all this already, but I don't know if their reference illuminant, color matrix etc. is public knowledge?

As people have been noticing, the all-round image quality as it stands is really quite good, and at this point I don't see any big improvements likely. Changing the tone curve after the fact may improve some highlights, but of course this sensor has 10-bit output, not 12 or 14 bits like DSLRs. In contrast-stretching very low light images, I see some fixed-pattern noise (vertical streaks) as well as random noise. One thing RAW may enable is a more consistent reference for black frame subtraction and flat-fielding, if you want to squeeze that last % of sensor performance.

jamesh wrote:You can change the JPEG quality you know...set it to 100, see if that improves things.

I think people are going to find out the long way how hard it is to make good images from the very RAW data coming out of the camera. Remember, some top notch experts did the camera tuning on the Raspi (although it could do with more)- this stuff doesn't happen overnight.

That said, I'm sure someone will spend the time on it.

Setting the JPG quality higher does nothing really in comparison to having RAW files to work with. As said above, you get much more than just slightly higher quality files when you have access to the RAW data. Many/most serious photographers only shoot in RAW when using DSLRs or other cameras which support that output format for a wide number of reasons. I think we all realise that the camera board isn't intended to be used in the same way as a DSLR, but there are still some of us who would like to push it and see just how far it could go. RAW file output (which can be imported for use in software such as Lightroom/Photoshop) and manual exposure control are 2 of the biggest features which would put us a lot closer to that goal. Simply specifying higher JPG quality or choosing a different auto exposure mode (such as night or beach) does nothing to help in either of these regards, sadly.

However, I think we all also appreciate that these features are really on the fringe and that the overwhelmingly vast majority of Raspberry Pi users probably wouldn't care for them, so they don't deserve a particularly high priority in relation to other aspects.

DSLR lenses are large enough, for example, that no lens shading correction is required which means that with RAW data you need to do that. That's one thing required that DSLR RAW's don't need, there are probably others. As jbeale said, after all the effort of sorting out RAW, the differences are going to be minor compared with the images you get right now!

(ps. My comment on JPG compression was based on the post quoted, not the benefits of using RAW)

Volunteer at the Raspberry Pi Foundation, helper at Picademy September, October, November 2014.