Signal processing in digital cameras notes

A typical video/digital camera will apply the following types of processing
to the image.

White Balance

To white balance the image, one or two of the RGB channels will be multiplied
by some number above 1 to balance out red, green, and blue. This will push
some values above white level and these values will need to be clipped for
the image to look right. So, white balance is a destructive process
that throws away image information. If white balance is re-applied in post,
then the first white balance may have thrown away useful information.

In a RAW scenario, white balance is not baked into the image so this type
of destructive manipulation is not applied to the image. Tricks such as highlight
recovery (explained in this article) can also be applied. It is essentially
recovering dynamic range that would otherwise be discarded.

Color Correction Matrix

An ideal camera would have the same color response as the cone cells in our
eyes. We typically assume that all humans have the same response as the 1931
standard observer (the average color response for a group of test subjects)
or some other similar reference. For any given color, the red, green, and
blue output should be at a certain level. Because the raw output from a camera
will not match the ideal color response, a color correction matrix
is applied to the signal.

A matrix is simply another way of writing out algebra. In this case, the
corresponding set of algebra equations would be:

The original RGB values go through the equations above, which will massage
the values so that the final colors look right.

Gamma correction

Human vision is more sensitive to variations in darker areas than for brighter
areas. Suppose there were a bunch of lights behind a piece of diffusion (e.g.
a silk, frosted window, whatever). We are better at detecting the difference
between 1 and 2 lights than 999 and 1000 lights.

Because of this, our video systems apply transfer functions (also called
gamma curves) to the video signal so that more bandwidth/bits are alloted
for shadows than highlights. This has the same effect as compression as it
throws away information so the entire video system doesn't have to deal with
as much information. (For whatever reason, most video engineers do not call
it or consider it compression.)

For HD formats, cameras are expected to apply the Rec. 709 transfer function
to the signal coming off the sensor. Before this function is applied, the
signal is in the linear light domain. The signal is proportional
to the number of photons of light hitting the sensor. After the transfer function
is applied, the signal is in the gamma corrected domain. The gamma
corrected signal is pretty close to how we interpret what we see and uses
bandwidth more efficiently. It is close to the opposite of the inherent transfer
function of a CRT (it has to convert the video signal into light). The two
come close to cancelling each other out, so the CRT doesn't have to apply
any signal processing to make its output right. (Also see Charles
Poynton's Gamma FAQ on "What is gamma correction?").

Why might gamma correction matter?

If you want to perform optically-correct compositing, the signal must
be converted into linear light before apply the compositing operations.
Things like glows and cross dissolves can look better when done this way.

Some cameras will cheat the signal processing and apply gamma correction
before the color correction matrix. This is the wrong order to
do things. The color correction matrix should be applied to the linear
light signal, not the gamma corrected signal. However, many cameras do
this because gamma correction allows them to reduce the bit depth. Applying
the color correction matrix after the bit depth has been reduced means
that the calculation is less computationally expensive. The errors are
very small unless the colors are extremely saturated, so this is a cheat
that is extremely difficult to notice.

Sharpness

Most video cameras tend to apply some sort of sharpness enhancement to the
image. This increases the perceived sharpness of the image. There may be side
effects such as the appearance of halos around edges. Note that actual resolution
does not increase. A good test of actual resolution would be to see how easy
it is to read fine text. An image may be perceived to be sharper than another
with higher resolution, but the image with higher resolution will let you
read finer text and see finer details.

When evaluating cameras for resolution, be aware that some cameras apply
more aggressive sharpening tricks and should not be confused with actually
having higher resolution.

Also note that any sharpening baked into the image can make keying and secondary
color correction operations more difficult.

Some cameras have the ability to reduce sharpening for flesh tones, which
will result in smoother flesh tones that may be more pleasing to the eye.

Video Knee

Most/all video cameras will compress highlights so that they will gently
roll-off between clipping. One way to do this would be to apply some sort
of curve to the RGB channels. Normally this would de-saturate the highlights.
So many video knee algorithms will apply their own (proprietary) recipe for
compensating for this effect.

On some cameras the video knee can be disabled. Sometimes the effect is desirable
and sometimes it is not.

Other image processing

There is also a lot of other image processing that may be applied:

Chromatic aberration adjustment.

Dead pixel correction.

Fixed-pattern noise removal.

For Bayer cameras, de-Bayering is necessary. For cameras that employ pixel
shifting, the equivalent process is needed.