The Imaging Resource Test SuiteMore than you probably wanted to know about testing digital cameras

NOTE: This is a pretty BIG document, with more detail than most
will want to wade through. For those who want to skip ahead, here's a guide:
The beginning is mostly philosophy. Here's a link
to a discussion of our testing rationale. Click here
for a description of what to look for in the various test shots.

Introduction
The main purpose of the Imaging Resource web site is to provide people with
resources for making intelligent purchase decisions about digital imaging devices
(digital cameras and scanners). Since The Image is The Object (tm) (that's such
a cool phrase we decided we should trademark it), a lot of what you need to
know is how good a job the camera or scanner does of capturing the image. What
better way to decide whether you'll be happy with a camera or scanner than by
actually looking at images captured by it? (A novel concept, no?)
To this end, a major part of our product-testing and presentation involves
capturing images for you to use in evaluating the various devices. Look
at them on-screen, download them to print on your own printer, view them
however you intend to view the images from the product you'll eventually
purchase. Looking at images won't do much for you though, if you don't know
what to look for. Likewise, it would help to know how the various shots
were captured, and what the rationale was behind the whole test series.
Over the months that the site has been up, we've fielded a wide range of
questions about our test procedures, the images themselves, and how this
all relates to a camera's eventual performance after you own it. This page
sets out the many considerations that went into developing our test suite,
and offers a guide to understanding and making the best use of the images
we've captured.Outline
To save at least some of you from reading this entire tome, here's a brief
outline of how it's organized: We'll begin with some philosophical background
on digital camera testing, which will provide a perspective on how we arrived
at the mix of test targets that we use. Next, we'll discuss the key performance characteristics of digital cameras, and
how to test for them. Finally, we'll present an image-by-image
guide to evaluating the results of each test image in our standard suite.Testing Philosophy I: Relevant, Reproducible, and Revealing
The first objective we set for ourselves in establishing a test suite was
to make the images as relevant as possible to our readers' eventual usage
of digital cameras. While we clearly can't test for every possible situation
users will encounter, we want our testing to show how the cameras might
perform in typical situations, or with typical subjects our readers will
encounter in "real life."
Regardless of how relevant our tests are to the ultimate end-user experience,
we also need to insure that the results are reproducible. If we can't control
the test conditions from one camera to another, any results we obtained
will be meaningless and less than worthless: Images shot under different
conditions could actually mislead people as to the cameras' actual performance
relative to one another.
Finally, our test shots need to be revealing of the cameras' characteristics:
There'd be no point to capturing all these test images if they didn't tell
us something useful about each camera's operation and behavior.We Live in an Imperfect World
(Since this segment is about test philosophy, we figured we could get away
with a section heading like that.) We unfortunately have to accept that
it's impossible to accomplish our three goals in a single image or even
a set of images. At the most fundamental level, the very need for control
creates an artificiality in our testing that won't reflect your actual applications.
Digital camera owners use their cameras under all sorts of conditions, with
an enormous variety of subject matter, shooting conditions, lighting, etc.,
etc. Our need to lock-down every possible variable is immediately at odds
with how people will use the cameras in "real life." Likewise,
we can't count on any single image to tell us everything about how the camera
will behave under widely varying conditions. Bottom line, we're forced to
compromise, extracting some bits of information from one image, others from
another, and so on. The result is that some of our subjects are fairly "natural"
in that they involve a live person (or actual house), under a real sun in
the real sky, in the bona fide outdoors. Unfortunately, these most "realistic"
images will also be the least controlled, and hardest to compare from camera
to camera. At the opposite end of the spectrum, our "laboratory"
shots are completely controlled, but won't represent "real life"
camera usage as closely.We're Imperfect People
After all this self-righteous ballyhoo about how careful we are to control
for every variable, we're sure some readers will be moved to point out the
many situations in which our control has faltered. (Such as in the imperfect
and highly variable framing in our outdoor model shots - We're working on
it) While we absolutely welcome input, understand that we realize some of
our shots aren't perfect: It's part of being human, and having just so many
hours in the day, and (often) just so much time available with the camera
before we have to hurry it back to the manufacturer. The entire site is
a work-in-progress Don't let this apologetic justification of our faults
keep you from pointing out downright mistakes though: We're perpetually
indebted to our many readers who've taken the time to email and point out
numerous minor (and some major) quality-control slip-ups!One Solution: Posters
One of the more unusual things we've done in the impossible quest for control
and relevance is to take pictures of pictures: The "house" and
"musicians" test images are actually pictures of posters, shot
in the studio under carefully-controlled lighting conditions. This lets
us show some typical subject matter, but shot in a way that guarantees absolute
uniformity from one exposure to the next. In the case of the "musicians"
shot, it makes it practical to show how the cameras handle (reasonably representative)
skin tones from white, black, and oriental models, with good control, and
without having to assemble a studio full of models (an unbelievably expensive
undertaking) every time we need to test a camera. Note though, that there
are some obvious limitations in this image: In particular, fine details
such as hair texture and clothing patterns won't be anywhere near as fine
as in the original scene. Likewise, the "flesh tones" really only
roughly approximate how real skin reflects and colors light. In the case
of the "house" poster, both the detail and the tonal range (from
highlight to shadow) will be considerably less than in the original scene.
Do these factors mean that the poster shots are invalid, and shouldn't be
used to represent the cameras' capabilities? Not at all, but they do warn
us to be careful how we interpret the results, to look to other test subjects
to fill in the gaps, and to rely on them primarily for relative comparisons
between the cameras. (We'll talk about each of these test targets in greater
detail below.)Another Solution: Purely Analytical Targets
A more conventional approach to determining camera performance is to use
"scientific" test targets that have little or nothing to do with
actual shooting conditions, but which provide very reliable absolute measurements
of camera performance. In this vein, we use the "WG-18" resolution
test target, the "Davebox" composite target, and the viewfinder
accuracy target. All of these test one or more operational parameter with
absolute reliability.The Third Leg: "Real" Subjects
Which brings us to the remaining category of our test subjects, the "real
life" targets. There are some things we simply can't measure in the
studio, and that require a certain acceptance of chance variation and uncertainty.
Tests in this category include both the indoor and outdoor model shots,
and the outdoor "far field" optics test. (The outdoor, highly-variable
version of the "house" shot.)What to Test: The Key Parameters
So, after all the philosophy about the how and the why, we need to think
for a bit about what it is that we're trying to test for. Here's a short
list of critical camera parameters, along with some discussion about how
we set about testing them:Tonal Range
The ability of a camera to accurately render a wide range of tone from light
to dark is one of the most critical imaging functions. Surprisingly, this
is a much more difficult task than most of us would assume. It sounds simple,
doesn't it? Just measure how light or dark each portion of the image is,
and assign a numeric value to it. The problem is that our eyes don't respond
proportionately (or "linearly") to light, but compensate for brighter
and darker scenes. Also, the final output medium, whether CRT or printed
page, can't remotely approximate the brightness range of most real scenes.
The result is that the range of subject brightness values typically has
to be compressed to fit into the range that can be reproduced by the screen
or paper we view the final image on. The art lies in how this compression
is done, and how successfully it conveys an impression of what the original
image looked like.
Digital imaging is also very unforgiving of over- or under-exposure. Conventional
film (especially color negative film) has a tremendous exposure latitude,
meaning that it responds predictably to a tremendous range of brightness
values. If the negative is a bit dark or light, no matter: It's easy enough
to adjust the exposure in the printing process to make the final print look
OK. With digital devices though, we're restricted to an absolute (and fairly
small) range of numeric values. (In most cases, the numbers available for
representing brightness run from 0 to 255.) If any part of the subject is
either brighter or darker than the values corresponding to the ends of our
number scale, we lose all detail in that area. All the digital camera can
tell us at that point is "it's white" or "it's black",
and any variations in brightness are lost.
In order to not lose any detail, you'd think we'd want the camera or scanner
to make sure that no part of the image ever went completely "white"
or "black." It's true that this approach preserves the maximum
tonal detail, but the resulting pictures look very dull and flat, with no
contrast. Besides low tonal contrast, such images will also show very dull
and "muddy" color. (In fact, what appear to be color problems
on many digital cameras are often just the result of poor tonal reproduction.)
The key is for the camera to make the maximum use of the available range
of brightness numbers it has to play with, without losing key subject detail
in the process. In practice, for things to look good, the camera is almost
always going to have to throw away some of the tonal information, the question
is how much, and how noticeable it is in the finished image.
To test a camera's tonal range, we ideally want several subjects, varying
from high to low contrast. At the extreme, we need a subject with a really
wide range from lightest highlight to darkest shadow. This will tell you
us well the camera can handle extreme situations. Not all real-life images
will encompass such a broad range however, and we also want to understand
how the camera's electronics process more normal images as well. There are
two targets in the Imaging Resource test suite that we commonly examine
for tonal detail: For extreme tonal range, we use the "Davebox"
target, which provides us with everything from direct reflections of the
photo lights (off the shiny pot lid) to the "black cat in a coal bin"
shadow-detail test of the charcoal bricks in the black box. For a more moderate
tonal range, we look to the "house" poster, evaluating how well
the camera holds the strong highlight of the white painted surfaces, and
how well it handles the dark area under the trees to the right.Color Purity and Accuracy
After tonal reproduction, color reproduction is probably the single most
important digital camera characteristic. To adequately discuss color reproduction
though, we need two key vocabulary words: Hue, and Saturation. Hue refers
to what color we're talking about. In other words, hue tells us where along
the rainbow a color would fall. Conversely, saturation refers to how much
color is present: A monochrome image such as a black & white photograph
has zero saturation, while bright primary colors have very high saturation.
The reason we've made such a distinction between hue and saturation is that
camera or scanner errors in these two areas can have very different consequences
for your images. In general, errors in saturation are quite easy to fix
in image-editing programs. (As mentioned earlier, correcting tonal errors
frequently fixes saturation problems as well, but most image-editing applications
have a separate saturation control as well.)
Hue errors are another matter entirely, though. Overall hue error, or "color
cast" in images can usually be adjusted out fairly easily with most
image-editing programs. By contrast, errors in color accuracy, or "color
contamination" are virtually impossible to get rid of. Color contamination
refers to situations in which primary colors aren't captured as pure primaries,
but rather are "contaminated" with varying amounts of the other
colors. As a simple (but artificial) example, suppose we had a "pure"
red, which our camera should capture as RGB values of 255, 0, 0. (That is,
the maximum possible value in the red channel, and nothing at all in the
green or blue channels.) If our camera has poor color filtration though,
instead of the perfect 255, 0, 0, we might actually see something like 180,
30, 25. This means that significant amounts of green and blue have crept
into what should be a pure red, turning it dull and brownish.
Looking at the above example, you might ask: "Why can't we just dial-down
the green and blue channels? That would clean up the red, wouldn't it?"
It is true that you can adjust any single color this way to come out looking
however you'd like: The problem comes when you have to take other colors
into account at the same time. If we try to fix the red we're talking about
by taking out the offending green and blue, the effect will be disastrous
on any green or blue objects that might be in the same picture.
The bottom line of this is that it's fairly important for a digital camera
or scanner to produce "pure" colors. Testing for this is fairly
easy: Simply include at least one test subject with very bright, pure, primary
colors, and see how well they're reproduced. (Technically, to isolate color
accuracy errors, you have to adjust the tonal range of the captured image
first, to make sure you're not being confused by tonal errors. In practice,
looking at the MacBeth target in the Davebox images, it will be pretty apparent
how each camera does in terms of color accuracy, without going to any extraordinary
lengths.)
Actually, because of the extensive color processing digital cameras do before
you ever see the image, you need to look at not just primary colors, but
a range of colors at varying saturation levels. The MacBeth target includes
a limited range of such colors, while the smaller Q60 target shows a much
broader spectrum.White Balance
So-called "white balance" is one area where digital cameras have
a huge advantage over film-based ones, making it an important parameter
to consider. Any light source has an inherent color associated with it,
but our eyes are incredibly adept at compensating for color casts, tending
to balance things out to "white" across an amazing range of illumination.
Film has no such luxury, and will always respond to incoming light the same
way, and isn't able to adjust to compensate for the differences between
daylight and incandescent lighting. Thus, to get correct color from a "daylight"
film under incandescent lighting, you need to use a strong blue filter to
compensate for the heavy yellow cast of the light.
With digital cameras though, the electronics can compensate for variations
in lighting, simply by adjusting the "volume control" (gain) for
the red, green, or blue channels. Appropriately enough, this adjustment
process is called "white balance," and most digital cameras perform
it automatically.
You'll find substantial differences between cameras though, in how effective
and/or accurate their white balance is: When shooting the same subject under
widely varying lighting conditions, how consistent are the colors? Do whites
stay white, and colored subjects retain their hue and saturation? For many
people, white balance is a judgment call, in that some prefer the camera
to leave some of the color cast of the original lighting in the final image,
preserving more of the "feeling" of the shot. Others prefer color
casts to be completely neutralized. The shots at right show a fairly extreme
range of white-balance responses to the same lighting condition.
There's a wrinkle to automatic white-balance correction though: How does
the camera know what should be "white?" In many cases, it's easy
- just find the brightest thing in the picture, and adjust the red, green,
and blue levels until it's pure white. But what if there aren't any pure-white
objects in the scene? Or, what if the subject has a strongly dominant color
of its own? - Things get tricky! In fact, some shots will absolutely "fool"
the white-balance circuitry, and produce a significant color cast every
time. (Our own "musicians" test shot seems to do this to many
cameras, although the responses run the gamut from warm to cool color casts.)
What do you do then? If the camera only provides an automatic white-balance
function, you could be stuck! Some more-sophisticated units allow you to
"bail out" to a manual white-balance adjustment, usually giving
you the choice between several preconfigured settings, such as daylight,
incandescent, fluorescent, etc.
So how do you test for white balance? Our indoor, non-flash portrait shot
provides a good example of typical indoor residential lighting conditions,
so you can see how cameras handle this relatively extreme (but common) situation.
In the other shots, you can see how the camera reacts to a range of subjects,
although most of these are reasonably well-balanced in their color content.
(As noted earlier, the "musicians" image appears to be a bit of
an exception, in that we've noticed rather unusual behavior from several
cameras with it.)Resolution
Resolution is perhaps the most misunderstood and distorted of all digital
camera/scanner characteristics. Cameras are frequently referred to in terms
of the number of pixels they produce in their final files, and people generally
think of the pixel count as the camera's "resolution." Unfortunately,
resolution and pixels have only a passing relationship to each other. Think
of it this way: A blurry 8x10 photo print may not be "better"
than a sharp 4x5 one. Likewise, sharp 1024x768 pixel images may actually
look better than blurry 1280x960 ones. The only reason there's any relationship
between pixel count and resolution is that all the manufacturers are pushing
the image size/sharpness tradeoff as hard as they can, and all more or less
equally.
Really, resolution ultimately comes down to how much detail you can see
in the image, which should be a fairly easy thing to quantify: Just shoot
a subject that shows progressively finer detail, and note the finest detail
you can actually see. Unfortunately, this straightforward approach is complicated
by the realities of digital imaging, with the dual factors of aliasing and
JPEG image compression contributing to the confusion.
We probably should take a brief sentence or two to explain what the term
"aliasing" means. Basically, aliasing is what happens whenever
a sensor doesn't have enough resolution to reproduce the finest detail in
a scene. Rather than seeing just the image, what you end up seeing is some
combination of the image and the sensor pixels themselves. Aliasing shows
up in images in two ways: First, as "jaggies" or "stairsteps"
caused when smooth lines cross pixel boundaries, and the edges of the pixels
become apparent. The second form of aliasing occurs when abrupt contrast
changes in the subject interact with the "striping" of the color
filters on the camera or scanner's sensor and produce colored artifacts.
In combination, these two effects can make it difficult to define exactly
when a camera or scanner runs out of steam in resolving fine detail. (The
image at the right shows an example of fairly severe aliasing.)
The second conundrum in measuring resolution arises from the effect of JPEG
compression on the image data. (JPEG compression is only an issue in digital
cameras, since scanners don't inherently compress their images.)The JPEG
algorithm compresses image files by removing information that it considers
redundant or unimportant. In practice, it does a fairly good job of preserving
detail in image areas having strong contrast, but is more aggressive about
throwing away data when the contrast isn't as strong. This means that you
ultimately have to rely on your own eyeballs in judging resolution, based
on how well the camera responds to the sort of subjects you're likely to
be shooting. In the Imaging Resource test suite, we include a standardized
resolution test target, but also refer to some of the "natural"
images to reveal camera performance.
A final (and IMPORTANT) issue in evaluating resolution is to take
into account the effect of different pixel resolutions in the on-screen
displays: All the test images on this site appear on-screen at a 1:1 pixel-to-pixel
size. That is, each pixel in the image will occupy a single pixel on your
display. This means that images from cameras with higher pixel-counts will
appear larger on the screen. When comparing images visually, an image from
a lower pixel-count camera may appear "sharper", because an equivalent
portion of the test subject is spread across more screen pixels in the shot
from the higher pixel-count unit. Thus, the only really reliable way to
evaluate resolution between cameras (unless they just happen to produce
files with the identical pixel dimensions) is to download the images and
print them at the same size on your printer: This will really be the best
"apples to apples" comparison, and also factors-in the issue of
how your particular printer responds to images from various devices.Image Noise
If you were to capture an image of a perfectly smooth gray object with either
a digital camera or scanner, you'd find that the individual pixels don't
have identical brightness or color values. This random variation from the
"ideal" value is called noise, and can be an image-quality issue
for some devices. Noise in images is generally the biggest problem when
you're trying to pull significant detail out of extreme shadows, by adjusting
brightness and contrast values after the shot is taken. Noise can also appear
though, as unwanted graininess in areas of uniform color. Of our various
test targets, the Davebox will be the most useful for those of you who want
to evaluate noise characteristics: The flat color swatches of the MacBeth
chart are excellent for seeing noise in the individual color channels, while
the very deep shadows of the charcoal bricks will show how well the camera
does in low light. On our scanner tests, the color and grayscale swatches
in the Q60 target will perform the same function, and the "train"
slide presents a very severe test for slide scanners.Lighting!
(Warning - heavy techie-talk ahead.) Obviously, lighting is critical in testing
digital cameras: In order to insure that our studio shots are absolutely fair
to the cameras involved, we had to find a "daylight" lighting source
that conformed to international standards for "white" light. After
much research, we found what we were looking for in the "Solux" lamps
from Tailored Lighting, Inc. ("TLI" for short, at www.solux.net). While TLI makes pre-assembled units
calibrated to run at the 5500K "standard daylight" for photographic
film, cost factors led us to build our own lighting setup using their bulbs.
(We needed 14 lamps to get even enough illumination across our large posters,
and frankly couldn't afford the $160 unit price of TLI's assembled units.) We're
currently running the 4700K Solux bulbs (the color temperature here is specified
at their design rating of 12 volts) at 15.9 volts, in self-constructed shrouds
that make sure only the spectrally accurate output from the lamps reaches the
target area. At 15.9 volts, the lamps put out an exact match to the ISO 7589
standard daylight spectral curve, giving us a precise and absolutely repeatable
"daylight" to photograph under. The lamps are supported on a rectangular
framework 5 feet tall by 7 feet wide, with five lamps top and bottom, and two
on each side. The entire framework is positioned about two feet away from the
3 foot by 5 foot target area. This arrangement produces a uniform lighting level
of EV14 (1400 lux or 130 footcandles) across the entire target area. Another
really great characteristic of the Solux lamps is that they produce very little
IR to confuse the sensors in digital cameras. (Most of the IR is cleverly shunted
out the back of the lamp, through the dichroic coating on the lamp's reflector.)
The image at right shows a Solux bulb: Note the blue highlights,
and how yellow the light shining through the reflector is. This is how the
Solux bulb converts light from an incandescent filament to an accurate daylight
balance - It "throws away" much of the yellow light out the back
of the bulb, reflecting more of the blue end of the spectrum toward the
subject. TLI sells the bulbs as a matter of course, and can probably be
coaxed into selling the special sockets as well (as they did for us), even
though this isn't their main interest. Overall, if you intend to do this
sort of thing commercially, you're probably better off getting their pre-assembled
units for $160 each. A word of caution though: The Solux' (or any other
incandescent lamp's) color spectrum is a fairly strong function of the filament
voltage. You'll thus need to provide some sort of voltage regulation in
order to achieve truly repeatable results. If you're looking for a very
accurate "daylight simulator" at a reasonable price, we'll happily
plug TLI's bulbs - they've worked great for us!Looking at the Test Images
To those of you who chose to skip the preceding section, welcome back! (To
the hardy souls who made it all the way through, congratulations!) We're
now ready to talk about the individual test images: Why we chose them, and
what to look for in them...Outdoor Portrait
We generally list this image first in our picture index pages, both because
it's fairly representative of a "typical" subject, and because
it provides a good, quick check of many of the camera characteristics we
discussed above. First of all, this shot is a very harsh test of a camera's
ability to handle tonal extremes, from the very strong highlights in the
model's shirt, to the shadows among and underneath the flowers and their
foliage. Look to see if there's visible detail in both highlights and shadows:
If you find it, the camera has a broad tonal range.
This picture is also good for evaluating color accuracy: While none of the
colors in the subject are scientific standards, you can very quickly get
an idea of how well different cameras reproduce colors, simply by looking
at the flowers and their leaves. Caucasian skin tones are also difficult
to reproduce: The subtle pastel skin coloration is very revealing of any
over-saturation. Because this skin tone is such a strong memory color (at
least for Caucasians ;-) any hue shift is likewise apparent.
This shot is also good for developing a quick sense of camera resolution,
by looking at edges and detail in the flowers and leaves, as well as in
the model's hair. The hair is a good test of low-contrast resolution, as
JPEG compression often seems to flatten the image detail in this area.
The biggest problem with this image is that it is rather poorly controlled.
We have generally done a poor job of controlling framing on the shot, with
the result that the subject occupies a larger or smaller portion of the
image with different cameras. (In our defense, viewfinder vagaries contribute
greatly to this problem, and we have been more consistent in our later work.)
A bigger issue is that the lighting can vary substantially from shot to
shot. Although we try to take the shot at roughly the same time of day,
north/south sun angle varies considerably over the seasons. Likewise, even
"open sun" can be quite different, if the day is humid and hazy
or crisp and clear. Bottom line: This picture is good for a "quick
look," but don't use it as the only basis for your judgment.Outdoor Portrait, Close-Up
This shot only exists for more recently-tested cameras, as it came about
in response to a reader request for a test that would better show the cameras'
suitability for portrait work. The main thing to look for here is whether
the camera's lens does or does not distort the model's features. The shorter
subject distance also means the detail in the model's hair is more accessible
to most cameras, making this shot a good one to use for evaluating low-contrast
resolution and compression artifacts.Indoor Portrait, No Flash
This shot is intended to represent indoor shots taken with available light.
It's a good test of white balance and camera behavior under more subdued
lighting. The lighting for this shot corresponds to a fairly brightly lit
residential interior. (The light level is EV 12, produced by a total of
about 500 watts of residential tungsten lighting in a 200 square foot room
with off-white walls.)
In this shot, look to see how good a job the camera does of correcting for
the rather severe yellow cast of the lighting, and how true the flower colors
are. Is there excessive noise in any part of the image? Is the image reasonably
bright? If the image is overly dark, the camera probably isn't going to
be able to take adequate pictures in more typical, less brightly-lit indoor
scenes, without using the flash.Indoor Portrait, Flash
While we suspect this represents a pretty typical use of a digital camera
(indoors, using flash to supplement the ambient illumination), it has turned
out to be pretty challenging for many cameras. The problem arises because
the color balance of most camera flashes is very different from that of
the tungsten room lighting. Some cameras seem to handle it fairly well,
others will show strong bluish highlights (from the flash), while still
others turn the scene very yellow; apparently color-correcting for the anticipated
flash lighting, even when the flash is contributing only a minor "fill"
illumination. (As a side note, if you find yourself getting strong blue
highlights in your flash shots in situations like this, just tape a piece
of light-yellow mylar or photographic "gel" material over the
flash unit: You'll be surprised at the difference it can make!)
Beyond color balance issues, how well does the camera being tested do at
blending the light from the strobe with the room light? Some flashes completely
wash out the room lighting, while others produce a pleasing blend of the
two light sources. Some cameras allow you to adjust the default white balance,
which can help compensate for the difference between the flash and room
lighting. In viewing these images, look for odd color casts in the highlights
vs. the shadows, and how natural the overall image looks.The "Musicians" Poster
This image is one of the studio shots we mentioned earlier that is actually
a picture of a poster, rather than a "natural" subject. The advantage
is that we can completely control the lighting and the subject matter from
one camera to the next. The disadvantage of course, is that the image doesn't
exactly represent real-life shooting conditions: While we color-balanced
the poster pretty carefully to produce natural flesh tones, the dyes in
the poster won't reflect light in exactly the same way that human skin does.
Also, the resolution of the original file used to produce this image was
somewhat limited, meaning that the finest detail is rather coarse, when
compared to the original subjects: The models' hair is quite heavy, and
fine texture in their clothing is lost. Nonetheless, this picture is quite
useful for several relative observations about the cameras.
First, check out the overall color cast: The image was shot under "daylight
simulator" lighting that very carefully reproduces the color spectrum
of daylight lighting, as specified by the ISO 7589 photographic standard.
Thus, any color cast will be the fault of the individual camera, and its
white balance circuitry. (For some reason, this image tends to produce widely
varying results with different camera's automatic white balance setting.)
Also look at how well each camera handles both the delicate pastel shades
of the model's skin tones, as well as the bright colors of their clothing
and jewelry.
Even though the detail in this image is somewhat coarser than that achievable
with the best 35mm film, it's still a bit beyond the limits of even the
best digital point & shoots, at least as of this writing in late 1998.
Resolution differences are generally apparent in the model's hair, and in
the delicate silvery patterns in the Oriental model's robe. While this target
will continue to be useful in the future for evaluating color values, we
expect its limited resolution will start to show as CCD sensors reach 2
- 2.5 megapixels.)The "House" Poster
This is another studio poster shot, created mainly to stress the detail-resolving
power of the cameras. The original was shot on 35mm Kodak Royal Gold 25
color-negative film, perhaps the sharpest and finest-grained color emulsion
on the market today. It was scanned to a 72 megabyte RGB file via PhotoCD
Pro, then cropped, converted to the CMYK color space, and printed on a large-format
poster machine. You'll find the best areas for evaluating detail are in
the center and top of the image: The bricks, details in the windows, and
the fine patterns of leaves, sticks, and pine needles against the sky are
all good subjects for seeing detail in the cameras. The subtle gradations
of gray in the shingles on the house's roof also turn out to be a excellent
indicator of how well cameras do in preserving subtle tonal variations in
the face of the JPEG image-compression most cameras use. Overall, the detail
in this poster is quite fine, and should work well for evaluating cameras
up to about 2.5-3 megapixels. (At that point, we'll need to make a new poster,
perhaps starting with medium-format film, and output on one of the new 1440-dpi
high-resolution large-format inkjet printers.)
A couple of deficiencies in this poster are important to note though, one
having to do with the lens used to capture the original shot, and one with
the reproduction process itself. Sharp eyes looking at pictures taken with
higher-resolution cameras will note a "softness" in the corners
of the picture, most evident as a lack of texture in the grass at lower
left. This is an artifact of the camera lens used, a Nikkor 35-85 mm f4.5-5.6
zoom set at about a 40-45mm focal length, mounted on a Nikon 6006 camera.
We didn't realize until after the poster was made that this lens loses some
resolution in the corners, resulting in the lack of fine texture in the
grass at lower left. The other issue with this image is that the amount
of "unsharp masking" applied to the image was slightly high for
the printing process, with the result that there are very thin, but noticeable
"halos" around the fine branches silhouetted against the sky.
While these halos themselves are a feature that can test camera resolution,
they can also aggravate the effects of in-camera image sharpening. Neither
of these issues is a "killer" in our view, but they do somewhat
restrict the usefulness of images from this target. (For instance, we can't
very well use it to evaluate corner sharpness of the cameras we test!)Outdoor "Far-Field" Test Shot
A reader pointed out that (at one point), all of our shots involved subjects
fairly close to the camera: Lens performance at "infinity" can
be very different than at 10 feet, and we didn't have any shots that tested
for this. Since our studio is only about 20-25 feet long, testing lenses
at infinity unfortunately meant moving outside, with all the attendant variations
in lighting, not to mention obvious seasonal changes in vegetation. Nonetheless,
this is in fact a pretty important image, since many shots you'll take with
your digital camera will be of objects at some distance from the camera.
To evaluate differences between cameras, you'll need to pay closest attention
to details in the house itself, which won't vary from day to day or season
to season. Look for how the cameras handle the fine detail around the windows,
and in the bricks themselves. Try as much as possible to ignore the vegetation,
and pay no attention at all to variations in color balance and lighting!
One drawback with this "target" is that framing is rather difficult
to control. We wanted to use the same house as in our "house"
poster, so there'd be some common ground between the images, but the topography
of the landscape combines with variations in lens focal lengths to make
framing a bit problematic: The ground slopes fairly sharply down and away
from the house, then back up again in the yard of the house across the street.
We try to frame the scene so the house will end up filling about the same
amount of the image, but this means that some shots will be taken looking
up at the house from the street, while others will be captured from closer
to the same level as the house, with the camera oriented more horizontally.
We make this observation here to point out that this image should NOT be
used to evaluate geometric distortion in the camera lenses, since the camera
angle can change so radically.The Macro Shot
Most digital point & shoots have some form of "macro" capability
to facilitate close-up shooting, but there's a tremendous variation in how
close different cameras can shoot, and how sharp the optics are at that
distance. Macro focusing specifications are almost always stated in camera
data sheets in terms of how close the lens can focus, but that's a virtually
meaningless parameter, since lens focal length dramatically affects the
actual size of the captured image. Here, we wanted an image with a good
range of tonal values, and a range of extreme detail. Initially, the target
consisted of only the brooch and the two coins, but (again in response to
a reader suggestion) we later included the dollar bill to add more extreme
detail to the subject. We always shoot this target at the closest focusing
distance of the camera, so the images will show how small an area the camera
can reliably photograph.
Aside from the obvious issues of detail and sharpness, this subject also
stresses the tonal capability of the cameras somewhat: Most cameras have
difficulty maintaining detail in both the bright highlights of the white
birds in the brooch, and in the dark shadow values of the blue-black background.The "DaveBox" Multi-Purpose Target
Actually, other than far-field focus capability, this single test target
can tell you virtually everything you need to know to evaluate a digital
camera! (It just doesn't represent "typical" subjects very well,
so it's difficult for most people to translate what they're seeing back
to performance in real-world situations. I (Dave Etchells) originally developed
this target for testing high-end studio cameras, to quickly assess color
accuracy, tonal reproduction, highlight and shadow detail, sensor performance
to extreme light overloads, and resolution. Originally, it was a white box
with the various elements fastened to it. When I began using it to test
low-end digital cameras though, the bright white background of the box tended
to fool the cameras autoexposure systems, resulting in dark, underexposed
pictures. To correct for this, I painted most exposed parts of the box black,
which brought the overall reflectance down to a pretty close approximation
of the 18% gray standard used to calibrate most exposure systems.
Taking the DaveBox parts in turn, we'll start with the color tests: The
box includes both a MacBeth Color Checker (tm) and Kodak Q60 Ektacolor test
targets. (The MacBeth target is the large one on top, with large blocks
of color. The Q60 target is the smaller one on the bottom, with many small
color chips and gradations.) Both of these are commercially available, and
produced under rigid quality control: You can count on the colors in the
DaveBox being the same as those you'd get if you bought a Color Checker
from a camera store, or ordered a Q60 target from Kodak. The things to look
for here are the accuracy and purity of the colors in both targets, and
the level of color saturation. (See our earlier comments, or the article
on evaluating digital cameras in our hints & tips section for a description
of these terms.) Also look at how well the camera holds the subtle pastels
in the brighter range of the Q60 target - many cameras tend to blow-out
the fainter pastels all the way to white.
Turning next to tonal range, look carefully at the long Kodak grayscale
placed vertically near the center of the box: How wide a range of grayscale
"steps" can the camera resolve? (Most cameras do OK at the white
end, having more problems separating the very darkest steps.) Again, this
is a standard commercial target that you can purchase at most camera stores
catering to professional photographers.
Since we just mentioned problems with the shadow end of the tonal scale,
lets talk next about the shadow detail test object, the "black cat
in a coal bin" charcoal bricks at bottom center. Actually, it turns
out that just the charcoal bricks by themselves aren't too challenging,
particularly in the very "flat" lighting setup we use for the
studio shots. To make the test a bit more difficult, we cast a shadow into
the box by placing a piece of black-anodized aluminum "Cinefoil"(tm)
into the bottom of the little black box, projecting out the front toward
the light source. The result is that the bottom bricks have a very deep
shadow cast across them, posing a severe challenge for cameras' shadow-detail
capabilities. Again, for details on how to take advantage of this test element,
we refer you to the article on evaluating
cameras elsewhere on this site.
At the opposite end of the tonal scale, the white gauze next to the charcoal
bricks (at lower right) is a good test of a camera's ability to hold detail
in fairly strong highlights. Can you see any of the texture of the gauze,
or is everything blown out to pure white? Again (not to sound like a broken
record), check out the article on camera
evaluation, to see how to play with this image in Photoshop to extract
hidden detail.
The DaveBox target also includes several resolution elements, but the scaled-down
ISO resolution target is too fine for most point & shoots to be able
to do much with it. More useful for a quick look at resolution is the starburst
pattern at upper right: How far can the camera resolve the individual spokes
into the center? (For a much more powerful resolution test, see our description
of the ISO test below.)
Finally, there's the pot lid, which tests how well the sensor responds to
light overloads. As it turns out, this doesn't appear to be too much of
an issue these days, with modern sensor technology. I the early days of
digital studio cameras though, the strong reflections of the studio lights
directly back into the camera lens could produce all sorts of problems at
the sensor level. The thing to look for here (but you probably won't find
it) is brightly colored fringes or streaks around the images of the individual
lights."WG-18" Resolution Target
Actually, these days, this target is properly referred to as "ISO 12233
Photography, Electronic still picture cameras - Resolution measurements";
that being the international standard that specifies it, just now entering
international draft status. We've been working with it since it was still
in the "working group" committee though, and so have referred
to it in the past as the WG-18 target, after "working group 18"
that defined it. As you'd expect from something designed by an international
committee of imaging experts, there's a tremendous amount of information
you can extract from this target about how well a digital camera system
resolves fine detail. At its most extreme, you can use the ISO 12233 target
to extract a full "spatial frequency response" (SFR) curve for
a camera/sensor system. While this is absolutely the most accurate and comprehensive
test of a camera's resolution performance, it's also by FAR the most time-consuming:
Computing the SFR requires capturing 9 separate images of another, gray-scale
target, measuring and averaging the gray-scale tonal response of the camera,
then using that information to modify the tonal balance of the resolution
target itself. Finally, a Photoshop plug-in can be used to extract the SFR
data, which in turn can be loaded into a spreadsheet program for graphing
and display. We figure this would take us at least another 2-4 hours per
camera to execute, and just don't have the time to devote to it.
Fortunately, your own eyeballs can tell you quite a bit about how well a
camera does with this target, just by looking at the output. The number
values next to the resolution wedges refer to resolution in units of line
pairs per picture height. This is an excellent, consistent measure of camera
resolution, but the term undoubtedly requires a little explanation for the
uninitiated.
The dilemma for the standards-makers was to define a resolution measurement
that would apply equally well to cameras with a variety of sensor sizes
and different height-to-width ratios. The goal was to express resolution
relative to the total image area, not pixels, since the number of pixels
involved could change significantly from camera to camera. Rather than expressing
resolution as a number of pixels, the standards committee decided to measure
resolution in terms of the number of pairs of black/white lines across the
image area that the camera could distinguish. To avoid confusion with cameras
having different width/height ratios, the "fineness" of the line
pairs was expressed in terms of how many of them would fit across the picture
from top to bottom. Thus, the term "line pairs per picture height,"
or "lp/ph." (Note though, that the reference to picture height
only refers to the size of the lines, not to the direction the measurement
is being made in. That is, even when resolution is being measured along
the long axis of the camera's frame, the results are still expressed in
lp/ph.) When the target is properly framed in the image, the numbers adjacent
to the resolution elements indicate the pattern "pitch" in hundreds
of line pairs per picture height. (That is, "5" on the target
means 500 lp/ph.)Confused? Look at the adjacent (to the right and below) resolution target
clips: Both show the same resolution, the one on the right showing how well
the camera does in the horizontal direction, while the one below shows the
results in the vertical direction. (Note that we're concerned with how
many lines the camera can resolve across the pattern. As a result, the "tall"
pattern shows resolution in the horizontal direction, and the "wide"
one in the vertical direction.)Viewfinder Accuracy/Flash Uniformity (VFA)
This is probably the most boring of our test targets, but actually one of
the more important ones: Few people realize just how inaccurate viewfinders
are on even fairly costly point & shoot cameras, and digital point &
shoots are no exception! We use this target by exactly lining up the darker
rectangle with the boundaries of the viewfinder, and then checking to see
how much of the target area is actually captured. Most cameras deliberately
capture a bit more image than what's shown in the viewfinder, as protection
against inadvertently losing part of your subject off the edge of the picture.
This is generally a "good thing", but you don't want the camera
to be too generous in what it grabs, since this can make it hard to accurately
frame your pictures. It's also best if the region shown by the viewfinder
is well-centered in the final image area, again to make framing easier.
We were initially surprised to find that most cameras with LCD viewfinders
also crop the image area somewhat: We'd expected that LCD viewfinders would
show exactly what the sensor was seeing, but this is often not the case
- see the individual reviews (or scan the images in the Comparometer(tm)
to learn which cameras have the most accurate viewfinders and LCD panels.
The final parameter we test for is flash uniformity: For most uses, you
may not care too much about flash uniformity, as many flash shots are of
individuals, close to the center of the frame. If you do care about even
flash illumination though, this test will be important to you, as there's
quite a bit of variation between cameras. All of our VFA tests are shot
in a darkened studio, with the only illumination on the target coming from
the camera's on-board flash (provided of course that the camera has an on-board
flash). You'll see that some units provide a fairly even illumination, whereas
others have a "hot spot" in the center, or darker corners.Summary
If you made it all this way, congratulations! This page ended up being WAY
long, but we wanted to cover all the questions people have had about the
test targets and procedures once and for all. Hopefully we did so, but email us if there's anything
obvious we left out! Hopefully too, this lengthy discourse will help you
gain more benefit from our laboriously-assembled test shots. Good luck &
happy shopping!