.
the projection geometry by considering lenses with circular boundaries lying in a
plane that contains the optical axis. In particular, consider a lens with two spherical
surfaces of radius R and index of refraction n. We will assume that this lens is
surrounded by vacuum (or, to an excellent approximation, by air), with an index
of refraction equal to 1, and that it is thin, i.e., that a ray entering the lens and
refracted at its right boundary is immediately refracted again at the left boundary.
Consider a point P located at (negative) depth Z oﬀ the optical axis, and
denote by (PO) the ray passing through this point and the center O of the lens
(Figure 1.8). It easily follows from the paraxial form of Snell’s law that (PO) is
not refracted, and that all the other rays passing through P are focused by the thin
lens on the point p with depth z along (PO) such that
1
z
−
1
Z
=
1
f
, (1.5)
where f =
R
2(n−1)
is the focal length of the lens.
Note that the equations relating the positions of P and p are exactly the same
as under pinhole perspective projection if we take d = z since P and p lie on a ray
passing through the center of the lens, but that points located at a distance −Z
from O will be in sharp focus only when the image plane is located at a distance z
from O on the other side of the lens that satisﬁes Eq. (1.5), the thin lens equation.
Letting Z →−∞shows that f is the distance between the center of the lens and the
plane where objects such as stars (that are eﬀectively located at Z = −∞) focus.
The two points F and F

located at distance f from the lens center on the optical
axis are called the focal points of the lens. In practice, objects within some range
of distances (called depth of ﬁeld or depth of focus) will be in acceptable focus. As
shown in the problems at the end of this chapter, the depth of ﬁeld increases with
the f-number of the lens, i.e., the ratio between the focal length of the lens and its
diameter.
Note that the ﬁeld of view of a camera, i.e., the portion of scene space that
actually projects onto the retina of the camera, is not deﬁned by the focal length
alone but also depends on the eﬀective area of the retina (e.g., the area of ﬁlm that
can be exposed in a photographic camera, or the area of the sensor in a digital
camera; see Figure 1.9).
Section 1.1 Image Formation 10
film
a
φ
lens
f
FIGURE 1.9: The ﬁeld of view of a camera. It can be deﬁned as 2φ, where φ
def
= arctan
a
2f
,
a is the diameter of the sensor (ﬁlm, CCD, or CMOS chip), and f is the focal length of
the camera.
A more realistic model of simple optical systems is the thick lens. The equa-
tions describing its behavior are easily derived from the paraxial refraction equation,
and they are the same as the pinhole perspective and thin lens projection equations,
except for an oﬀset (Figure 1.10). If H and H

denote the principal points of the
lens, then Eq. (1.5) holds when −Z (resp. z) is the distance between P (resp. p)
and the plane perpendicular to the optical axis and passing through H (resp. H

).
In this case, the only undeﬂected ray is along the optical axis.
H’ H F’ F
P
z
f f
-Z
Y
-y
p
FIGURE 1.10: A simple thick lens with two spherical surfaces.
Simple lenses suﬀer from a number of aberrations. To understand why, let us
remember that the paraxial refraction model is only an approximation, valid when
the angle α between each ray along the optical path and the optical axis of the
length is small and sin α ≈ α. This corresponds to a ﬁrst-order Taylor expansion of
the sine function. For larger angles, additional terms yield a better approximation,
and it is easy to show that rays striking the interface farther from the optical axis
are focused closer to the interface. The same phenomenon occurs for a lens, and
it is the source of two types of spherical aberrations (Figure 1.11 [a]): Consider a
point P on the optical axis and its paraxial image p. The distance between p and
the intersection of the optical axis with a ray issued from P and refracted by the
lens is called the longitudinal spherical aberration of that ray. Note that if an image
plane Π were erected in P, the ray would intersect this plane at some distance from
Section 1.1 Image Formation 11
the axis, called the transverse spherical aberration of that ray. Together, all rays
passing through P and refracted by the lens form a circle of confusion centered
in P as they intersect Π. The size of that circle will change if we move Π along
the optical axis. The circle with minimum diameter is called the circle of least
confusion, and its center does not coincide (in general) with p.
(a)
e
P p
Π
(b)
(c)
FIGURE 1.11: Aberrations. (a) Spherical aberration: The gray region is the paraxial zone
where the rays issued from P intersect at its paraxial image p. If an image plane π were
erected in p, the image of p in that plane would form a circle of confusion of diameter e.
The focus plane yielding the circle of least confusion is indicated by a dashed line. (b)
Distortion: From left to right, the nominal image of a fronto-parallel square, pincushion
distortion, and barrel distortion. (c) Chromatic aberration: The index of refraction of a
transparent medium depends on the wavelength (or color) of the incident light rays. Here,
a prism decomposes white light into a palette of colors. Figure from US NAVY MANUAL
OF BASIC OPTICS AND OPTICAL INSTRUMENTS, prepared by the Bureau of Naval
Personnel, reprinted by Dover Publications, Inc. (1969).
Besides spherical aberration, there are four other types of primary aberrations
caused by the diﬀerences between ﬁrst- and third-order optics, namely coma, astig-
matism, ﬁeld curvature, and distortion. A precise deﬁnition of these aberrations is
beyond the scope of this book. Suﬃce to say that, like a spherical aberration, the
ﬁrst three degrade the image by blurring the picture of every object point. Distor-
tion, on the other hand, plays a diﬀerent role and changes the shape of the image
Section 1.1 Image Formation 12
FIGURE 1.12: Vignetting eﬀect in a two-lens system. The shaded part of the beam never
reaches the second lens. Additional apertures and stops in a lens further contribute to
vignetting.
as a whole (Figure 1.11 [b]). This eﬀect is due to the fact that diﬀerent areas of
a lens have slightly diﬀerent focal lengths. The aberrations mentioned so far are
monochromatic, i.e., they are independent of the response of the lens to various
wavelengths. However, the index of refraction of a transparent medium depends on
wavelength (Figure 1.11 [c]), and it follows from the thin lens equation (Eq. [1.5])
that the focal length depends on wavelength as well. This causes the phenomenon
of chromatic aberration: refracted rays corresponding to diﬀerent wavelengths will
intersect the optical axis at diﬀerent points (longitudinal chromatic aberration) and
form diﬀerent circles of confusion in the same image plane (transverse chromatic
aberration).
Aberrations can be minimized by aligning several simple lenses with well-
chosen shapes and refraction indices, separated by appropriate stops. These com-
pound lenses can still be modeled by the thick lens equations, but they suﬀer from
one more defect relevant to machine vision: light beams emanating from object
points located oﬀ-axis are partially blocked by the various apertures (including the
individual lens components themselves) positioned inside the lens to limit aberra-
tions (Figure 1.12). This phenomenon, called vignetting, causes the image bright-
ness to drop in the image periphery. Vignetting may pose problems to automated
image analysis programs, but it is not quite as important in photography, thanks to
the human eye’s remarkable insensitivity to smooth brightness gradients. Speaking
of which, it is time to have a look at this extraordinary organ.
1.1.4 The Human Eye
Here we give a (brief) overview of the anatomical structure of the eye. It is largely
based on the presentation in Wandell (1995), and the interested reader is invited
to read this excellent book for more details. Figure 1.13 (left) is a sketch of the
section of an eyeball through its vertical plane of symmetry, showing the main
elements of the eye: the iris and the pupil, which control the amount of light
penetrating the eyeball; the cornea and the crystalline lens, which together refract
the light to create the retinal image; and ﬁnally the retina, where the image is
Section 1.1 Image Formation 13
formed. Despite its globular shape, the human eyeball is functionally similar to
a camera with a ﬁeld of view covering a 160
◦
(width) 135
◦
(height) area. Like
any other optical system, it suﬀers from various types of geometric and chromatic
aberrations. Several models of the eye obeying the laws of ﬁrst-order geometric
optics have been proposed, and Figure 1.13 (right) shows one of them, Helmoltz’s
schematic eye. There are only three refractive surfaces, with an inﬁnitely thin
cornea and a homogeneous lens. The constants given in Figure 1.13 are for the
eye focusing at inﬁnity (unaccommodated eye). This model is of course only an
approximation of the real optical characteristics of the eye.
20mm 15mm
0.42mm
H
H’
F’ F
FIGURE 1.13: Left: the main components of the human eye. Reproduced with permission,
the American Society for Photogrammetry and Remote Sensing. A.L. Nowicki, “Stere-
oscopy.” MANUAL OF PHOTOGRAMMETRY, edited by M.M. Thompson, R.C. Eller,
W.A. Radlinski, and J.L. Speert, third edition, pp. 515–536. Bethesda: American Society
of Photogrammetry, (1966). Right: Helmoltz’s schematic eye as modiﬁed by Laurance
(after Driscoll and Vaughan, 1978). The distance between the pole of the cornea and the
anterior principal plane is 1.96 mm, and the radii of the cornea, anterior, and posterior
surfaces of the lens are respectively 8 mm, 10 mm, and 6 mm.
Let us have a second look at the components of the eye one layer at a time.
The cornea is a transparent, highly curved, refractive window through which light
enters the eye before being partially blocked by the colored and opaque surface of
the iris. The pupil is an opening at the center of the iris whose diameter varies from
about 1 to 8 mm in response to illumination changes, dilating in low light to increase
the amount of energy that reaches the retina and contracting in normal lighting
conditions to limit the amount of image blurring due to spherical aberration in the
eye. The refracting power (reciprocal of the focal length) of the eye is, in large
part, an eﬀect of refraction at the the air–cornea interface, and it is ﬁne-tuned
by deformations of the crystalline lens that accommodates to bring objects into
sharp focus. In healthy adults, it varies between 60 (unaccommodated case) and
68 diopters (1 diopter =1 m
−1
), corresponding to a range of focal lengths between
15 and 17 mm.
The retina itself is a thin, layered membrane populated by two types of
photoreceptors—rods and cones. There are about 100 million rods and 5 million
cones in a human eye. Their spatial distribution varies across the retina: The mac-
ula lutea is a region in the center of the retina where the concentration of cones is
particularly high and images are sharply focused whenever the eye ﬁxes its attention
on an object (Figure 1.13). The highest concentration of cones occurs in the fovea,
Section 1.2 Intrinsic and Extrinsic Parameters 14
a depression in the middle of the macula lutea where it peaks at 1.6 10
5
/mm
2
,
with the centers of two neighboring cones separated by only half a minute of visual
angle. Conversely, there are no rods in the center of the fovea, but the rod density
increases toward the periphery of the visual ﬁeld. There is also a blind spot on the
retina, where the ganglion cell axons exit the retina and form the optic nerve.
The rods are extremely sensitive photoreceptors, capable of responding to a
single photon, but they yield relatively poor spatial detail despite their high number
because many rods converge to the same neuron within the retina. In contrast,
cones become active at higher light levels, but the signal output by each cone in
the fovea is encoded by several neurons, yielding a high resolution in that area.
As discussed further in Chapter 3, there are three types of cones with diﬀerent
spectral sensitivities, and these play a key role in the perception of color. Much
more could (and should) be said about the human eye—for example how our two
eyes verge and ﬁxate on targets, and how they cooperate in stereo vision, an issue
brieﬂy discussed in Chapter 7.
1.2 INTRINSIC AND EXTRINSIC PARAMETERS
Digital images, like animal retinas, are spatially discrete, and divided into (usually)
rectangular picture elements, or pixels. This is an aspect of the image formation
process that we have neglected so far, assuming instead that the image domain is
spatially continuous. Likewise, the perspective equation derived in the previous
section is valid only when all distances are measured in the camera’s reference
frame, and when image coordinates have their origin at the image center where the
axis of symmetry of the camera pierces its retina. In practice, the world and camera
coordinate systems are related by a set of physical parameters, such as the focal
length of the lens, the size of the pixels, the position of the image center, and the
position and orientation of the camera. This section identiﬁes these parameters.
We will distinguish the intrinsic parameters, which relate the camera’s coordinate
system to the idealized coordinate system used in Section 1.1, from the extrinsic
parameters, which relate the camera’s coordinate system to a ﬁxed world coordinate
system and specify its position and orientation in space.
We ignore in the rest of this section the fact that, for cameras equipped with
a lens, a point will be in focus only when its depth and the distance between the
optical center of the camera and its image plane obey Eq. (1.5). In particular, we
assume that the camera is focused at inﬁnity, so d = f. Likewise, the nonlinear
aberrations associated with real lenses are not taken into account by Eq. (1.1). We
neglect these aberrations in this section, but revisit radial distortion in Section 1.3
when we address the problem of estimating the intrinsic and extrinsic parameters
of a camera (a process known as geometric camera calibration).
1.2.1 Rigid Transformations and Homogeneous Coordinates
This section features our ﬁrst use of homogeneous coordinates to represent the
position of points in two or three dimensions. Consider a point P whose position
in some coordinate frame (F) = (O, i, j, k) is given by
−−→
OP = Xi +Y j +Zk.
Section 1.2 Intrinsic and Extrinsic Parameters 15
We deﬁne the usual (nonhomogeneous) coordinate vector of P to be the vector
(X, Y, Z)
T
in R
3
and its homogeneous coordinate vector as the vector (X, Y, Z, 1)
T
in R
4
. We use bold letters to denote (homogeneous and nonhomogeneous) coordi-
nate vectors in this book, and always state which type of coordinates we use when
it is not obvious from the context. We also use a superscript on the left side of
coordinate vectors when necessary to indicate which coordinate frame a position is
expressed in. For example,
F
P stands for the coordinate vector of the point P in
the frame (F). Homogeneous coordinates are a convenient device for representing
various geometric transformations by matrix products. For example, the change of
coordinates between two Euclidean coordinate systems (A) and (B) may be rep-
resented by a 3 3 rotation matrix ! and a translation vector t in R
3
, and the
corresponding rigid transformation can be written in nonhomogeneous coordinates
as
A
P = !
B
P +t, (1.6)
where
A
P and
B
P are elements of R
3
. In homogeneous coordinates, we write
instead
A
P = T
B
P, where T =
_
! t
0
T
1
_
, (1.7)
and
A
P and
B
P are this time elements of R
4
.
Before going further, let us recall a few facts about rotations. Rotation matri-
ces form a mulitplicative group. From an analytical viewpoint, they are character-
ized by the facts that (1) the inverse of a rotation matrix is equal to its transpose,
and (2) its determinant is equal to one. It can also be shown that any rotation
matrix can be parameterized by three Euler angles, or written as the product of
three elementary rotations about the i, j, and k vectors of some coordinate sys-
tem. As shown in Chapters 7 and 14, other parameterizations—by exponentials of
antisymmetric matrices or quaternions for example—may prove useful as well. Ge-
ometrically, the matrix ! in Eq. (1.6) also represents the basis vectors (i
B
, j
B
, k
B
)
of (B) in the coordinate frame (A)—that is, the matrix ! in Eq. (1.6) is given by:
!
def
=
_
A
i
B
,
A
j
B
,
A
k
B
_
=
⎛
⎝
i
A
i
B
j
A
i
B
k
A
i
B
i
A
j
B
j
A
j
B
k
A
j
B
i
A
k
B
j
A
k
B
k
A
k
B
⎞
⎠
, (1.8)
and, as shown in the problems at the end of this chapter, Eq. (1.6) easily follows
from this deﬁnition. By deﬁnition, the columns of a rotation matrix form a right-
handed orthonormal coordinate system of R
3
. It follows from properties (1) and (2)
that their rows also form such a coordinate system. One may wonder what happens
when ! is replaced in Eq. (1.7) by some arbitrary nonsingular 33 matrix, or when
the matrix T itself is replaced by some arbitrary nonsingular 44 matrix. As further
discussed in Chapter 8, the coordinate frames (A) and (B) are no longer separated
by rigid transformations in this case, but by aﬃne and projective transformations
respectively.
As will be shown in the rest of this section, homogeneous coordinates also
provide an algrebraic representation of the perspective projection process in the
form of a 34 matrix /, so that the coordinate vector P = (X, Y, Z, 1)
T
of a point
P in some ﬁxed world coordinate system and the coordinate vector p = (x, y, 1)
T
of
Section 1.2 Intrinsic and Extrinsic Parameters 16
0
c
c
x
y
y
x
Normalized
image plane
Physical
retina
O
c
k
j
P
p
p
Pinhole
i
FIGURE 1.14: Physical and normalized image coordinate systems.
its image p in the camera’s reference frame are related by the perspective projection
equation
p =
1
Z
/P. (1.9)
1.2.2 Intrinsic Parameters
It is possible to associate with a camera a normalized image plane parallel to its
physical retina but located at a unit distance from the pinhole. We attach to this
plane its own coordinate system with an origin located at the point ˆ c where the
optical axis pierces it (Figure 1.14). Equation (1.1) can be written in this normalized
coordinate system as
⎧
⎪
⎪
⎨
⎪
⎪
⎩
ˆ x =
X
Z
ˆ y =
Y
Z
⇐⇒ ˆ p =
1
Z
_
Id 0
_
P, (1.10)
where ˆ p
def
= (ˆ x, ˆ y, 1)
T
is the vector of homogeneous coordinates of the projection ˆ p of
the point P into the normalized image plane, and P is as before the homogeneous
coordinate vector of P in the world coordinate frame.
The physical retina of the camera is in general diﬀerent (Figure 1.14): It is
located at a distance f , = 1 from the pinhole (remember that we assume that the
camera is focused at inﬁnity, so the distance between the pinhole and the image
plane is equal to the focal length), and the coordinates (x, y) of the image point p
are usually expressed in pixel units (instead of, say, meters). In addition, pixels may
be rectangular instead of square, so the camera has two additional scale parameters
Section 1.2 Intrinsic and Extrinsic Parameters 17
k and l, and
⎧
⎪
⎪
⎨
⎪
⎪
⎩
x = kf
X
Z
= kf ˆ x,
y = lf
Y
Z
= lf ˆ y.
(1.11)
Let us talk units for a second: f is a distance, expressed in meters, for example,
and a pixel will have dimensions
1
k

1
l
, where k and l are expressed in pixel m
−1
.
The parameters k, l, and f are not independent, and they can be replaced by the
magniﬁcations α = kf and β = lf expressed in pixel units.
Now, in general, the actual origin of the camera coordinate system is at a
corner c of the retina (in the case depicted in Figure 1.14, the lower-left corner,
or sometimes the upper-left corner, when the image coordinates are the row and
column indices of a pixel) and not at its center, and the center of the CCD matrix
usually does not coincide with the image center c
0
. This adds two parameters x
0
and y
0
that deﬁne the position (in pixel units) of c
0
in the retinal coordinate system.
Thus, Eq. (1.11) is replaced by
_
x = αˆ x +x
0
,
y = βˆ y +y
0
.
(1.12)
Finally, the camera coordinate system might also be skewed, due to some
manufacturing error, so the angle θ between the two image axes is not equal to
(but of course not very diﬀerent from) 90 degrees. In this case, it is easy to show
that Eq. (1.12) transforms into
⎧
⎨
⎩
x = αˆ x −αcot θˆ y +x
0
,
y =
β
sin θ
ˆ y +y
0
.
(1.13)
This can be written in matrix form as
p = /ˆ p, where p =
⎛
⎝
x
y
1
⎞
⎠
and /
def
=
⎛
⎜
⎜
⎝
α −αcot θ x
0
0
β
sin θ
y
0
0 0 1
⎞
⎟
⎟
⎠
. (1.14)
The 33 matrix / is called the (internal) calibration matrix of the camera. Putting
Eqs. (1.10) and (1.14) together, we obtain
p =
1
Z
/
_
Id 0
_
P =
1
Z
/P, where /
def
=
_
/ 0
_
, (1.15)
which is indeed an instance of Eq. (1.9). The ﬁve parameters α, β, θ, x
0
, and y
0
are called the intrinsic parameters of the camera.
Several of these parameters, such as the focal length, or the physical size
of the pixels, are often available in the EXIF tags attached to the JPEG images
recorded by digital cameras (this information might not be available, of course, as
in the case of stock ﬁlm footage). For zoom lenses, the focal length may vary with
Section 1.2 Intrinsic and Extrinsic Parameters 18
time, along with the image center when the optical axis of the lens is not exactly
perpendicular to the image plane. Simply changing the focus of the camera will
also aﬀect the magniﬁcation because it will change the lens-to-retina distance, but
we will continue to assume that the camera is focused at inﬁnity and ignore this
eﬀect in the rest of this chapter.
1.2.3 Extrinsic Parameters
Equation (1.15) is written in a coordinate frame (C) attached to the camera. Let
us now consider the case where this frame is distinct from the world coordinate
system (W). To emphasize this, we rewrite Eq. (1.15) as p =
1
Z
/
C
P, where
C
P
denotes the vector of homogeneous coordinates of the point P expressed in (C).
The change of coordinates between (C) and (W) is a rigid transformation, and it
can be written as
C
P =
⎛
⎝
! t
0
T
1
⎞
⎠
W
P,
where
W
P is the vector of homogeneous coordinates of the point P in the coordinate
frame (W). Taking P =
W
P and substituting in Eq. (1.15) ﬁnally yields
p =
1
Z
/P, where /= /
_
! t
_
. (1.16)
This is the most general form of the perspective projection equation, and indeed an
instance of Eq. (1.9). Knowing / determines the position of the camera’s optical
center in the coordinate frame (W)—that is, its homogeneous coordinate vector
O =
W
O. Indeed, as shown in the problems at the end of this chapter, /O = 0.
As mentioned earlier, a rotation matrix such as ! is deﬁned by three indepen-
dent parameters (for example, Euler angles). Adding to these the three coordinates
of the vector t, we obtain a set of six extrinsic parameters that deﬁne the position
and orientation of the camera relative to the world coordinate frame.
It is very important to understand that the depth Z in Eq. (1.16) is not
independent of / and P, because if m
T
1
, m
T
2
and m
T
3
denote the three rows of
/, it follows directly from Eq. (1.16) that Z = m
3
P. In fact, it is sometimes
convenient to rewrite Eq. (1.16) in the equivalent form:
⎧
⎪
⎪
⎨
⎪
⎪
⎩
x =
m
1
P
m
3
P
,
y =
m
2
P
m
3
P
.
(1.17)
A perspective projection matrix can be written explicitly as a function of its
ﬁve intrinsic parameters, the three rows r
T
1
, r
T
2
, and r
T
3
of the matrix !, and the
three coordinates t
1
, t
2
, and t
3
of the vector t, namely:
/=
⎛
⎜
⎜
⎝
αr
T
1
−αcot θr
T
2
+x
0
r
T
3
αt
1
−αcot θt
2
+x
0
t
3
β
sin θ
r
T
2
+y
0
r
T
3
β
sin θ
t
2
+y
0
t
3
r
T
3
t
3
⎞
⎟
⎟
⎠
. (1.18)
Section 1.2 Intrinsic and Extrinsic Parameters 19
When ! is written as the product of three elementary rotations, the vectors r
i
(i = 1, 2, 3) can of course be written in terms of the corresponding three angles,
and Eq. (1.18) gives an explicit parameterization of / in terms of all 11 camera
parameters.
1.2.4 Perspective Projection Matrices
This section examines the conditions under which a 34 matrix /can be written in
the form given by Eq. (1.18). Let us write without loss of generality /=
_
/ b
_
,
where / is a 3 3 matrix and b is an element of R
3
, and let us denote by a
T
3
the
third row of /. Clearly, if / is an instance of Eq. (1.18), then a
T
3
must be a unit
vector since it is equal to r
T
3
, the last row of a rotation matrix. Note, however,
that replacing / by λ/ in Eq. (1.17) for some arbitrary λ ,= 0 does not change
the corresponding image coordinates. This will lead us in the rest of this book
to consider projection matrices as homogeneous objects, only deﬁned up to scale,
whose canonical form, as expressed by Eq. (1.18), can be obtained by choosing a
scale factor such that [[a
3
[[ = 1. Note that the parameter Z in Eq. (1.16) can
only rightly be interpreted as the depth of the point P when / is written in this
canonical form. Note also that the number of intrinsic and extrinsic parameters of
a camera matches the 11 free parameters of the (homogeneous) matrix /.
We say that a 3 4 matrix that can be written (up to scale) as Eq. (1.18)
for some set of intrinsic and extrinsic parameters is a perspective projection matrix.
It is of practical interest to put some restrictions on the intrinsic parameters of a
camera because, as noted earlier, some of these parameters will be ﬁxed and might
be known. In particular, we will say that a 3 4 matrix is a zero-skew perspective
projection matrix when it can be rewritten (up to scale) as Eq. (1.18) with θ = π/2,
and that it is a perspective projection matrix with zero skew and unit aspect-ratio
when it can be rewritten (up to scale) as Eq. (1.18) with θ = π/2 and α = β. A
camera with known nonzero skew and nonunit aspect-ratio can be transformed into
a camera with zero skew and unit aspect-ratio by an appropriate change of image
coordinates. Are arbitrary 3 4 matrices perspective projection matrices? The
following theorem answers this question.
Theorem 1. Let / =
_
/ b
_
be a 3 4 matrix, and let a
T
i
(i = 1, 2, 3)
denote the rows of the matrix / formed by the three leftmost columns of /.
• A necessary and suﬃcient condition for / to be a perspective projection
matrix is that Det(/) ,= 0.
• A necessary and suﬃcient condition for / to be a zero-skew perspective
projection matrix is that Det(/) ,= 0 and
(a
1
a
3
) (a
2
a
3
) = 0.
• A necessary and suﬃcient condition for / to be a perspective projection
matrix with zero skew and unit aspect-ratio is that Det(/) ,= 0 and
_
(a
1
a
3
) (a
2
a
3
) = 0,
(a
1
a
3
) (a
1
a
3
) = (a
2
a
3
) (a
2
a
3
).
Section 1.2 Intrinsic and Extrinsic Parameters 20
p
q
r
r
q
p
π
π
r
r
Q
P P’
R
Q’
Q
R
P
P’
Q’
O
O
FIGURE 1.15: Aﬃne projection models: (top) weak-perspective and (bottom) paraper-
spective projections.
The conditions of the theorem are clearly necessary: By deﬁnition, given some
perspective projection matrix /, we can always write ρ/ = /! for some nonzero
scalar ρ, calibration matrix /, rotation matrix !, and vector t. In particular,
ρ
3
Det(/) = Det(/) ,= 0 since calibration matrices are nonsingular by construction,
so / is nonsingular. Further, a simple calculation shows that the rows of the matrix
1
ρ
/! satisfy the conditions of the theorem under the various assumptions imposed
by its statement. These conditions are proven to also be suﬃcient in Faugeras
(1993).
1.2.5 Weak-Perspective Projection Matrices
As noted in Section 1.1.2, when a scene’s relief is small compared to the overall
distance separating it from the camera observing it, a weak-perspective projection
model can be used to approximate the imaging process (Figure 1.15, top). Let O
denote the optical center of the camera, and let R denote a scene reference point.
The weak-perspective projection of a scene point P is constructed in two steps: the
point P is ﬁrst projected orthogonally onto a point P

of the plane Π
r
parallel to
the image plane Π and passing through R; perspective projection is then used to
map the point P

onto the image point p. Since π
r
is a fronto-parallel plane, the
net eﬀect of the second projection step is a scaling of the image coordinates.
As shown in this section, the weak-perspective projection process can be rep-
resented in terms of a 2 4 matrix /, so that the homogeneous coordinate vector
P = (X, Y, Z, 1)
T
of a point P in some ﬁxed world coordinate system and the non-
homogeneous coordinate vector p = (x, y)
T
of its image p in the camera’s reference
Section 1.2 Intrinsic and Extrinsic Parameters 21
frame are related by the aﬃne projection equation
p = /P. (1.19)
It turns out that this general model accomodates various other approximations
of the perspective projection process. These include the orthographic projection
model discussed earlier, as well as the parallel projection model, which subsumes the
orthographic one, and takes into account the fact that the objects of interest may
lie oﬀ the optical axis of the camera. In this model, the viewing rays are parallel to
each other but not necessarily perpendicular to the image plane. Paraperspective
is another aﬃne projection model that takes into account both the distortions
associated with a reference point that is oﬀ the optical axis of the camera and
possible variations in depth (Figure 1.15, bottom). Using the same notation as
before, and denoting by Δ the line joining the optical center O to the reference
point R, parallel projection in the direction of Δ is ﬁrst used to map P onto a
point P

of the plane Π
r
; perspective projection is then used to map the point P

onto the image point p.
We will focus on weak perspective in the rest of this section. Let us derive
the corresponding projection equation. If Z
r
denotes the depth of the reference
point R, the two elementary projection stages P → P

→ p can be written in the
normalized coordinate system attached to the camera as
⎛
⎝
X
Y
Z
⎞
⎠
−→
⎛
⎝
Z
Y
Z
r
⎞
⎠
−→
⎛
⎝
ˆ x
ˆ y
1
⎞
⎠
=
⎛
⎝
X/Z
r
Y/Z
r
1
⎞
⎠
,
or, in matrix form,
⎛
⎝
ˆ x
ˆ y
1
⎞
⎠
=
1
Z
r
⎛
⎝
1 0 0 0
0 1 0 0
0 0 0 Z
r
⎞
⎠
⎛
⎜
⎜
⎝
X
Y
Z
1
⎞
⎟
⎟
⎠
.
Introducing the calibration matrix / of the camera and its extrinsic parame-
ters ! and t gives the general form of the projection equation, i.e.,
p =
1
Z
r
/
⎛
⎝
1 0 0 0
0 1 0 0
0 0 0 Z
r
⎞
⎠
_
! t
0
T
1
_
P, (1.20)
where P and p denote as before the homogeneous coordinate vector of the point P in
the world reference frame, and the homogeneous coordinate vector of its projection
p in the camera’s coordinate system. Finally, noting that Z
r
is a constant and
writing
/ =
_
/
2
p
0
0
T
1
_
, where /
2
def
=
⎛
⎝
α −αcot θ
0
β
sin θ
⎞
⎠
and p
0
def
=
_
x
0
y
0
_
,
allows us to rewrite Eq. (1.20) as
p = /P, where /=
_
/ b
_
, (1.21)
Section 1.3 Geometric Camera Calibration 22
where p is, this time, the nonhomogeneous coordinate vector of the point p, and /
is a 2 4 projection matrix (compare to the general perspective case of Eq. [1.16]).
In this expression, the 2 3 matrix / and the 2-vector b are respectively deﬁned
by
/ =
1
Z
r
/
2
!
2
and b =
1
Z
r
/
2
t
2
+p
0
,
where !
2
denotes the 23 matrix formed by the ﬁrst two rows of !, and t
2
denotes
the 2-vector formed by the ﬁrst two coordinates of t.
Note that t
3
does not appear in the expression of /, and that t
2
and p
0
are coupled in this expression: the projection matrix does not change when t
2
is
replaced by t
2
+ a and p
0
is replaced by p
0
−
1
Z
r
/
2
a. This redundancy allows us
to arbitrarily choose x
0
= y
0
= 0. In other words, the position of the center of the
image is immaterial for weak-perspective projection. Note that the values of Z
r
, α,
and β are also coupled in the expression of /, and that the value of Z
r
is a priori
unknown in most applications. This allows us to write
/=
1
Z
r
_
k s
0 1
_
_
!
2
t
2
_
, (1.22)
where k and s denote the aspect ratio and the skew of the camera, respectively. In
particular, a weak-perspective projection matrix is deﬁned by two intrinsic param-
eters (k and s), ﬁve extrinsic parameters (the three angles deﬁning !
2
and the two
coordinates of t
2
), and one scene-dependent structure parameter Z
r
.
A 2 4 matrix / =
_
/ b
_
where / is an arbitrary rank-2 2 3 matrix
and b is an arbitrary vector in R
2
is called an aﬃne projection matrix. Both weak-
perspective and general aﬃne projection matrices are deﬁned by eight independent
parameters. Weak-perspective projection matrices are aﬃne ones of course. Con-
versely, a simple parameter-counting argument suggests that it should be possible
to write an arbitrary aﬃne projection matrix as a weak-perspective one. This is
conﬁrmed by the following theorem.
Theorem 2. An aﬃne projection matrix can be written uniquely (up to a
sign ambiguity) as a general weak-perspective projection matrix as deﬁned by
Eq. (1.22).
This theorem is proven in Faugeras et al. (2001, Propositions 4.26 and 4.27)
and the problems.
1.3 GEOMETRIC CAMERA CALIBRATION
This section addresses the problem of estimating the intrinsic and extrinsic param-
eters of a camera from the image positions of scene features such as points of lines,
whose positions are known in some ﬁxed world coordinate system (Figure 1.16). In
this context, camera calibration can be modeled as an optimization process, where
the discrepancy between the observed image features and their theoretical positions
is minimized with respect to the camera’s intrinsic and extrinsic parameters.
Speciﬁcally, we assume that the image positions (x
i
, y
i
) of n ﬁducial points
P
i
(i = 1, . . . , n) with known homogeneous coordinate vectors P
i
have been found
Section 1.3 Geometric Camera Calibration 23
j
C
(C)
(W)
j
k
i
i
O
W
W
W
W
C
C
k
p
P
i
i
FIGURE 1.16: Camera calibration setup: In this example, the calibration rig is formed by
three grids drawn in orthogonal planes. Other patterns could be used as well, and they
may involve lines or other geometric ﬁgures.
in a picture of a calibration rig, either automatically or by hand. In the absence of
modeling and measurement errors, geometric camera calibration amounts to ﬁnding
the intrinsic and extrinsic parameters ξ such that
⎧
⎪
⎪
⎨
⎪
⎪
⎩
x
i
=
m
1
(ξ) P
i
m
3
(ξ) P
i
,
y
i
=
m
2
(ξ) P
i
m
3
(ξ) P
i
,
(1.23)
where m
T
i
(ξ) denotes the i
th
row of the projection matrix /, explicitly parameter-
ized in this equation by the camera parameters. In the typical case where there are
more measurements than unknowns (at least six points for 11 intrinsic and extrin-
sic parameters), Eq. (1.23) does not admit an exact solution, and an approximate
one has to be found as the solution of a least-squares minimization problem (see
Chapter 22). We present two least-squares formulations of the calibration problem
in the rest of this section. The corresponding algorithms are illustrated with the
calibration data shown in Figure 1.17.
1.3.1 A Linear Approach to Camera Calibration
We decompose the calibration process into (1) the computation of the perspective
projection matrix / associated with the camera, followed by (2) the estimation of
the intrinsic and extrinsic parameters of the camera from this matrix.
Estimation of the Projection Matrix. Let us assume that our camera has
nonzero skew. According to Theorem 1, the matrix / is not singular, but other-
wise arbitrary. Clearing the denominators in Eq. (1.23) yields two linear equations
in m
1
, m
2
, and m
3
(we omit the parameters ξ from now on for the sake of con-
Section 1.3 Geometric Camera Calibration 24
0
50
100
150
200
250
0
50
100
150
200
250
0
50
100
150
200
250
0 100 200 300 400 500 600 700 800
0
100
200
300
400
500
600
FIGURE 1.17: Camera calibration data. Left: A rendering of 491 3D ﬁducial points
measured on a calibration rig. Right: The corresponding image points. Data courtesy of
Janne Heikkil¨ a; data copyright c _2000 University of Oulu.
ciseness), namely
_
(m
1
−x
i
m
3
) P
i
= P
T
i
m
1
+0
T
m
2
−x
i
P
T
i
m
3
= 0,
(m
2
−y
i
m
3
) P
i
= 0
T
m
1
+P
T
i
m
2
−y
i
P
T
i
m
3
= 0.
Collecting the constraints associated with all points yields a system of 2n
homogeneous linear equations in the 12 coeﬃcients of the matrix /, namely,
Tm = 0, (1.24)
where
T
def
=
⎛
⎜
⎜
⎜
⎜
⎝
P
T
1
0
T
−x
1
P
T
1
0
T
P
T
1
−y
1
P
T
1
. . . . . . . . .
P
T
n
0
T
−x
n
P
T
n
0
T
P
T
n
−y
n
P
T
n
⎞
⎟
⎟
⎟
⎟
⎠
and m
def
=
⎛
⎝
m
1
m
2
m
3
⎞
⎠
= 0.
When n ≥ 6, homogeneous linear least-squares can be used to compute the
value of the unit vector m (hence the matrix /) that minimizes [[Tm[[
2
as the
eigenvector of the 12 12 matrix T
T
T associated with its smallest eigenvalue (see
Chapter 22). Note that any nonzero multiple of the vector m would have done just
as well, reﬂecting the fact that / is deﬁned by only 11 independent parameters.
Degenerate Point Conﬁgurations. Before showing how to recover the intrinsic
and extrinsic parameters of the camera, let us pause to examine the degenerate
conﬁgurations of the points P
i
(i = 1, . . . , n) that may cause the failure of the
camera calibration process. We focus on the (ideal) case where the positions p
i
(i = 1, . . . , n) of the image points can be measured with zero error, and identify the
nullspace of the matrix T (i.e., the subspace of R
12
formed by the vectors l such
that Tl = 0).
Let l be such a vector. Introducing the vectors formed by successive quadru-
ples of its coordinates—that is, λ = (l
1
, l
2
, l
3
, l
4
)
T
, μ = (l
5
, l
6
, l
7
, l
8
)
T
, and
Section 1.3 Geometric Camera Calibration 25
ν = (l
9
, l
10
, l
11
, l
12
)
T
—allows us to write
0 = Tl =
⎛
⎜
⎜
⎜
⎜
⎝
P
T
1
0
T
−x
1
P
T
1
0
T
P
T
1
−y
1
P
T
1
. . . . . . . . .
P
T
n
0
T
−x
n
P
T
n
0
T
P
T
n
−y
n
P
T
n
⎞
⎟
⎟
⎟
⎟
⎠
⎛
⎝
λ
μ
ν
⎞
⎠
=
⎛
⎜
⎜
⎜
⎜
⎝
P
T
1
λ −x
1
P
T
1
ν
P
T
1
μ −y
1
P
T
1
ν
. . .
P
T
n
λ −x
n
P
T
n
ν
P
T
n
μ −y
n
P
T
n
ν
⎞
⎟
⎟
⎟
⎟
⎠
. (1.25)
Combining Eq. (1.23) with Eq. (1.25) yields
⎧
⎪
⎪
⎨
⎪
⎪
⎩
P
T
i
λ −
m
T
1
P
i
m
T
3
P
i
P
T
i
ν = 0,
P
T
i
μ −
m
T
2
P
i
m
T
3
P
i
P
T
i
ν = 0,
for i = 1, . . . , n.
Thus, after clearing the denominators and rearranging the terms, we ﬁnally obtain:
_
P
T
i
(λm
T
3
−m
1
ν
T
)P
i
= 0,
P
T
i
(μm
T
3
−m
2
ν
T
)P
i
= 0,
for i = 1, . . . , n. (1.26)
As expected, the vector l associated with λ = m
1
, μ = m
2
, and ν = m
3
is a
solution of these equations. Are there other solutions?
Let us ﬁrst consider the case where the points P
i
(i = 1, . . . , n) all lie in some
plane Π, so P
i
Π = 0 for some 4-vector Π. Clearly, choosing (λ, μ, ν) equal to
(Π, 0, 0), (0, Π, 0), (0, 0, Π), or any linear combination of these vectors will yield
a solution of Eq. (1.26). In other words, the nullspace of T contains the four-
dimensional vector space spanned by these vectors and m. In practice, this means
that the ﬁducial points P
i
should not all lie in the same plane.
In general, for a given nonzero value of the vector l, the points P
i
that satisfy
Eq. (1.26) must lie on the curve where the two quadric surfaces deﬁned by the
corresponding equations intersect. A closer look at Eq. (1.26) reveals that the
straight line where the planes deﬁned by m
3
P = 0 and ν P = 0 intersect lies
on both quadrics. It can be shown that the intersection curve of these two surfaces
consists of this line and of a twisted cubic curve Γ passing through the origin. A
twisted cubic is determined entirely by six points lying on it, and it follows that
seven points chosen at random will not fall on Γ. In addition, since this curve
passes through the origin, choosing n ≥ 6 random points will in general guarantee
that the matrix T has rank 11 and that the projection matrix can be recovered in
a unique fashion.
Estimation of the Intrinsic and Extrinsic Parameters. Once the projection
matrix / has been estimated, its expression in terms of the camera’s intrinsic and
extrinsic parameters (Eq. [1.18]) can be used to recover these parameters as follows:
We write as before / =
_
/ b
_
, with a
T
1
, a
T
2
, and a
T
3
denoting the rows of /,
and obtain
ρ
_
/ b
_
= /
_
! t
_
⇐⇒ρ
⎛
⎝
a
T
1
a
T
2
a
T
3
⎞
⎠
=
⎛
⎜
⎜
⎝
αr
T
1
−αcot θr
T
2
+x
0
r
T
3
β
sin θ
r
T
2
+y
0
r
T
3
r
T
3
⎞
⎟
⎟
⎠
,
Section 1.3 Geometric Camera Calibration 26
where ρ is an unknown scale factor, introduced here to account for the fact that
the recovered matrix / has unit Frobenius form since [[/[[
F
= [[m[[ = 1.
In particular, using the fact that the rows of a rotation matrix have unit
length and are perpendicular to each other yields immediately
⎧
⎪
⎪
⎨
⎪
⎪
⎩
ρ = ε/[[a
3
[[,
r
3
= ρa
3
,
x
0
= ρ
2
(a
1
a
3
),
y
0
= ρ
2
(a
2
a
3
),
(1.27)
where ε = ∓1.
Since θ is always in the neighborhood of π/2 with a positive sine, we have
⎧
⎨
⎩
ρ
2
(a
1
a
3
) = −αr
2
−αcot θr
1
,
ρ
2
(a
2
a
3
) =
β
sin θ
r
1
,
and
⎧
⎪
⎪
⎨
⎪
⎪
⎩
ρ
2
[[a
1
a
3
[[ =
[α[
sin θ
,
ρ
2
[[a
2
a
3
[[ =
[β[
sin θ
,
(1.28)
thus:
⎧
⎪
⎪
⎨
⎪
⎪
⎩
cos θ = −
(a
1
a
3
) (a
2
a
3
)
[[a
1
a
3
[[ [[a
2
a
3
[[
,
α = ρ
2
[[a
1
a
3
[[ sin θ,
β = ρ
2
[[a
2
a
3
[[ sin θ,
(1.29)
since the sign of the magniﬁcation parameters α and β is normally known in advance
and can be taken to be positive.
We can now compute r
1
and r
2
from the second equation in Eq. (1.28) as
⎧
⎨
⎩
r
1
=
ρ
2
sinθ
β
(a
2
a
3
) =
1
[[a
2
a
3
[[
(a
2
a
3
),
r
2
= r
3
r
1
.
(1.30)
Note that there are two possible choices for the matrix !, depending on the
value of ε. The translation parameters can now be recovered by writing /t = ρb,
and hence t = ρ/
−1
b. In practical situations, the sign of t
3
is often known in
advance (this corresponds to knowing whether the origin of the world coordinate
system is in front of or behind the camera), which allows the choice of a unique
solution for the calibration parameters.
Figure 1.18 shows the results of an experiment with the dataset from Fig-
ure 1.17. The recovered calibration matrix is
/ =
⎛
⎝
970.2841 0.0986 372.0050
0 963.3466 299.2921
0 0 1
⎞
⎠
for this 768 576 camera, with estimated values of 1.0072 for the aspect ratio, and
0.0058 degree for the skew angle [θ −π/2[.
1
The recovered image center is located
about 15 pixels away from the center of the image array.
1
In this book, an m×n matrix normally has m rows and n columns. Digital images and camera
retinas are the only exceptions, and we follow the tradition by assuming that an m× n picture
has m columns and n rows. For example, the camera used in this experiment has 768 columns
and 576 rows.
Section 1.3 Geometric Camera Calibration 27
−400 −300 −200 −100 0 100 200 300 400
−300
−200
−100
0
100
200
300
FIGURE 1.18: Results of camera calibration on the dataset shown in Figure 1.17. The
original data points (circles) are overlaid with the reprojected 3D points (dots). The
root-mean-squared error is 0.96 pixel for this 768 576 image.
1.3.2 A Nonlinear Approach to Camera Calibration
The method presented in the previous section ignores some of the constraints as-
sociated with the calibration process. For example, the camera skew was assumed
to be arbitrary instead of (very close to) zero in Section 1.3.1. We present in this
section a nonlinear approach to camera calibration that takes into account all the
relevant constraints.
This approach is borrowed from photogrammetry, an engineering ﬁeld whose
aim is to recover quantitative geometric information from one or several pictures,
with applications in cartography, military intelligence, city planning, etc. For many
years, photogrammetry relied on a combination of geometric, optical, and mechani-
cal methods to recover three-dimensional information from pictures, but the advent
of computers in the 1950s has made a purely computational approach to this prob-
lem feasible. This is the domain of analytical photogrammetry, where the intrinsic
parameters of a camera deﬁne its interior orientation, and the extrinsic parameters
deﬁne its exterior orientation.
In this setting, we assume once again that we observe n ﬁducial points P
i
(i = 1, . . . , n) whose positions in some world coordinate system are known, and
minimize the mean-squared distance between the measured positions of their im-
ages and those predicted by the perspective projection equation with respect to a
vector of camera parameters ξ in R
11+q
, where q ≥ 0, which might include various
distortion coeﬃcients in addition to the usual intrinsic and extrinsic parameters.
(This assumes that the aspect-ratio and skew are unknown. When they are known,
fewer parameters are necessary.) In particular, let us see how to account for radial
distortion, a type of aberration that depends on the distance separating the optical
axis from the point of interest. We model the projection process by
p =
1
Z
⎛
⎝
1/λ 0 0
0 1/λ 0
0 0 1
⎞
⎠
/P, (1.31)
Section 1.3 Geometric Camera Calibration 28
−400 −300 −200 −100 0 100 200 300 400
−300
−200
−100
0
100
200
300
FIGURE 1.19: Results of nonlinear camera calibration on the dataset shown in Figure 1.17.
The original data points (circles) are overlaid with the reprojected 3D points (dots). The
root-mean-squared error is 0.39 pixel for this 768 576 image. Three radial distortion
parameters were used in this case.
where λ is a polynomial function of the squared distance between the image center
and the image point p in normalized image coordinates, or:
d
2
= ˆ x
2
+ ˆ y
2
= [[/
−1
p[[
2
−1. (1.32)
In most applications, it is suﬃcient to use a low-degree polynomial (e.g., λ =
1 +

= −mY
m = −
d
Z
0
X, Y : world coordinates
x, y: image coordinates
d: pinhole-to-retina distance
Z
0
: reference-point depth (< 0)
m: magniﬁcation (> 0)
Orthographic
projection
_
x = X
y = Y
X, Y : world coordinates
x, y: image coordinates
Thin lens
equation
1
z
−
1
Z
=
1
f
Z: object-point depth (< 0)
z: image-point depth (> 0)
f: focal length
TABLE 1.1: Reference card: Projection models.
for this 768 576 camera, with estimated values of 1.0051 for the aspect ratio, and
less than 10
−
5 degree for the skew angle. The recovered image center is located
about 9 pixels away from the center of the image array. The three radial distortion
coeﬃcients are −0.1183, −0.3657, and 1.9112 in this case.
1.4 NOTES
The classical textbook by Hecht (1987) is an excellent introduction to geometric
optics, paraxial refraction, thin and thick lenses, and their aberrations, as brieﬂy
discussed in Section 1.1. Vignetting is discussed in Horn (1986). Wandell (1995)
gives an excellent treatment of image formation in the human visual system. Thor-
ough presentations of the geometric camera models discussed in Section 1.2 can
be found in Faugeras (1993), Hartley and Zisserman (2000b), and Faugeras et
al. (2001). The paraperspective projection model was introduced in computer vi-
sion by Ohta, Maenobu, and Sakai (1981), and its properties have been studied by
Aloimonos (1990).
The linear calibration technique described in Section 1.3.1 is detailed in Fau-
geras (1993). Its variant that takes radial distortion into account is adapted from
Tsai (1987). The book of Haralick and Shapiro (1992) presents a concise introduc-
tion to analytical photogrammetry. The Manual of Photogrammetry is of course the
gold standard, and newcomers to this ﬁeld (like the authors of this book) will prob-
ably ﬁnd the ingenious mechanisms and rigorous methods described in the various
editions of this book fascinating (Thompson et al. 1966; Slama et al. 1980). We will
come back to photogrammetry in the context of structure from motion in Chap-
ter 8. The linear and nonlinear least-squares techniques used in the approaches to
camera calibration discussed in the present chapter are presented in some detail in
Chapter 22. An excellent survey and discussion of these methods in the context of
analytical photogrammetry can be found in Triggs et al. (2000).
We have assumed in this chapter that a 3D calibration rig is available. This is
Section 1.4 Notes 30
Perspective projection
equation (homogeneous)
p =
1
Z
/P
Matrix of intrinsic
parameters
/ =
_
α −αcot θ x
0
0 β/sin θ y
0
0 0 1
_
Perspective projection
matrix
/= /
_
1 t
_
Aﬃne projection
equation
(nonhomogeneous)
p = /
_
P
1
_
= /P +b
Weak-perspective
projection matrix
/=
_
/ b
_
=
1
Z
r
_
k s
0 1
_
_
1
2
t
2
_
TABLE 1.2: Reference card: Geometric camera models.
the setting used in (Faig 1975; Tsai 1987; Faugeras 1993; Heikkil¨ a 2000) for exam-
ple. However, it is diﬃcult to build such a rig accurately—see Lavest, Viala, and
Dhome (1998) for a discussion of this problem and an ingenious solution—and many
authors prefer using multiple checkerboards or similar planar patterns (Devy, Gar-
ric & Orteu 1997; Zhang 2000). This includes the widely used C implementation of
J.-Y. Bouguet’s algorithm, distributed as part of OpenCV, an open-source library of
computer vision routines, available at http://opencv.willowgarage.com/wiki/.
A MATLAB version is also freely available at his web site; see: http://www.
vision.caltech.edu/bouguetj/calib_doc/.
Given the fundamental importance of the notions introduced in this chapter,
the main equations derived in its course have been collected in Tables 1.1 and 1.2
for reference.
PROBLEMS
1.1. Derive the perspective equation projections for a virtual image located at a
distance d in front of the pinhole.
1.2. Prove geometrically that the projections of two parallel lines lying in some
plane Φ appear to converge on a horizon line h formed by the intersection
of the image plane Π with the plane parallel to Φ and passing through the
pinhole.
1.3. Prove the same result algebraically using the perspective projection Eq. (1.1).
You can assume for simplicity that the plane Φ is orthogonal to the image
plane Π.
1.4. Consider a camera equipped with a thin lens, with its image plane at position
z and the plane of scene points in focus at position Z. Now suppose that the
image plane is moved to ˆ z. Show that the diameter of the corresponding blur
circle is
a
[z − ˆ z[
z
,
where a is the lens diameter. Use this result to show that the depth of ﬁeld
(i.e., the distance between the near and far planes that will keep the diameter
Section 1.4 Notes 31
of the blur circles below some threshold ε) is given by
D = 2εfZ(Z +f)
a
f
2
a
2
−ε
2
Z
2
,
and conclude that, for a ﬁxed focal length, the depth of ﬁeld increases as the
lens diameter decreases, and thus the f number f/a increases.
Hint: Solve for the depth
ˆ
Z of a point whose image is focused on the image
plane at position ˆ z, considering both the case where ˆ z is larger than z and the
case where it is smaller.
1.5. Give a geometric construction of the image p of a point P given the two focal
points F and F

of a thin lens.
1.6. Show that when the camera coordinate system is skewed and the angle θ be-
tween the two image axes is not equal to 90 degrees, then Eq. (1.12) transforms
into Eq. (1.13).
1.7. Consider two Euclidean coordinate systems (A) = (O
A
, i
A
, j
A
, k
A
) and (B) =
(O
B
, i
B
, j
B
, k
B
). Show that Eq. (1.8) follows from Eq. (1.6).
Hint: First consider the case where both coordinate systems share the same
basis vectors—that is, i
A
= i
B
, j
A
= j
B
, and k
A
= k
B
. Then consider
the case where they have the same origin—that is, O
A
= O
B
, before ﬁnally
treating the general case.
1.8. Write formulas for the matrices 1 in Eq. (1.8) when (B) is deduced from (A)
via a rotation of angle θ about the axes i
A
, j
A
, and k
A
respectively.
1.9. Let O denote the homogeneous coordinate vector of the optical center of a cam-
era in some reference frame, and let / denote the corresponding perspective
projection matrix. Show that /O = 0. Explain why this intuitively makes
sense.
1.10. Show that the conditions of Theorem 1 are necessary.
1.11. Show that any aﬃne projection matrix / =
_
/ b
_
can be written as a
general weak-perspective projection matrix as deﬁned by Eq. (1.22), i.e.,
/=
1
Z
r
_
k s
0 1
_
_
1
2
t
2
_
.
1.12. Give an analytical expression for the Jacobian of the vector function f(ξ) =
(f
1
[ξ], . . . , f
2n
[ξ])
T
featured in Eq. (1.33) with respect to the vector ξ of un-
known parameters.
PROGRAMMING EXERCISES
1.13. Implement the linear calibration algorithm presented in Section 1.3.1.
1.14. Implement the nonlinear calibration algorithm from Section 1.3.2.
C H A P T E R 2
Light and Shading
The brightness of a pixel in the image is a function of the brightness of the surface
patch in the scene that projects to the pixel. In turn, the brightness of the patch
depends on how much incident light arrives at the patch and on the fraction of the
incident light that gets reﬂected (Models in Section 2.1).
This means that the brightness of a pixel is profoundly ambiguous. Surpris-
ingly, people can disentangle these eﬀects quite accurately. Often, but not always,
people can tell whether objects are in bright light or in shadow, and do not perceive
objects in shadow as having dark surfaces. People can usually tell whether changes
of brightness are caused by changes in reﬂection or by shading (cinemas wouldn’t
work if we got it right all the time, however). Typically, people can tell that shading
comes from the geometry of the object, but sometimes get shading and markings
mixed up. For example, a streak of dark makeup under a cheekbone will often
look like a shading eﬀect, making the face look thinner. Quite simple models of
shading (Section 2.1) support a range of inference procedures (Section 2.2). More
complex models are needed to explain some important eﬀects (Section 2.1.4), but
make inference very diﬃcult indeed (Section 2.4).
2.1 MODELLING PIXEL BRIGHTNESS
Three major phenomena determine the brightness of a pixel: the response of the
camera to light, the fraction of light reﬂected from the surface to the camera, and
the amount of light falling on the surface. Each can be dealt with quite straight-
forwardly.
Camera response: Modern cameras respond linearly to middling intensities
of light, but have pronounced nonlinearities for darker and brighter illumination.
This allows the camera to reproduce the very wide dynamic range of natural light
without saturating. For most purposes, it is enough to assume that the camera
response is linearly related to the intensity of the surface patch. Write X for a
point in space that projects to x in the image, I
patch
(X) for the intensity of the
surface patch at X, and I
camera
(x) for the camera response at x. Then our model
is:
I
camera
(x) = kI
patch
(x),
where k is some constant to be determined by calibration. Generally, we assume
that this model applies and that k is known if needed. Under some circumstances,
a more complex model is appropriate; we discuss how to recover such models in
Section 2.2.1.
Surface reﬂection: Diﬀerent points on a surface may reﬂect more or less
of the light that is arriving. Darker surfaces reﬂect less light, and lighter surfaces
reﬂect more. There is a rich set of possible physical eﬀects, but most can be ignored.
Section 2.1.1 describes the relatively simple model that is suﬃcient for almost all
32
Section 2.1 Modelling Pixel Brightness 33
N
L
N
L
S
θ
θ
Specularities
N
L
S
θ
θ
Diffuse reflection
Specular reflection
Specularity
FIGURE 2.1: The two most important reﬂection modes for computer vision are diﬀuse re-
ﬂection (left), where incident light is spread evenly over the whole hemisphere of outgoing
directions, and specular reﬂection (right), where reﬂected light is concentrated in a single
direction. The specular direction S is coplanar with the normal and the source direction
(L), and has the same angle to the normal that the source direction does. Most sur-
faces display both diﬀuse and specular reﬂection components. In most cases, the specular
component is not precisely mirror like, but is concentrated around a range of directions
close to the specular direction (lower right). This causes specularities, where one sees a
mirror like reﬂection of the light source. Specularities, when they occur, tend to be small
and bright. In the photograph, they appear on the metal spoon and on the plate. Large
specularities can appear on ﬂat metal surfaces (arrows). Most curved surfaces (such as
the plate) show smaller specularities. Most of the reﬂection here is diﬀuse; some cases are
indicated by arrows. Martin Brigdale c _ Dorling Kindersley, used with permission.
purposes in computer vision.
Illumination: The amount of light a patch receives depends on the overall
intensity of the light, and on the geometry. The overall intensity could change
because some luminaires (the formal term for light sources) might be shadowed, or
might have strong directional components. Geometry aﬀects the amount of light
arriving at a patch because surface patches facing the light collect more radiation
and so are brighter than surface patches tilted away from the light, an eﬀect known
as shading. Section 2.1.2 describes the most important model used in computer
vision; Section 2.3 describes a much more complex model that is necessary to explain
some important practical diﬃculties in shading inference.
2.1.1 Reﬂection at Surfaces
Most surfaces reﬂect light by a process of diﬀuse reﬂection. Diﬀuse reﬂection scat-
ters light evenly across the directions leaving a surface, so the brightness of a diﬀuse
surface doesn’t depend on the viewing direction. Examples are easy to identify with
Section 2.1 Modelling Pixel Brightness 34
this test: most cloth has this property, as do most paints, rough wooden surfaces,
most vegetation, and rough stone or concrete. The only parameter required to
describe a surface of this type is its albedo, the fraction of the light arriving at the
surface that is reﬂected. This does not depend on the direction in which the light
arrives or the direction in which the light leaves. Surfaces with very high or very
low albedo are diﬃcult to make. For practical surfaces, albedo lies in the range
0.05 – 0.90 (see Brelstaﬀ and Blake (1988b), who argue the dynamic range is closer
to 10 than the 18 implied by these numbers). Mirrors are not diﬀuse, because what
you see depends on the direction in which you look at the mirror. The behav-
ior of a perfect mirror is known as specular reﬂection. For an ideal mirror, light
arriving along a particular direction can leave only along the specular direction,
obtained by reﬂecting the direction of incoming radiation about the surface normal
(Figure 2.1). Usually some fraction of incoming radiation is absorbed; on an ideal
specular surface, this fraction does not depend on the incident direction.
If a surface behaves like an ideal specular reﬂector, you could use it as a
mirror, and based on this test, relatively few surfaces actually behave like ideal
specular reﬂectors. Imagine a near perfect mirror made of polished metal; if this
surface suﬀers slight damage at a small scale, then around each point there will be
a set of small facets, pointing in a range of directions. In turn, this means that
light arriving in one direction will leave in several diﬀerent directions because it
strikes several facets, and so the specular reﬂections will be blurred. As the surface
becomes less ﬂat, these distortions will become more pronounced; eventually, the
only specular reﬂection that is bright enough to see will come from the light source.
This mechanism means that, in most shiny paint, plastic, wet, or brushed metal
surfaces, one sees a bright blob—often called a specularity—along the specular di-
rection from light sources, but few other specular eﬀects. Specularities are easy to
identify, because they are small and very bright (Figure 2.1; Brelstaﬀ and Blake
(1988b)). Most surfaces reﬂect only some of the incoming light in a specular compo-
nent, and we can represent the percentage of light that is specularly reﬂected with
a specular albedo. Although the diﬀuse albedo is an important material property
that we will try to estimate from images, the specular albedo is largely seen as a
nuisance and usually is not estimated.
2.1.2 Sources and Their Eﬀects
The main source of illumination outdoors is the sun, whose rays all travel parallel to
one another in a known direction because it is so far away. We model this behavior
with a distant point light source. This is the most important model of lighting
(because it is like the sun and because it is easy to use), and can be quite eﬀective
for indoor scenes as well as outdoor scenes. Because the rays are parallel to one
another, a surface that faces the source cuts more rays (and so collects more light)
than one oriented along the direction in which the rays travel. The amount of light
collected by a surface patch in this model is proportional to the cosine of the angle
θ between the illumination direction and the normal (Figure 2.2). The ﬁgure yields
Lambert’s cosine law, which states the brightness of a diﬀuse patch illuminated by
a distant point light source is given by
I = ρI
0
cos θ,
Section 2.1 Modelling Pixel Brightness 35
A
B
θ
θ
Cast shadow
Diffuse reflection,
bright
Diffuse reflection,
dark
Light
Blocker
Source
FIGURE 2.2: The orientation of a surface patch with respect to the light aﬀects how
much light the patch gathers. We model surface patches as illuminated by a distant point
source, whose rays are shown as light arrowheads. Patch A is tilted away from the source
(θ is close to 90
0
) and collects less energy, because it cuts fewer light rays per unit surface
area. Patch B, facing the source (θ is close to 0
0
), collects more energy, and so is brighter.
Shadows occur when a patch cannot see a source. The shadows are not dead black, because
the surface can see interreﬂected light from other surfaces. These eﬀects are shown in the
photograph. The darker surfaces are turned away from the illumination direction. Martin
Brigdale c _ Dorling Kindersley, used with permission.
where I
0
is the intensity of the light source, θ is the angle between the light source
direction and the surface normal, and ρ is the diﬀuse albedo. This law predicts that
bright image pixels come from surface patches that face the light directly and dark
pixels come from patches that see the light only tangentially, so that the shading
on a surface provides some shape information. We explore this cue in Section 2.4.
If the surface cannot see the source, then it is in shadow. Since we assume
that light arrives at our patch only from the distant point light source, our model
suggests that shadows are deep black; in practice, they very seldom are, because
the shadowed surface usually receives light from other sources. Outdoors, the most
important such source is the sky, which is quite bright. Indoors, light reﬂected
from other surfaces illuminates shadowed patches. This means that, for example,
we tend to see few shadows in rooms with white walls, because any shadowed
patch receives a lot of light from the walls. These interreﬂections also can have a
signiﬁcant eﬀect on the brightness surfaces that are not in shadow. Interreﬂection
eﬀects are sometimes modelled by adding a constant ambient illumination term
to the predicted intensity. The ambient term ensures that shadows are not too
dark, but this is not a particularly successful model of the spatial properties of
interreﬂections. More detailed models require some familiarity with radiometric
terminology, but they are important in some applications; we have conﬁned this
topic to Section 2.3.
Section 2.1 Modelling Pixel Brightness 36
2.1.3 The Lambertian+Specular Model
For almost all purposes, it is enough to model all surfaces as being diﬀuse with
specularities. This is the lambertian+specular model. Specularities are relatively
seldom used in inference (Section 2.2.2 sketches two methods), and so there is
no need for a formal model of their structure. Because specularities are small and
bright, they are relatively easy to identify and remove with straightforward methods
(ﬁnd small bright spots, and replace them by smoothing the local pixel values).
More sophisticated specularity ﬁnders use color information (Section 3.5.1). Thus,
to apply the lambertian+specular model, we ﬁnd and remove specularities, and
then use Lambert’s law (Section 2.1.2) to model image intensity.
We must choose which source eﬀects to model. In the simplest case, a local
shading model, we assume that shading is caused only by light that comes from the
luminaire (i.e., that there are no interreﬂections).
Now assume that the luminaire is an inﬁnitely distant source. For this case,
write N(x) for the unit surface normal at x, S for a vector pointing from x toward
the source with length I
o
(the source intensity), ρ(x) for the albedo at x, and
V is(S, x) for a function that is 1 when x can see the source and zero otherwise.
Then, the intensity at x is
I(x) = ρ(x) (N S) Vis(S, x) + ρ(x)A + M
Image = Diﬀuse + Ambient + Specular (mirror-like)
intensity term term term
This model can still be used for a more complex source (for example, an area
source), but in that case it is more diﬃcult to determine an appropriate S(x).
2.1.4 Area Sources
An area source is an area that radiates light. Area sources occur quite com-
monly in natural scenes—an overcast sky is a good example—and in synthetic
environments—for example, the ﬂuorescent light boxes found in many industrial
ceilings. Area sources are common in illumination engineering, because they tend
not to cast strong shadows and because the illumination due to the source does not
fall oﬀ signiﬁcantly as a function of the distance to the source. Detailed models of
area sources are complex (Section 2.3), but a simple model is useful to understand
shadows. Shadows from area sources are very diﬀerent from shadows cast by point
sources. One seldom sees dark shadows with crisp boundaries indoors. Instead,
one could see no visible shadows, or shadows that are rather fuzzy diﬀuse blobs, or
sometimes fuzzy blobs with a dark core (Figure 2.3). These eﬀects occur because
rooms tend to have light walls and diﬀuse ceiling ﬁxtures, which act as area sources.
As a result, the shadows one sees are area source shadows.
To compute the intensity at a surface patch illuminated by an area source, we
can break the source up into inﬁnitesimal source elements, then sum eﬀects from
each element. If there is an occluder, then some surface patches may see none of
the source elements. Such patches will be dark, and lie in the umbra (a Latin word
meaning “shadow”). Other surface patches may see some, but not all, of the source
elements. Such patches may be quite bright (if they see most of the elements), or
Section 2.2 Inference from Shading 37
A B
Gradient
Umbra
Penumbra
Area
Source
Occluder
1 2 3
1
2
3
FIGURE 2.3: Area sources generate complex shadows with smooth boundaries, because
from the point of view of a surface patch, the source disappears slowly behind the occluder.
Left: a photograph, showing characteristic area source shadow eﬀects. Notice that A is
much darker than B; there must be some shadowing eﬀect here, but there is no clear
shadow boundary. Instead, there is a fairly smooth gradient. The chair leg casts a
complex shadow, with two distinct regions. There is a core of darkness (the umbra—
where the source cannot be seen at all) surrounded by a partial shadow (penumbra—
where the source can be seen partially). A good model of the geometry, illustrated right,
is to imagine lying with your back to the surface looking at the world above. At point 1,
you can see all of the source; at point 2, you can see some of it; and at point 3, you can
see none of it. Peter Anderson c _ Dorling Kindersley, used with permission.
relatively dark (if they see few elements), and lie in the penumbra (a compound of
Latin words meaning “almost shadow”). One way to build intuition is to think of
a tiny observer looking up from the surface patch. At umbral points, this observer
will not see the area source at all whereas at penumbral points, the observer will see
some, but not all, of the area source. An observer moving from outside the shadow,
through the penumbra and into the umbra will see something that looks like an
eclipse of the moon (Figure 2.3). The penumbra can be large, and can change quite
slowly from light to dark. There might even be no umbral points at all, and, if the
occluder is suﬃciently far away from the surface, the penumbra could be very large
and almost indistinguishable in brightness from the unshadowed patches. This is
why many objects in rooms appear to cast no shadow at all (Figure 2.4).
2.2 INFERENCE FROM SHADING
Shading can be used to infer a variety of properties of the visual world. Successful
inference often requires that we calibrated the camera radiometrically, so that we
know how pixel values map to radiometric values (Section 2.2.1). As Figure 2.1
Section 2.2 Inference from Shading 38
1
1
FIGURE 2.4: The photograph on the left shows a room interior. Notice the lighting has
some directional component (the vertical face indicated by the arrow is dark, because
it does not face the main direction of lighting), but there are few visible shadows (for
example, the chairs do not cast a shadow on the ﬂoor). On the right, a drawing to show
why; here there is a small occluder and a large area source. The occluder is some way
away from the shaded surface. Generally, at points on the shaded surface the incoming
hemisphere looks like that at point 1. The occluder blocks out some small percentage of
the area source, but the amount of light lost is too small to notice (compare ﬁgure 2.3).
Jake Fitzjones c _ Dorling Kindersley, used with permission.
suggests, specularities are a source of information about the shape of a surface, and
Section 2.2.2 shows how this information can be interpreted. Section 2.2.3 shows
how to recover the albedoes of surfaces from images. Finally, Section 2.2.4 shows
how multiple shaded images can be used to recover surface shape.
2.2.1 Radiometric Calibration and High Dynamic Range Images
Real scenes often display a much larger range of intensities than cameras can cope
with. Film and charge-coupled devices respond to energy. A property called reci-
procity means that, if a scene patch casts intensity E onto the ﬁlm, and if the
shutter is open for time Δt, the response is a function of EΔt alone. In particular,
we will get the same outcome if we image one patch of intensity E for time Δt and
another patch of intensity E/k for time kΔt. The actual response that the ﬁlm
produces is a function of EΔt; this function might depend on the imaging system,
but is typically somewhat linear over some range, and sharply non-linear near the
top and bottom of this range, so that the image can capture very dark and very
light patches without saturation. It is usually monotonically increasing.
There are a variety of applications where it would be useful to know the actual
radiance (equivalently, the intensity) arriving at the imaging device. For example,
we might want to compare renderings of a scene with pictures of the scene, and to
do that we need to work in real radiometric units. We might want to use pictures
of a scene to estimate the lighting in that scene so we can postrender new objects
into the scene, which would need to be lit correctly. To infer radiance, we must
determine the ﬁlm response, a procedure known as radiometric calibration. As we
Section 2.2 Inference from Shading 39
shall see, doing this will require more than one image of a scene, each obtained at
diﬀerent exposure settings. Imagine we are looking at a scene of a stained glass
window lit from behind in a church. At one exposure setting, we would be able
to resolve detail in the dark corners, but not on the stained glass, which would be
saturated. At another setting, we would be able to resolve detail on the glass, but
the interior would be too dark. If we have both settings, we may as well try to
recover radiance with a very large dynamic range—producing a high dynamic range
image.
Now assume we have multiple registered images, each obtained using a dif-
ferent exposure time. At the i, j’th pixel, we know the image intensity value I
(k)
ij
for the k’th exposure time, we know the value of the k’th exposure time Δt
k
, and
we know that the intensity of the corresponding surface patch E
ij
is the same for
each exposure, but we do not know the value of E
ij
. Write the camera response
function f, so that
I
(k)
ij
= f(E
ij
Δt
k
).
There are now several possible approaches to solve for f. We could assume a
parametric form—say, polynomial—then solve using least squares. Notice that we
must solve not only for the parameters of f, but also for E
ij
. For a color camera,
we solve for calibration of each channel separately. Mitsunaga and Nayar (1999)
have studied the polynomial case in detail. Though the solution is not unique,
ambiguous solutions are strongly diﬀerent from one another, and most cases are
easily ruled out. Furthermore, one does not need to know exposure times with
exact accuracy to estimate a solution, as long as there are suﬃcient pixel values;
instead, one estimates f from a ﬁxed set of exposure times, then estimates the
exposure times from f, and then re-estimates. This procedure is stable.
Alternatively, because the camera response is monotonic, we can work with
its inverse g = f
−1
, take logs, and write
log g(I
(k)
ij
) = log E
ij
+ log Δt
k
.
We can now estimate the values that g takes at each point and the E
ij
by placing
a smoothness penalty on g. In particular, we minimize

i,j,k
(log g(I
(k)
ij
) −(log E
ij
+ log Δt
k
))
2
+ smoothness penalty on g
by choice of g. Debevec and Malik (1997) penalize the second derivative of g. Once
we have a radiometrically calibrated camera, estimating a high dynamic range image
is relatively straightforward. We have a set of registered images, and at each pixel
location, we seek the estimate of radiance that predicts the registered image values
best. In particular, we assume we know f. We seek an E
ij
such that

k
w(I
ij
)(I
(k)
ij
−f(E
ij
Δt
k
))
2
is minimized. Notice the weights because our estimate of f is more reliable when
I
ij
is in the middle of the available range of values than when it is at larger or
smaller values.
Section 2.2 Inference from Shading 40
Image measurement
S
c
e
n
e
R
a
d
i
a
n
c
e
FIGURE 2.5: It is possible to calibrate the radiometric response of a camera from multiple
images obtained at diﬀerent exposures. The top row shows four diﬀerent exposures of the
same scene, ranging from darker (shorter shutter time) to lighter (longer shutter time).
Note how, in the dark frames, the lighter part of the image shows detail, and in the light
frames, the darker part of the image shows detail; this is the result of non-linearities in
the camera response. On the bottom left, we show the inferred calibration curves for
each of the R, G, and B camera channels. On the bottom right, a composite image
illustrates the results. The dynamic range of this image is far too large to print; instead,
the main image is normalized to the print range. Overlaid on this image are boxes where
the radiances in the box have also been normalized to the print range; these show how
much information is packed into the high dynamic range image. This ﬁgure was originally
published as Figure 7 of “Radiometric Self Calibration,” by T. Mitsunaga and S. Nayar,
Proc. IEEE CVPR 1999, c _ IEEE, 1999.
2.2.2 The Shape of Specularities
Specularities are informative. They oﬀer hints about the color of illumination (see
Chapter 3) and oﬀer cues to the local geometry of a surface. Understanding these
cues is a useful exercise in diﬀerential geometry. We consider a smooth specular
surface and assume that the brightness reﬂected in the direction V is a function of
V P, where P is the specular direction. We expect the specularity to be small and
isolated, so we can assume that the source direction S and the viewing direction
V are constant over its extent. Let us further assume that the specularity can be
deﬁned by a threshold on the specular energy, i.e., V P ≥ 1 −ε for some constant
ε, denote by N the unit surface normal, and deﬁne the half-angle direction as
H = (S + V )/2 (Figure 2.6(left)). Using the fact that the vectors S, V and P
have unit length and a whit of plane geometry, it can easily be shown that the
boundary of the specularity is deﬁned by (see exercises)
1 −ε = V P = 2
(H N)
2
(H H)
−1. (2.1)
Because the specularity is small, the second-order structure of the surface
Section 2.2 Inference from Shading 41
u
u
1
2
y
x
H
N
V
P
S
H N
z
α/2
α
FIGURE 2.6: A specular surface viewed by a distant observer. We establish a coordinate
system at the brightest point of the specularity (where the half-angle direction is equal to
the normal) and orient the system using the normal and principal directions.
will allow us to characterize the shape of its boundary as follows: there is some
point on the surface inside the specularity (in fact, the brightest point) where H
is parallel to N. We set up a coordinate system at this point, oriented so that
the z-axis lies along N and the x- and y-axes lie along the principal directions u
1
and u
2
(Figure 2.6(right)). As noted earlier, the surface can be represented up to
second order as z = −1/2(κ
1
x
2
+ κ
2
y
2
) in this frame, where κ
1
and κ
2
are the
principal curvatures. Now, let us deﬁne a parametric surface as a diﬀerentiable
mapping x : U ⊂ R
2
→ R
3
associating with any couple (u, v) ∈ U the coordinate
vector (x, y, z)
T
of a point in some ﬁxed coordinate system. It is easily shown (see
exercises) that the normal to a parametric surface is along the vector
∂
∂u
x
∂
∂v
x.
Our second-order surface model is a parametric surface parameterized by x and y,
thus its unit surface normal is deﬁned in the corresponding frame by
N(x, y) =
1
_
1 +κ
2
1
x
2
+κ
2
2
y
2
⎛
⎝
κ
1
x
κ
2
y
1
⎞
⎠
,
and H = (0, 0, 1)
T
. Because H is a constant, we can rewrite Eq. (2.1) as κ
2
1
x
2
+
κ
2
2
y
2
= ζ, where ζ is a constant depending on ε. In particular, the shape of
the specularity on the surface contains information about the second fundamental
form. The specularity will be an ellipse, with major and minor axes oriented along
the principal directions, and an eccentricity equal to the ratio of the principal
curvatures. Unfortunately, the shape of the specularity on the surface is not, in
general, directly observable, so this property can be exploited only when a fair
amount about the viewing and illumination setup is known (Healey and Binford
1986).
Although we cannot get much out of the shape of the specularity in the
image, it is possible to tell a convex surface from a concave one by watching how a
Section 2.2 Inference from Shading 42
specularity moves as the view changes (you can convince yourself of this with the
aid of a spoon).
1
Let us consider a point source at inﬁnity and assume that the
specular lobe is very narrow, so the viewing direction and the specular direction
coincide. Initially, the specular direction is V , and the specularity is at the surface
point P; after a small eye motion, V changes to V

, while the specularity moves
to the close-by point P

(Figure 2.7).
V
N
V’
N’
observer
Motion of
observer
Motion of
V
Motion of
P
P’
N
V’
N’
direction
specularity Motion of specularity
P
P’
Source
FIGURE 2.7: Specularities on convex and concave surfaces behave diﬀerently when the
view changes. With an appropriate choice of source direction and motion, this could be
used to obtain the signs of the principal curvatures.
The quantity of interest is δa = (V

− V ) t, where t =
1
δs
−−→
PP

is tangent
to the surface, and δs is the (small) distance between P and P

: if δa is positive,
then the specularity moves in the direction of the view (back of the spoon), and if
it is negative, the specularity moves against the direction of the view (bowl of the
spoon). By construction, we have V = 2(S N)N −S, and
V

) t = 2(S N)(δN t) = 2(S N)(δs)(II(t, t)).
Thus, for a concave surface, the specularity always moves against the view, and for
a convex surface, it always moves with the view. Things are more complex with
hyperbolic surfaces; the specularity may move with the view, against the view, or
perpendicular to the view (when t is an asymptotic direction).
1
Of course, there is a simpler way to distinguish (by sight) the concave bowl of a spoon from
its convex back: it is just the side where your reﬂection is upside down. This property of concave
mirrors was demonstrated to one of the authors by one of his friends at a dinner party, causing
much consternation since the author in question was at the time bragging about diﬀerential
geometry, and his friend, who does not pretend to know anything about mathematics, just looked
at herself in the spoon.
Section 2.2 Inference from Shading 43
log I log p
dlog
dx
dlog I
dx
dlog p
dx
dlog p
dx
Thresholded
Integrate
this to get
lightness,
which is
log + constant
logρ
ρ
ρ
FIGURE 2.8: The lightness algorithm is easiest to illustrate for a 1D image. In the top
row, the graph on the left shows log ρ(x), that in the center log I(x), and that on the
right their sum, which is log C. The log of image intensity has large derivatives at changes
in surface reﬂectance and small derivatives when the only change is due to illumination
gradients. Lightness is recovered by diﬀerentiating the log intensity, thresholding to dis-
pose of small derivatives, and integrating at the cost of a missing constant of integration.
2.2.3 Inferring Lightness and Illumination
If we could estimate the albedo of a surface from an image, then we would know a
property of the surface itself, rather than a property of a picture of the surface. Such
properties are often called intrinsic representations. They are worth estimating,
because they do not change when the imaging circumstances change. It might seem
Section 2.2 Inference from Shading 44
that albedo is diﬃcult to estimate, because there is an ambiguity linking albedo
and illumination; for example, a high albedo viewed under middling illumination
will give the same brightness as a low albedo viewed under bright light. However,
humans can report whether a surface is white, gray, or black (the lightness of the
surface), despite changes in the intensity of illumination (the brightness). This skill
is known as lightness constancy. There is a lot of evidence that human lightness
constancy involves two processes: one process compares the brightness of various
image patches and uses this comparison to determine which patches are lighter and
which darker; the second establishes some form of absolute standard to which these
comparisons can be referred (e.g. Gilchrist et al. (1999)).
Current lightness algorithms were developed in the context of simple scenes.
In particular, we assume that the scene is ﬂat and frontal; that surfaces are diﬀuse,
or that specularities have been removed; and that the camera responds linearly. In
this case, the camera response C at a point x is the product of an illumination
term, an albedo term, and a constant that comes from the camera gain:
C(x) = k
c
I(x)ρ(x).
If we take logarithms, we get
log C(x) = log k
c
+ log I(x) + log ρ(x).
We now make a second set of assumptions:
• First, we assume that albedoes change only quickly over space. This means
that a typical set of albedoes will look like a collage of papers of diﬀerent grays.
This assumption is quite easily justiﬁed: There are relatively few continuous
changes of albedo in the world (the best example occurs in ripening fruit),
and changes of albedo often occur when one object occludes another (so we
would expect the change to be fast). This means that spatial derivatives of
the term log ρ(x) are either zero (where the albedo is constant) or large (at a
change of albedo).
• Second, illumination changes only slowly over space. This assumption is
somewhat realistic. For example, the illumination due to a point source will
change relatively slowly unless the source is very close, so the sun is a par-
ticularly good source for this method, as long as there are no shadows. As
another example, illumination inside rooms tends to change very slowly be-
cause the white walls of the room act as area sources. This assumption fails
dramatically at shadow boundaries, however. We have to see these as a spe-
cial case and assume that either there are no shadow boundaries or that we
know where they are.
We can now build algorithms that use our model. The earliest algorithm is
the Retinex algorithm of Land and McCann (1971); this took several forms, most
of which have fallen into disuse. The key insight of Retinex is that small gradients
are changes in illumination, and large gradients are changes in lightness. We can
use this by diﬀerentiating the log transform, throwing away small gradients, and
integrating the results (Horn 1974); these days, this procedure is widely known as
Section 2.2 Inference from Shading 45
Retinex. There is a constant of integration missing, so lightness ratios are available,
but absolute lightness measurements are not. Figure 2.8 illustrates the process for
a one-dimensional example, where diﬀerentiation and integration are easy.
This approach can be extended to two dimensions as well. Diﬀerentiating
and thresholding is easy: at each point, we estimate the magnitude of the gradient;
if the magnitude is less than some threshold, we set the gradient vector to zero;
otherwise, we leave it alone. The diﬃculty is in integrating these gradients to get
the log albedo map. The thresholded gradients may not be the gradients of an
image because the mixed second partials may not be equal (integrability again;
compare with Section 2.2.4).
Form the gradient of the log of the image
At each pixel, if the gradient magnitude is below
a threshold, replace that gradient with zero
Reconstruct the log-albedo by solving the minimization
problem described in the text
Obtain a constant of integration
Add the constant to the log-albedo, and exponentiate
Algorithm 2.1: Determining the Lightness of Image Patches.
The problem can be rephrased as a minimization problem: choose the log
albedo map whose gradient is most like the thresholded gradient. This is a relatively
simple problem because computing the gradient of an image is a linear operation.
The x-component of the thresholded gradient is scanned into a vector p, and the y-
component is scanned into a vector q. We write the vector representing log-albedo
as l. Now the process of forming the x derivative is linear, and so there is some
matrix /
x
, such that /
x
l is the x derivative; for the y derivative, we write the
corresponding matrix /
y
.
The problem becomes ﬁnding the vector l that minimizes
[ /
x
l −p [
2
+ [ /
y
l −q [
2
.
This is a quadratic minimization problem, and the answer can be found by a linear
process. Some special tricks are required because adding a constant vector to l
cannot change the derivatives, so the problem does not have a unique solution. We
explore the minimization problem in the exercises.
The constant of integration needs to be obtained from some other assumption.
There are two obvious possibilities:
• we can assume that the brightest patch is white;
• we can assume that the average lightness is constant.
We explore the consequences of these models in the exercises.
More sophisticated algorithms are now available, but there were no quan-
titative studies of performance until recently. Grosse et al. built a dataset for
Section 2.2 Inference from Shading 46
Image
Shading
Albedo
FIGURE 2.9: Retinex remains a strong algorithm for recovering albedo from images. Here
we show results from the version of Retinex described in the text applied to an image of
a room (left) and an image from a collection of test images due to Grosse et al. (2009).
The center-left column shows results from Retinex for this image, and the center-
right column shows results from a variant of the algorithm that uses color reasoning to
improve the classiﬁcation of edges into albedo versus shading. Finally, the right column
shows the correct answer, known by clever experimental methods used when taking the
pictures. This problem is very hard; you can see that the albedo images still contain
some illumination signal. Part of this ﬁgure courtesy Kevin Karsch, U. Illinois. Part
of this ﬁgure was originally published as Figure 3 of “Ground truth dataset and baseline
evaluations for intrinsic image algorithms,” by R. Grosse, M. Johnson, E. Adelson, and
W. Freeman, Proc. IEEE ICCV 2009, c _ IEEE, 2009.
evaluating lightness algorithms, and show that a version of the procedure we de-
scribe performs extremely well compared to more sophisticated algorithms (2009).
The major diﬃculty with all these approaches is caused by shadow boundaries,
which we discuss in Section 3.5.2.
2.2.4 Photometric Stereo: Shape from Multiple Shaded Images
It is possible to reconstruct a patch of surface from a series of pictures of that surface
taken under diﬀerent illuminants. First, we need a camera model. For simplicity,
we choose a camera situated so that the point (x, y, z) in space is imaged to the
point (x, y) in the camera (the method we describe works for the other camera
models described in Chapter 1).
In this case, to measure the shape of the surface, we need to obtain the
depth to the surface. This suggests representing the surface as (x, y, f(x, y))—a
Section 2.2 Inference from Shading 47
representation known as a Monge patch after the French military engineer who ﬁrst
used it (Figure 2.10). This representation is attractive because we can determine a
unique point on the surface by giving the image coordinates. Notice that to obtain
a measurement of a solid object, we would need to reconstruct more than one patch
because we need to observe the back of the object.
x
y
height
Image
plane
direction
of projection
FIGURE 2.10: A Monge patch is a representation of a piece of surface as a height function.
For the photometric stereo example, we assume that an orthographic camera—one that
maps (x, y, z) in space to (x, y) in the camera—is viewing a Monge patch. This means
that the shape of the surface can be represented as a function of position in the image.
Photometric stereo is a method for recovering a representation of the Monge
patch from image data. The method involves reasoning about the image intensity
values for several diﬀerent images of a surface in a ﬁxed view illuminated by diﬀerent
sources. This method recovers the height of the surface at points corresponding to
each pixel; in computer vision circles, the resulting representation is often known
as a height map, depth map, or dense depth map.
Fix the camera and the surface in position, and illuminate the surface using
a point source that is far away compared with the size of the surface. We adopt a
local shading model and assume that there is no ambient illumination (more about
this later) so that the brightness at a point x on the surface is
B(x) = ρ(x)N(x) S
1
,
where N is the unit surface normal and S
1
is the source vector. We can write
B(x, y) for the radiosity of a point on the surface because there is only one point on
the surface corresponding to the point (x, y) in the camera. Now we assume that
the response of the camera is linear in the surface radiosity, and so have that the
Section 2.2 Inference from Shading 48
value of a pixel at (x, y) is
I(x, y) = kB(x)
= kB(x, y)
= kρ(x, y)N(x, y) S
1
= g(x, y) V
1
,
where g(x, y) = ρ(x, y)N(x, y) and V
1
= kS
1
, where k is the constant connecting
the camera response to the input radiance.
FIGURE 2.11: Five synthetic images of a sphere, all obtained in an orthographic view
from the same viewing position. These images are shaded using a local shading model
and a distant point source. This is a convex object, so the only view where there is no
visible shadow occurs when the source direction is parallel to the viewing direction. The
variations in brightness occuring under diﬀerent sources code the shape of the surface.
In these equations, g(x, y) describes the surface, and V
1
is a property of the
illumination and of the camera. We have a dot product between a vector ﬁeld g(x, y)
and a vector V
1
, which could be measured; with enough of these dot products, we
could reconstruct g and so the surface.
Now if we have n sources, for each of which V
i
is known, we stack each of
these V
i
into a known matrix 1, where
1 =
⎛
⎜
⎜
⎝
V
T
1
V
T
2
. . .
V
T
n
⎞
⎟
⎟
⎠
.
For each image point, we stack the measurements into a vector
i(x, y) = ¦I
1
(x, y), I
2
(x, y), . . . , I
n
(x, y)¦
T
.
Section 2.2 Inference from Shading 49
Notice that we have one vector per image point; each vector contains all the image
brightnesses observed at that point for diﬀerent sources. Now we have
i(x, y) = 1g(x, y),
and g is obtained by solving this linear system—or rather, one linear system per
point in the image. Typically, n > 3, so that a least-squares solution is appropriate.
This has the advantage that the residual error in the solution provides a check on
our measurements.
Substantial regions of the surface might be in shadow for one or the other
light (see Figure 2.11). We assume that all shadowed regions are known, and deal
only with points that are not in shadow for any illuminant. More sophisticated
strategies can infer shadowing because shadowed points are darker than the local
geometry predicts.
We can extract the albedo from a measurement of g because N is the unit
normal. This means that [g(x, y)[= ρ(x, y). This provides a check on our measure-
ments as well. Because the albedo is in the range zero to one, any pixels where
[g[ is greater than one are suspect—either the pixel is not working or 1 is incor-
rect. Figure 2.12 shows albedo recovered using this method for the images shown
in Figure 2.11.
0
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
-5
0
5
10
15
0
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
-10
0
10
20
FIGURE 2.12: The image on the left shows the magnitude of the vector ﬁeld g(x, y) recov-
ered from the input data of Figure 2.11 represented as an image—this is the reﬂectance
of the surface. The center ﬁgure shows the normal ﬁeld, and the right ﬁgure shows the
height ﬁeld.
We can extract the surface normal from g because the normal is a unit vector
N(x, y) =
g(x, y)
[g(x, y)[
.
Figure 2.12 shows normal values recovered for the images of Figure 2.11.
The surface is (x, y, f(x, y)), so the normal as a function of (x, y) is
N(x, y) =
1
_
1 +
∂f
∂x
2
+
∂f
∂y
2
_
∂f
∂x
,
∂f
∂y
, 1
_
T
.
To recover the depth map, we need to determine f(x, y) from measured values of
the unit normal.
Section 2.2 Inference from Shading 50
Obtain many images in a ﬁxed view under diﬀerent illuminants
Determine the matrix 1 from source and camera information
Inferring albedo and normal:
For each point in the image array that is not shadowed
Stack image values into a vector i
Solve 1g = i to obtain g for this point
Albedo at this point is [g[
Normal at this point is
g
|g|
p at this point is
N
1
N
3
q at this point is
N
2
N
3
end
Check: is (
∂p
∂y
−
∂q
∂x
)
2
small everywhere?
Integration:
Top left corner of height map is zero
For each pixel in the left column of height map
height value = previous height value + corresponding q value
end
For each row
For each element of the row except for leftmost
height value = previous height value + corresponding p value
end
end
Algorithm 2.2: Photometric Stereo.
Assume that the measured value of the unit normal at some point (x, y) is
(a(x, y), b(x, y), c(x, y)). Then
∂f
∂x
=
a(x, y)
c(x, y)
and
∂f
∂y
=
b(x, y)
c(x, y)
.
We have another check on our data set, because
∂
2
f
∂x∂y
=
∂
2
f
∂y∂x
,
so we expect that
∂
_
a(x,y)
c(x,y)
_
∂y
−
∂
_
b(x,y)
c(x,y)
_
∂x
should be small at each point. In principle it should be zero, but we would have
to estimate these partial derivatives numerically and so should be willing to accept
small values. This test is known as a test of integrability, which in vision applications
always boils down to checking that mixed second partials are equal.
Section 2.2 Inference from Shading 51
FIGURE 2.13: Photometric stereo could become the method of choice to capture complex
deformable surfaces. On the top, three images of a garment, lit from diﬀerent directions,
which produce the reconstruction shown on the top right. A natural way to obtain three
diﬀerent images at the same time is to use a color camera; if one has a red light, a green
light, and a blue light, then a single color image frame can be treated as three images
under three separate lights. On the bottom, an image of the garment captured in this
way, which results in the photometric stereo reconstruction on the bottom right. This
ﬁgure was originally published as Figure 6 of “Video Normals from Colored Lights,” G. J.
Brostow, C. Hern´ andez, G. Vogiatzis, B. Stenger, and R. Cipolla, IEEE Transactions on
Pattern Analysis and Machine Intelligence, 2011 c _ IEEE, 2011.
Assuming that the partial derivatives pass this sanity test, we can reconstruct
the surface up to some constant depth error. The partial derivative gives the change
in surface height with a small step in either the x or the y direction. This means
we can get the surface by summing these changes in height along some path. In
particular, we have
f(x, y) =
_
C
_
∂f
∂x
,
∂f
∂y
_
dl +c,
where C is a curve starting at some ﬁxed point and ending at (x, y), and c is a
constant of integration, which represents the (unknown) height of the surface at
the start point. The recovered surface does not depend on the choice of curve (ex-
ercises). Another approach to recovering shape is to choose the function f(x, y)
whose partial derivatives most look like the measured partial derivatives. Fig-
ure 2.12 shows the reconstruction obtained for the data shown in Figure 2.11.
Current reconstruction work tends to emphasize geometric methods that re-
construct from multiple views. These methods are very important, but often require
feature matching, as we shall see in Chapters 7 and 8. This tends to mean that
it is hard to get very high spatial resolution, because some pixels are consumed
in resolving features. Recall that resolution (which corresponds roughly to the
spatial frequencies that can be reconstructed accurately) is not the same as ac-
Section 2.3 Modelling Interreﬂection 52
curacy (which involves a method providing the right answers for the properties it
estimates). Feature-based methods are capable of spectacularly accurate recon-
structions. Because photometric cues have such spatial high resolution, they are a
topic of considerable current interest. One way to use photometric cues is to try
and match pixels with the same brightness across diﬀerent cameras; this is diﬃ-
cult, but produces impressive reconstructions. Another is to use photometric stereo
ideas. For some applications, photometric stereo is particularly atractive because
one can get reconstructions from a single view direction—this is important, because
we cannot always set up multiple cameras. In fact, with a trick, it is possible to
get reconstructions from a single frame. A natural way to obtain three diﬀerent
images at the same time is to use a color camera; if one has a red light, a green light
and a blue light, then a single color image frame can be treated as three images
under three separate lights, and photometric stereo methods apply. In turn, this
means that photometric stereo methods could be used to recover high-resolution
reconstructions of deforming surfaces in a relatively straightforward way. This is
particularly useful when it is diﬃcult to get many cameras to view the object.
Figure 2.13 shows one application to reconstructing cloth in video (from Brostow
et al. (2011)), where multiple view reconstruction is complicated by the need to
synchronize frames (alternatives are explored in, for example, White et al. (2007)
or Bradley et al. (2008b)).
2.3 MODELLING INTERREFLECTION
The diﬃculty with a local shading model is that it doesn’t account for all light. The
alternative is a global shading model, where we account for light arriving from other
surfaces as well as from the luminaire. As we shall see, such models are tricky to
work with. In such models, each surface patch receives power from all the radiating
surfaces it can see. These surfaces might radiate power that they generate internally
because they are luminaires, or they might simply reﬂect power. The general form
of the model will be:
_
Power leaving
a patch
_
=
_
Power generated
by that patch
_
+
_
Power received from
other patches and reﬂected
_
This means we need to be able to model the power received from other patches and
reﬂected. We will develop a model assuming that all surfaces are diﬀuse. This leads
to a somewhat simpler model, and describes all eﬀects that are currently of interest
to vision (it is complicated, but not diﬃcult, to build more elaborate models). We
will also need some radiometric terminology.
2.3.1 The Illumination at a Patch Due to an Area Source
The appropriate unit for illumination is radiance, deﬁned as
the power (amount of energy per unit time) traveling at some point in a
speciﬁed direction, per unit area perpendicular to the direction of travel,
per unit solid angle.
The units of radiance are watts per square meter per steradian (Wm
−2
sr
−1
). The
deﬁnition of radiance might look strange, but it is consistent with the most basic
Section 2.3 Modelling Interreﬂection 53
phenomenon in radiometry: the amount of energy a patch collects from a source
depends both on how large the source looks from the patch and on how large the
patch looks from the source.
r
x
dA
x
dA
N
N
θ
θ
s
s
s
s
S
FIGURE 2.14: A patch with area dA views an area source S. We compute the power
received by the patch by summing the contributions of each element on S, using the
notation indicated in this ﬁgure.
It is important to remember that the square meters in the units for radiance
are foreshortened (i.e., perpendicular to the direction of travel) to account for this
phenomenon. Assume we have two elements, one at x with area dA and the other
at x
s
with area dA
s
. Write the angular direction from x to x
s
as x → x
s
, and
deﬁne the angles θ and θ
s
as in Figure 2.14. Then the solid angle subtended by
element 2 at element 1 is
dω
2(1)
=
cos θ
s
dA
s
r
2
,
so the power leaving x toward x
s
is
d
2
P
1→2
= (radiance)(foreshortened area)(solid angle)
= L(x, x →x
s
)(cos θdA)(dω
2(1)
)
= L(x, x →x
s
)
_
cos θ cos θ
s
r
2
_
dA
s
dA.
By a similar argument, the same expression yields the power arriving at x from
x
2
; this means that, in a vacuum, radiance is constant along (unoccluded) straight
lines.
We can now compute the power that an element dA collects from an area
source, by summing the contributions of elements over that source. Using the
notation of Figure 2.14, we get
dP
S→dA
=
__
S
L(x
s
, x
s
→x)
_
cos θ
s
cos θ
r
2
_
dA
s
_
dA.
Section 2.3 Modelling Interreﬂection 54
To get a more useful area source model, we need further units.
θ
φ
dφ
dθ
dA
θ
n
r
FIGURE 2.15: A hemisphere on a patch of surface, to show our angular coordinates for
computing radiometric quantities. The coordinate axes are there to help you see the
drawing as a 3D surface. An inﬁnitesimal patch of surface with area dA which is distance
r away is projected onto the unit hemisphere centered at the relevant point; the resulting
area is the solid angle of the patch, marked as dθdφ. In this case, the patch is small so that
the area and hence the solid angle is (1/r
2
)dAcos θ
n
, where θ
n
is the angle of inclination
of the patch.
2.3.2 Radiosity and Exitance
We are dealing with diﬀuse surfaces, and our deﬁnition of a diﬀuse surface is that
the intensity (formally, the radiance) leaving the surface is independent of the
direction in which it leaves. There is no point in describing the intensity of such
a surface with radiance (which explicitly depends on direction). The appropriate
unit is radiosity, deﬁned as
the total power leaving a point on a surface per unit area on the surface.
Radiosity, which is usually written as B(x), has units watts per square meter
(Wm
−2
). To obtain the radiosity of a surface at a point, we can sum the radiance
leaving the surface at that point over the whole exit hemisphere. Thus, if x is a
point on a surface emitting radiance L(x, θ, φ), the radiosity at that point is
B(x) =
_
Ω
L(x, θ, φ) cos θdω,
where Ω is the exit hemisphere, dω is unit solid angle, and the term cos θ turns
foreshortened area into area (look at the deﬁnitions of radiance and of radiosity
again). We could substitute dω = sin θdθdφ, using the units of Figure 2.15.
Section 2.3 Modelling Interreﬂection 55
Consider a surface element as in Figure 2.14. We have computed how much
power it receives from the source as a function of the source’s radiance. The surface
element is diﬀuse, and its albedo is ρ(x). The albedo is the fraction of incoming
power that the surface radiates, so the radiosity due to power received from the
area source is
B(x) =
dP
S→dA
dA
= ρ(x)
__
S
L(x
s
, x
s
→x)
_
cos θ
s
cos θ
r
2
_
dA
s
_
.
Now if a point u on a surface has radiosity B(u), what is the radiance leaving
the surface in some direction? We write L for the radiance, which is independent
of angle, and we must have
B(u) =
_
Ω
L(x, θ, φ) cos θdω = L(u)
_
Ω
cos θdω = L(u)π.
This means that if the area source has radiosity B(x
s
), then the radiosity at the
element due to the power received from the area source is
B(x) = ρ
__
S
L(x
s
, x
s
→x)
_
cos θ
s
cos θ
r
2
_
dA
s
_
= ρ
__
S
B(x)
π
_
cos θ
s
cos θ
r
2
_
dA
s
_
=
ρ
π
__
S
B(x)
_
cos θ
s
cos θ
r
2
_
dA
s
_
.
Our ﬁnal step is to model illumination generated internally in a surface—light
generated by a luminaire, rather than reﬂected from a surface. We assume there
are no directional eﬀects in the luminaire and that power is uniformly distributed
across outgoing directions (this is the least plausible component of the model, but
is usually tolerable). We use the unit exitance, which is deﬁned as
the total power internally generated power leaving a point on a surface
per unit area on the surface.
2.3.3 An Interreﬂection Model
We can now write a formal model of interreﬂections for diﬀuse surfaces by substi-
tuting terms into the original expression. Recall that radiosity is power per unit
area, write E(x) for exitance at the point x, write x
s
for a coordinate that runs
over all surface patches, o for the set of all surfaces, dA for the element of area at
x, V(x, x
s
) for a function that is one if the two points can see each other and zero
otherwise, and cos θ, cos θ
s
, r, as in Figure 2.14. We obtain
Power leaving = Power generated + Power received from
a patch by that patch other patches and reﬂected
B(x)dA = E(x)dA +
ρ(x)
π
_
S
⎡
⎣
cos θ cos θ
s
r
2

V(x, x
s
)
⎤
⎦
B(x
s
)dA
s
dA
Section 2.3 Modelling Interreﬂection 56
and so, dividing by area, we have
B(x) = E(x) +
ρ(x)
π
_
S
_
cos θ cos θ
s
r
2
Vis(x, x
s
)
¸
B(x
s
)dA
s
.
It is usual to write
K(x, x
s
) =
cos θ cos θ
s
πr
2
and refer to K as the interreﬂection kernel. Substituting gives
B(x) = E(x) +ρ(x)
_
S
K(x, x
s
)Vis(x, x
s
)B(x
s
)dA
x
s
an equation where the solution appears inside the integral. Equations of this form
are known as Fredholm integral equations of the second kind. This particular
equation is a fairly nasty sample of the type because the interreﬂection kernel
generally is not continuous and may have singularities. Solutions of this equation
can yield quite good models of the appearance of diﬀuse surfaces, and the topic
supports a substantial industry in the computer graphics community (good places
to start for this topic are Cohen and Wallace (1993) or Sillion (1994)). The model
produces good predictions of observed eﬀects (Figure 2.16).
2.3.4 Qualitative Properties of Interreﬂections
Interreﬂections are a problem, because they are diﬃcult to account for in our il-
lumination model. For example, photometric stereo as we described it uses the
model that light at a surface patch comes only from a distant light source. One
could reﬁne the method to take into account nearby light sources, but it is much
more diﬃcult to deal with interreﬂections. Once one accounts for interreﬂections,
the brightness of each surface patch could be aﬀected by the conﬁguration of every
other surface patch, making a very nasty global inference problem. While there
have been attempts to build methods that can infer shape in the presence of in-
terreﬂections (Nayar et al. 1991a), the problem is extremely diﬃcult. One source
of diﬃculties is that one may need to account for every radiating surface in the
solution, even distant surfaces one cannot see.
An alternative strategy to straightforward physical inference is to understand
the qualitative properties of interreﬂected shading. By doing so, we may be able
to identify cases that are easy to handle, the main types of eﬀect, and so on. The
eﬀects can be quite large. For example, Figure 2.17 shows views of the interior of two
rooms. One room has black walls and contains black objects. The other has white
walls and contains white objects. Each is illuminated (approximately!) by a distant
point source. Given that the intensity of the source is adjusted appropriately, the
local shading model predicts that these pictures would be indistinguishable. In fact,
the black room has much darker shadows and crisper boundaries at the creases of the
polyhedra than the white room. This is because surfaces in the black room reﬂect
less light onto other surfaces (they are darker), whereas in the white room other
surfaces are signiﬁcant sources of radiation. The sections of the camera response to
the radiosity (these are proportional to radiosity for diﬀuse surfaces) shown in the
ﬁgure are hugely diﬀerent qualitatively. In the black room, the radiosity is constant
in patches, as a local shading model would predict, whereas in the white room slow
Section 2.3 Modelling Interreﬂection 57
Illumination from
an infinitely distant
point source, in this
direction
predicted
observed
FIGURE 2.16: The model described in the text produces quite accurate qualitative predic-
tions for interreﬂections. The top ﬁgure shows a concave right-angled groove illuminated
by a point source at inﬁnity where the source direction is parallel to the one face. On
the left of the bottom row is a series of predictions of the radiosity for this conﬁgura-
tion. These predictions have been scaled to lie on top of one another; the case ρ → 0
corresponds to the local shading model. On the right, an observed image intensity for
an image of this form for a corner made of white paper, showing the roof-like gradient in
radiosity associated with the edge. A local shading model predicts a step. This ﬁgure was
originally published as Figures 5 and 7 of “Mutual Illumination,” by D.A. Forsyth and
A.P. Zisserman, Proc. IEEE CVPR, 1989, c _ IEEE, 1989.
image gradients are quite common; these occur in concave corners, where object
faces reﬂect light onto one another.
First, interreﬂections have a characteristic smoothing eﬀect. This is most
obviously seen when one tries to interpret a stained glass window by looking at
the pattern it casts on the ﬂoor; this pattern is almost always a set of indistinct
colored blobs. The eﬀect is seen most easily with the crude model illustrated in
Figure 2.18. The geometry consists of a patch with a frontal view of an inﬁnite
plane, which is a unit distance away and carries a radiosity sin ωx. There is no
reason to vary the distance of the patch from the plane, because interreﬂection
problems have scale invariant solutions, which means that the solution for a patch
two units away can be obtained by reading our graph at 2ω. The patch is small
enough that its contribution to the plane’s radiosity can be ignored. If the patch is
Section 2.3 Modelling Interreﬂection 58
A B C D E F G A B C D
FIGURE 2.17: The column on the left shows data from a room with matte black walls
and containing a collection of matte black polyhedral objects; that on the right shows
data from a white room containing white objects. The images are qualitatively diﬀerent,
with darker shadows and crisper boundaries in the black room and bright reﬂexes in the
concave corners in the white room. The graphs show sections of the image intensity along
the corresponding lines in the images. This ﬁgure was originally published as Figures 17,
18, 19, and 20 of “Mutual Illumination,” by D.A. Forsyth and A.P. Zisserman, Proc.
IEEE CVPR, 1989, c _ IEEE, 1989.
slanted by σ with respect to the plane, it carries radiosity that is nearly periodic,
with spatial frequency ω cos σ. We refer to the amplitude of the component at this
frequency as the gain of the patch and plot the gain in Figure 2.18. The important
property of this graph is that high spatial frequencies have a diﬃcult time jumping
the gap from the plane to the patch. This means that shading eﬀects with high
spatial frequency and high amplitude generally cannot come from distant surfaces
(unless they are abnormally bright).
The extremely fast fall-oﬀ in amplitude with spatial frequency of terms due to
distant surfaces means that, if one observes a high-amplitude term at a high spatial
frequency, it is very unlikely to have resulted from the eﬀects of distant, passive
radiators (because these eﬀects die away quickly). There is a convention, which we
see in Section 2.2.3, that classiﬁes eﬀects in shading as due to reﬂectance if they are
fast (“edges”) and the dynamic range is relatively low and due to illumination oth-
erwise. We can expand this convention. There is a mid range of spatial frequencies
that are largely unaﬀected by mutual illumination from distant surfaces because
the gain is small. Spatial frequencies in this range cannot be transmitted by distant
passive radiators unless these radiators have improbably high radiosity. As a result,
spatial frequencies in this range can be thought of as regional properties, which can
result only from interreﬂection eﬀects within a region.
The most notable regional properties are probably reﬂexes— small bright
patches that appear mainly in concave regions (illustrated in Figure 2.19). A second
Section 2.4 Shape from One Shaded Image 59
0 0.5 1 1.5 2 2.5 3
−0.2
0
0.2
0.4
0.6
0.8
1
Source spatial frequency − radians per unit length
G
a
i
n
Frontal − exact solution
FIGURE 2.18: A small patch views a plane with sinusoidal radiosity of unit amplitude.
This patch has a (roughly) sinusoidal radiosity due to the eﬀects of the plane. We refer
to the amplitude of this component as the gain of the patch. The graph shows numerical
estimates of the gain for patches at 10 equal steps in slant angle, from 0 to π/2, as a
function of spatial frequency on the plane. The gain falls extremely fast, meaning that
large terms at high spatial frequencies must be regional eﬀects, rather than the result of
distant radiators. This is why it is hard to determine the pattern in a stained glass window
by looking at the ﬂoor at the foot of the window. This ﬁgure was originally published
as Figures 1 and 2 from “Shading Primitives: Finding Folds and Shallow Grooves,” J.
Haddon and D.A. Forsyth, Proc. IEEE ICCV, 1998 c _ IEEE, 1998.
important eﬀect is color bleeding, where a colored surface reﬂects light onto another
colored surface. This is a common eﬀect that people tend not to notice unless they
are consciously looking for it. It is quite often reproduced by painters.
2.4 SHAPE FROM ONE SHADED IMAGE
There is good evidence that people get some perception of shape from the shading
pattern in a single image, though the details are uncertain and quite complicated
(see the notes for a brief summary). You can see this evidence in practice: whenever
you display a reconstruction of a surface obtained from images, it is a good idea to
shade that reconstruction using image pixels, because it always looks more accurate.
In fact, quite bad reconstructions can be made to look good with this method.
White and Forsyth (2006) use this trick to replace surface albedos in movies; for
example, they can change the pattern on a plastic bag in a movie. Their method
builds and tracks very coarse geometric reconstructions, uses a form of regression
to recover the original shading pattern of the object, and then shades the coarse
geometric reconstruction using the original shading pattern (Figure 2.20). In this
ﬁgure, the pictures look plausible, not because the reconstruction is good (it isn’t),
but because the shading pattern masks the errors in geometric reconstruction.
The cue to shape must come from the fact that a surface patch that faces
the light source is brighter than one that faces away from the source. But going
from this observation to a working algorithm remains an open question. The key
seems to be an appropriate use of the image irradiance equation. Assume we have a
surface in the form (x, y, f(x, y)) viewed orthographically along the z-axis. Assume
that the surface is diﬀuse, and its albedo is uniform and known. Assume also that
Section 2.4 Shape from One Shaded Image 60
predicted
observed
FIGURE 2.19: Reﬂexes occur quite widely; they are usually caused by a favorable view of
a large reﬂecting surface. In the geometry shown on the top, the shadowed region of the
cylindrical bump sees the plane background at a fairly favorable angle: if the background
is large enough, nearly half the hemisphere of the patch at the base of the bump is a view
of the plane. This means there will be a reﬂex with a large value attached to the edge
of the bump and inside the cast shadow region (which a local model predicts as black).
There is another reﬂex on the other side, too, as the series of solutions (again normalized
for easy comparison) on the left show. On the right, an observation of this eﬀect in a real
scene. This ﬁgure was originally published as Figures 24 and 26 of “Mutual Illumination,”
by D.A. Forsyth and A.P. Zisserman, Proc. IEEE CVPR, 1989, c _ IEEE, 1989.
the model of Section 2.1.3 applies, so that the shading at a point with normal N
is given by some function R(N) (the function of our model is R(N) = N S, but
others could be used). Now the normal of our surface is a function of the two ﬁrst
partial derivatives
p =
∂f
∂x
, q =
∂f
∂y
so we can write R(p, q). Assume that the camera is radiometrically calibrated, so
we can proceed from image values to intensity values. Write the intensity at x, y
as I(x, y). Then we have
R(p, q) = I(x, y).
This is a ﬁrst order partial diﬀerential equation, because p and q are partial deriva-
tives of f. In principle, we could set up some boundary conditions and solve this
equation. Doing so reliably and accurately for general images remains outside our
competence, 40 years after the problem was originally posed by Horn (1970a).
Section 2.5 Notes 61
FIGURE 2.20: On the left, an original frame from a movie sequence of a deforming plastic
bag. On the right, two frames where the original texture has been replaced by another.
The method used is a form of regression; its crucial property is that it has a very weak
geometric model, but is capable of preserving the original shading ﬁeld of the image. If
you look closely at the albedo (i.e., the black pattern) of the bag, you may notice that it
is inconsistent with the wrinkles on the bag, but because the shading has been preserved,
the ﬁgures look quite good. This is indirect evidence that shading is a valuable cue to
humans. Little is known about how this cue is to be exploited, however. This ﬁgure was
originally published as Figure 10 of “Retexturing single views using texture and shading,”
by R. White and D.A. Forsyth, Proc. European Conference on Computer Vision. Springer
Lecture Notes in Computer Science, Volume 3954, 2006 c _ Springer 2006.
There are a variety of diﬃculties here. The physical model is a poor model
of what actually happens at surfaces because any particular patch is illuminated
by other surface patches, as well as by the source. We expect to see a rich variety
of geometric constraints on the surface we reconstruct, and it is quite diﬃcult to
formulate shape from shading in a way that accomodates these constraints and still
has a solution. Shading is a worthwhile cue to exploit, because we can observe
shading at extremely high spatial resolutions, but this means we must work with
very high dimensional models to reconstruct. Some schemes for shading recon-
struction can be unstable, but there appears to be no theory to guide us to stable
schemes. We very seldom actually see isolated surfaces of known albedo, and there
are no methods that are competent to infer both shading and albedo, though there
is some reason to hope that such methods can be built. We have no theory that is
capable of predicting the errors in shading-based reconstructions from ﬁrst princi-
ples. All this makes shape inference from shading in a single image one of the most
frustrating open questions in computer vision.
2.5 NOTES
Horn started the systematic study of shading in computer vision, with important
papers on recovering shape from a local shading model using a point source (in
(Horn 1970b), (Horn 1975)), with a more recent account in Horn (1990).
Section 2.5 Notes 62
A
B
FIGURE 2.21: This picture shows two important mechanisms by which it might be possible
to infer surface shape from single images. First, patches that face away from the light
(like A, on the left) are darker than those that face the light (B). Second, shadows pick
out relief—for example, small dents in a surface (more easily seen in the detail patch on
the right), have a bright face facing the light and a dark face which is in shadow. Peter
Wilson c _ Dorling Kindersley, used with permission.
Models of Shading
The ﬁrst edition of this book contained more formal radiometry, which was widely
disliked (and for good reason; making the subject exciting is beyond our skills).
We’ve cut this down, and tried to avoid using the ideas, but point those who
really want to know more toward that earlier edition. We strongly recommend
Fran¸ cois Sillion’s excellent book (Sillion 1994) for its clear account of radiometric
calculations. There are a variety of more detailed publications for reference (Nayar
et al. 1991c). Our discussion of reﬂection is thoroughly superﬁcial. The specu-
lar plus diﬀuse model appears to be originally due to Cook, Torrance, and Spar-
row (Torrance and Sparrow 1967, Cook and Torrance 1987). A variety of modiﬁ-
cations of this model appear in computer vision and computer graphics. Reﬂection
models can be derived by combining a statistical description of surface roughness
with electromagnetic considerations (e.g., Beckmann and Spizzichino (1987)) or by
Section 2.5 Notes 63
adopting scattering models (as in the work of Torrance and Sparrow (1967) and
of Cook and Torrance (1987)).
It is commonly believed that rough surfaces are Lambertian. This belief has a
substantial component of wishful thinking because rough surfaces often have local
shadowing eﬀects that make the radiance reﬂected quite strongly dependent on
the illumination angle. For example, a stucco wall illuminated at a near grazing
angle shows a clear pattern of light and dark regions where facets of the surface
face toward the light or are shadowed. If the same wall is illuminated along the
normal, this pattern largely disappears. Similar eﬀects at a ﬁner scale are averaged
to endow rough surfaces with measurable departures from a Lambertian model (for
details, see Koenderink et al. (1999), Nayar and Oren (1993), (1995), Oren and
Nayar (1995), and Wolﬀ et al. (1998)).
Inference from Shading
Registered images are not essential for radiometric calibration. For example, it is
suﬃcient to have two images where we believe the histogram of E
ij
values is the
same (Grossberg and Nayar 2002). This occurs, for example, when the images are
of the same scene, but are not precisely registered. Patterns of intensity around
edges also can reveal calibration (Lin et al. 2004).
There has not been much recent study of lightness constancy algorithms. The
basic idea is due to Land and McCann (1971). Their work was formalized for
the computer vision community by Horn (1974). A variation on Horn’s algorithm
was constructed by Blake (1985). This is the lightness algorithm we describe. It
appeared originally in a slightly diﬀerent form, where it was called the Retinex
algorithm (Land and McCann 1971). Retinex was originally intended as a color
constancy algorithm. It is surprisingly diﬃcult to analyze (Brainard and Wandell
1986).
Retinex estimates the log-illumination term by subtracting the log-albedo
from the log-intensity. This has the disadvantage that we do not impose any struc-
tural constraints on illumination. This point has largely been ignored, because the
main focus has been on albedo estimates. However, albedo estimates are likely to
be improved by balancing violations of albedo constraints with those of illumination
constraints.
Lightness techniques are not as widely used as they should be, particularly
given that there is some evidence that they produce useful information on real
images (Brelstaﬀ and Blake 1987). Classifying illumination versus albedo simply
by looking at the magnitude of the gradient is crude, and ignores important cues.
Sharp shading changes occur at shadow boundaries or normal discontinuities, but
using chromaticity (Funt et al. 1992) or multiple images under diﬀerent lighting
conditions (Weiss 2001) yields improved estimates. One can learn to distinguish
illumination from albedo (Freeman et al. 2000). Discriminative methods to classify
edges into albedo or shading help (Tappen et al. 2006b) and chromaticity cues
can contribute (Farenzena and Fusiello 2007). Shading and albedo are sometimes
known as intrinsic images. Tappen et al. (2006a) regress local intrinsic image
patches against the image, exploiting the constraint that patches join up. When
more than one image is available, recent methods can recover quite complex surface
Section 2.5 Notes 64
properties (Romeiro et al. 2008). When geometry is available, Yu et al. (1999)
showed signiﬁcant improvements in lightness recovery are possible.
In its original form, photometric stereo is due to Woodham. There are a
number of variants of this useful idea (Horn et al. (1978), Woodham (1979), (1980),
(1989), (1994), Woodham et al. (1991)). Current methods for photometric stereo
require at least two unshadowed views; see Hernandez et al. (2008) which describes
methods to cope in this case. There are a variety of variations on photometric
stereo. Color photometric stereo seems to date to Petrov (1987), with a variant in
Petrov (1991).
Photometric stereo depends only on adopting a local shading model. This
model need not be a Lambertian surface illuminated by a distant point source. If
the brightness of the surface is a known function of the surface normal satisfying a
small number of constraints, photometric stereo is still possible. This is because the
intensity of a pixel in a single view determines the normal up to a one-parameter
family. This means that two views determine the normal. The simplest example of
this case occurs for a surface of known albedo illuminated by a distant point source.
In fact, if the radiosity of the surface is a k-parameter function of the surface
normal, photometric stereo is still possible. The intensity of the pixel in a single
view determines the normal up to a k + 1 parameter family, and k + 1 views give
the normal. For this approach to work, the brightness needs to be given by a
function for which our arithmetic works (e.g., if the brightness of the surface is a
constant function of the surface normal, it isn’t possible to infer any constraint on
the normal from the brightness). One can then recover shape and reﬂectance maps
simultaneously (Garcia-Bermejo et al. (1996); Mukawa (1990); Nayar et al. (1990);
Tagare and de Figueiredo (1992); (1993)).
A converse to photometric stereo might be as follows: Assume we have a
diﬀuse sphere, immersed in an environment where illumination depends only on
direction. What can we determine about the illumination ﬁeld from the surface
brightness? The answer is very little, because diﬀuse surfaces engage in a form of
averaging that heavily smoothes the illumination ﬁeld (Ramamoorthi and Hanrahan
2001). This is valuable because it suggests that complex representations of the
directional properties illumination aren’t required in a diﬀuse world. For example,
this result allowed Jacobs (1981) to produce a form of photometric stereo that
requires no illuminant information, using suﬃcient images.
Interreﬂections
The eﬀects of global shading are often ignored in the shading literature, which
causes a reﬂex response of hostility in one of the authors. The reason to ignore
interreﬂections is that they are extremely hard to analyze, particularly from the
perspective of inferring object properties given the output of a global shading model.
If interreﬂection eﬀects do not change the output of a method much, then it is prob-
ably all right to ignore them. Unfortunately, this line of reasoning is seldom pursued
because it is quite diﬃcult to show that a method is stable under interreﬂections.
The discussion of spatial frequency issues follows Haddon and Forsyth (1998a), af-
ter an idea of Koenderink and van Doorn (1983). Apart from this, there is not
much knowledge about the overall properties of interreﬂected shading, which is an
Section 2.5 Notes 65
important gap in our knowledge. An alternative strategy is to iteratively reestimate
shape using a rendering model (Nayar et al. 1991b).
Horn is also the ﬁrst author to indicate the signiﬁcance of global shading
eﬀects (Horn 1977). Koenderink and van Doorn (1983) noted that the radiosity
under a global model is obtained by taking the radiosity under a local model,
and applying a linear operator. One then studies that operator; in some cases,
its eigenfunctions (often called geometrical modes) are informative. Forsyth and
Zisserman (1989, 1990, 1991) then demonstrated a variety of the qualitative eﬀects
due to interreﬂections.
Shape from One Shaded Image
Shape from shading is an important puzzle. Comprehensive surveys include (Horn
and Brooks (1989); Zhang et al. (1999); Durou et al. (2008b)). In practice, despite
the ongoing demand for high-resolution shape reconstructions, shape-from-shading
has been a disappointment. This may be because, as currently formulated, it solves
a problem that doesn’t commonly occur. Image irradiance equation methods are
formulated to produce reconstructions when there is very little geometric data, but
it is much more common to want to improve the resolution of a method that already
produces quite rich geometric data.
Methods are either too fragile or the reconstructions too poor for the method
to be useful. Some of this may be due to the eﬀects of interreﬂections. Another
source of diﬃculty could be the compromises that need to be made to obtain a
solution in the presence of existence diﬃculties. Most reconstructions shown in
the literature are poor. In a comparative review, Zhang et al. (2002) summarize:
“All the SFS algorithms produce generally poor results when given synthetic data
. . . Results are even worse on real images, and . . . [r]esults on synthetic data
are not generally predictive of results on real data.” More recently, Tankus et al.
(2005) showed good looking reconstructions of various body structures from endo-
scopic images, but cannot compare with veridical information. Prados and Faugeras
show a good-looking reconstruction of a face, but cannot compare with veridical
information (Prados and Faugeras (2005a); (2005b)). Durou et al. (2008a), in a
recent comparative review, show some fair reconstructions on both synthetic and
real data. However, on quite simple shapes methods still produce reconstructions
with profound eccentricities.
These problems have driven a search for methods that do not require a recon-
struction. Some local features of a shading ﬁeld—shading primitives—are revealing
because some geometric structures generate about the same shading pattern what-
ever the illumination. For example, a pit in a surface will always be dark; grooves
and folds tend to appear as a thin, light band next to a thin, dark band; and
the shading on a cylinder is usually either a dark band next to a light band, or
a light band with dark band on either side. This idea originates with Koenderink
and Doorn (1983), and is expounded in (Haddon and Forsyth 1997, Haddon and
Forsyth 1998b, Han and Zhu 2005, Han and Zhu 2007). On a larger spatial scale, the
family of shading patterns that can be produced by a particular object—the illumi-
nation cone—is smaller than one might expect (Basri and Jacobs 2003, Belhumeur
and Kriegman 1998, Georghiades et al. 2001), allowing illumination invariant de-
Section 2.5 Notes 66
tection by detecting elements of such cones or by matching with an image distance
that discounts changes in illumination (Chen et al. 2000, Jacobs et al. 1998).
PROBLEMS
2.1. We see a diﬀuse sphere centered at the origin, with radius one and albedo ρ, in
an orthographic camera, looking down the z-axis. This sphere is illuminated
by a distant point light source whose source direction is (0, 0, 1). There is no
other illumination. Show that the shading ﬁeld in the camera is
ρ
_
1 −x
2
−y
2
2.2. What shapes can the shadow of a sphere take if it is cast on a plane and the
source is a point source?
2.3. We have a square area source and a square occluder, both parallel to a plane.
The source is the same size as the occluder, and they are vertically above one
another with their centers aligned.
(a) What is the shape of the umbra?
(b) What is the shape of the outside boundary of the penumbra?
2.4. We have a square area source and a square occluder, both parallel to a plane.
The edge length of the source is now twice that of the occluder, and they are
vertically above one another with their centers aligned.
(a) What is the shape of the umbra?
(b) What is the shape of the outside boundary of the penumbra?
2.5. We have a square area source and a square occluder, both parallel to a plane.
The edge length of the source is now half that of the occluder, and they are
vertically above one another with their centers aligned.
(a) What is the shape of the umbra?
(b) What is the shape of the outside boundary of the penumbra?
2.6. A small sphere casts a shadow on a larger sphere. Describe the possible shadow
boundaries that occur.
2.7. Explain why it is diﬃcult to use shadow boundaries to infer shape, particularly
if the shadow is cast onto a curved surface.
2.8. As in Figure 2.18, a small patch views an inﬁnite plane at unit distance. The
patch is suﬃciently small that it reﬂects a trivial quantity of light onto the
plane. The plane has radiosity B(x, y) = 1 + sin ax. The patch and the plane
are parallel to one another. We move the patch around parallel to the plane,
and consider its radiosity at various points.
(a) Show that if one translates the patch, its radiosity varies periodically with
its position in x.
(b) Fix the patch’s center at (0, 0); determine a closed form expression for the
radiosity of the patch at this point as a function of a. You’ll need a table
of integrals for this (if you don’t, you’re entitled to feel very pleased with
yourself).
2.9. If one looks across a large bay in the daytime, it is often hard to distinguish
the mountains on the opposite side; near sunset, they are clearly visible. This
phenomenon has to do with scattering of light by air—a large volume of air
is actually a source. Explain what is happening. We have modeled air as a
vacuum and asserted that no energy is lost along a straight line in a vacuum.
Use your explanation to give an estimate of the kind of scales over which that
model is acceptable.
Section 2.5 Notes 67
2.10. Read the book Colour and Light in Nature, by Lynch and Livingstone, pub-
lished by Cambridge University Press, 1995.
PROGRAMMING EXERCISES
2.11. An area source can be approximated as a grid of point sources. The weakness
of this approximation is that the penumbra contains quantization errors, which
can be quite oﬀensive to the eye.
(a) Explain.
(b) Render this eﬀect for a square source and a single occluder casting a
shadow onto an inﬁnite plane. For a ﬁxed geometry, you should ﬁnd that
as the number of point sources goes up, the quantization error goes down.
(c) This approximation has the unpleasant property that it is possible to pro-
duce arbitrarily large quantization errors with any ﬁnite grid by changing
the geometry. This is because there are conﬁgurations of source and oc-
cluder that produce large penumbrae. Use a square source and a single
occluder, casting a shadow onto an inﬁnite plane, to explain this eﬀect.
2.12. Make a world of black objects and another of white objects (paper, glue, and
spraypaint are useful here) and observe the eﬀects of interreﬂections. Can you
come up with a criterion that reliably tells, from an image, which is which?
(If you can, publish it; the problem looks easy, but isn’t.)
2.13. (This exercise requires some knowledge of numerical analysis.) Do the nu-
merical integrals required to reproduce Figure 2.18. These integrals aren’t
particularly easy: if one uses coordinates on the inﬁnite plane, the size of the
domain is a nuisance; if one converts to coordinates on the view hemisphere
of the patch, the frequency of the radiance becomes inﬁnite at the bound-
ary of the hemisphere. The best way to estimate these integrals is using a
Monte Carlo method on the hemisphere. You should use importance sampling
because the boundary contributes rather less to the integral than the top does.
2.14. Set up and solve the linear equations for an interreﬂection solution for the
interior of a cube with a small square source in the center of the ceiling.
2.15. Implement a photometric stereo system.
(a) How accurate are its measurements (i.e., how well do they compare with
known shape information)? Do interreﬂections aﬀect the accuracy?
(b) How repeatable are its measurements (i.e., if you obtain another set of
images, perhaps under diﬀerent illuminants, and recover shape from those,
how does the new shape compare with the old)?
(c) Compare the minimization approach to reconstruction with the integra-
tion approach; which is more accurate or more repeatable and why? Does
this diﬀerence appear in experiment?
(d) One possible way to improve the integration approach is to obtain depths
by integrating over many diﬀerent paths and then average these depths
(you need to be a little careful about constants here). Does this improve
the accuracy or repeatability of the method?
C H A P T E R 3
Color
The light receptors in cameras and in the eye respond more or less strongly to
diﬀerent wavelengths of light. Most cameras and most eyes have several diﬀerent
types of receptor, whose sensitivity to diﬀerent wavelengths varies. Comparing
the response of several types of sensor yields information about the distribution
of energy with wavelength for the incoming light; this is color information. Color
information can be used to identify specularities in images and to remove shadows.
The color of an object seen in an image depends on how the object was lit, but
there are algorithms that can correct for this eﬀect.
3.1 HUMAN COLOR PERCEPTION
The light coming out of sources or reﬂected from surfaces has more or less energy
at diﬀerent wavelengths, depending on the processes that produced the light. This
distribution of energy with wavelength is sometimes called a spectral energy density;
Figure 3.1 shows spectral energy densities for sunlight measured under a variety of
diﬀerent conditions. The visual system responds to light in a range of wavelengths
from approximately 400nm to approximately 700nm. Light containing energy at
just one wavelength looks deeply colored (these colors are known as spectral colors).
The colors seen at diﬀerent wavelengths have a set of conventional names, which
originate with Isaac Newton (the sequence from 700nm to 400nm goes Red Or-
ange Yellow Green Blue Indigo Violet, or Richard of York got blisters in Venice,
although indigo is now frowned upon as a name because people typically cannot
distinguish indigo from blue or violet). If the intensity is relatively uniform across
the wavelengths, the light will look white.
Diﬀerent kinds of color receptor in the human eye respond more or less
strongly to light at diﬀerent wavelengths, producing a signal that is interpreted
as color by the human vision system. The precise interpretation of a particular
light is a complex function of context; illumination, memory, object identity, and
emotion can all play a part. The simplest question is to understand which spectral
energy densities produce the same response from people under simple viewing con-
ditions (Section 3.1.1). This yields a simple, linear theory of color matching that
is accurate and extremely useful for describing colors. We sketch the mechanisms
underlying the transduction of color in Section 3.1.2.
3.1.1 Color Matching
The simplest case of color perception is obtained when only two colors are in view
on a black background. In a typical experiment, a subject sees a colored light—the
test light—in one half of a split ﬁeld (Figure 3.2). The subject can then adjust
a mixture of lights in the other half to get it to match. The adjustments involve
changing the intensity of some ﬁxed number of primaries in the mixture. In this
68
Section 3.1 Human Color Perception 69
400 450 500 550 600 650 700
0
0.5
1
1.5
2
2.5
cloudless sky
cloudless sky, just before sunset
cloudless sky, bright snow
slightly cloudy, sun behind a cloud
slightly cloudy, sun visible
cloudless sky, sunset
cloudy, gray sky
FIGURE 3.1: Daylight has diﬀerent amounts of power at diﬀerent wavelengths. These
plots show the spectral energy density of daylight measured at diﬀerent times of day
and under diﬀerent conditions. The ﬁgure plots relative power against wavelength for
wavelengths from 400 nm to 700 nm for a series of seven diﬀerent daylight measurements,
made by Jussi Parkkinen and Pertti Silfsten, of daylight illuminating a sample of barium
sulphate (which gives a high reﬂectance white surface). At the foot of the plot, we show
the names used for spectral colors of the relevant wavelengths. Plot from data obtainable
at http://www.it.lut.fi/ip/research/color/database/database.html.
form, a large number of lights may be required to obtain a match, but many diﬀerent
adjustments may yield a match.
Write T for the test light, an equals sign for a match, the weights—which are
non-negative—as w
i
, and the primaries P
i
. A match can then be written in an
algebraic form as
T = w
1
P
1
+w
2
P
2
+. . . ,
meaning that test light T matches the particular mixture of primaries given by
(w
1
, w
2
, . . .). The situation is simpliﬁed if subtractive matching is allowed. In
subtractive matching, the viewer can add some amount of some primaries to the
test light instead of to the match. This can be written in algebraic form by allowing
the weights in the expression above to be negative.
Under these conditions, most observers require only three primaries to match
a test light. This phenomenon is known as the principle of trichromacy. However,
Section 3.1 Human Color Perception 70
Test
light
T
Weighted
Mixture
of
primaries
ϩ
P
1
P
2
P
3
FIGURE 3.2: Human perception of color can be studied by asking observers to mix colored
lights to match a test light shown in a split ﬁeld. The drawing shows the outline of such
an experiment. The observer sees a test light T and can adjust the amount of each of
three primaries in a mixture displayed next to the test light. The observer is asked to
adjust the amounts so that the mixture looks the same as the test light. The mixture of
primaries can be written as w
1
P
1
+ w
2
P
2
+ w
3
P
3
; if the mixture matches the test light,
then we write T = w
1
P
1
+ w
2
P
2
+ w
3
P
3
. It is a remarkable fact that for most people
three primaries are suﬃcient to achieve a match for many colors, and three primaries are
suﬃcient for all colors if we allow subtractive matching (i.e., some amount of some of the
primaries is mixed with the test light to achieve a match). Some people require fewer
primaries. Furthermore, most people choose the same mixture weights to match a given
test light.
there are some caveats. First, subtractive matching must be allowed; second, the
primaries must be independent, meaning that no mixture of two of the primaries
may match a third. There is now clear evidence that trichromacy occurs because
there are three distinct types of color transducer in the eye (Nathans et al. 1986a,
Nathans et al. 1986b). Given the same primaries and test light, most observers
select the same mixture of primaries to match that test light, because most people
have the same types of color receptor.
Matching is (to an accurate approximation) linear. This yields Grassman’s
laws. First, if we mix two test lights, then mixing the matches will match the
result—that is, if
T
a
= w
a1
P
1
+w
a2
P
2
+w
a3
P
3
and
T
b
= w
b1
P
1
+w
b2
P
2
+w
b3
P
3
,
then
T
a
+T
b
= (w
a1
+w
b1
)P
1
+ (w
a2
+w
b2
)P
2
+ (w
a3
+w
b3
)P
3
.
Second, if two test lights can be matched with the same set of weights, then
they will match each other—that is, if
T
a
= w
1
P
1
+w
2
P
2
+w
3
P
3
and
T
b
= w
1
P
1
+w
2
P
2
+w
3
P
3
,
Section 3.1 Human Color Perception 71
then
T
a
= T
b
.
Finally, matching is linear: if
T
a
= w
1
P
1
+w
2
P
2
+w
3
P
3
,
then
kT
a
= (kw
1
)P
1
+ (kw
2
)P
2
+ (kw
3
)P
3
for non-negative k.
Given the same test light and set of primaries, most people use the same set of
weights to match the test light. This, trichromacy, and Grassman’s laws are about
as true as any law covering biological systems can be. The exceptions include the
following:
• people with too few kinds of color receptor as a result of genetic ill fortune
(who may be able to match everything with fewer primaries);
• people with neural problems (who may display all sorts of eﬀects, including
a complete absence of the sensation of color);
• some elderly people (whose choice of weights diﬀer from the norm because of
the development of macular pigment in the eye);
• very bright lights (whose hue and saturation look diﬀerent from less bright
versions of the same light);
• and very dark conditions (where the mechanism of color transduction is some-
what diﬀerent than in brighter conditions).
3.1.2 Color Receptors
Trichromacy occurs because there are (usually!) three distinct types of receptor in
the eye that mediate color perception. Each of these receptors turns incident light
into neural signals. The principle of univariance states that the activity of these
receptors is of one kind (i.e., they respond strongly or weakly, but do not signal the
wavelength of the light falling on them). Experimental evidence can be obtained
by carefully dissecting light-sensitive cells and measuring their responses to light at
diﬀerent wavelengths or by reasoning backward from color matches. Univariance is
a powerful idea because it gives us a good and simple model of human reaction to
colored light: two lights will match if they produce the same receptor responses,
whatever their spectral energy densities.
Because the system of matching is linear, the receptors must be linear. Write
p
k
for the response of the kth type of receptor, σ
k
(λ) for its sensitivity, E(λ) for
the light arriving at the receptor, and Λ for the range of visible wavelengths. We
can obtain the overall response of a receptor by adding up the response to each
separate wavelength in the incoming spectrum so that
p
k
=
_
Λ
σ
k
(λ)E(λ)dλ.
Section 3.1 Human Color Perception 72
350 400 450 500 550 600 650 700 750 800 850
Ϫ8
Ϫ7
Ϫ6
Ϫ5
Ϫ4
Ϫ3
Ϫ2
Ϫ1
0
1
S
M
L
Wavelength in nm
FIGURE 3.3: There are three types of color receptor in the human eye, usually called cones.
These receptors respond to all photons in the same way, but in diﬀerent amounts. The
ﬁgure shows the log of the relative spectral sensitivities of the three kinds of color receptor
in the human eye, plotted against wavelength. On the wavelength axis, we have shown the
color name usually associated with lights which contain energy only at that wavelength.
The ﬁrst two receptors—properly named the long- and medium-wavelength receptors—
have peak sensitivities at quite similar wavelengths. The third receptor (short-wavelength
receptor) has a diﬀerent peak sensitivity. The response of a receptor to incoming light can
be obtained by summing the product of the sensitivity and the spectral energy density of
the light over all wavelengths. Notice that each receptor responds to quite a broad range
of wavelengths. This means that human observers must perceive color by comparing the
response of the receptors to one another, and that there must be many spectral energy
densities that cannot be distinguished by humans. Figures plotted from data disseminated
by the Color and Vision Research Laboratories database, compiled by Andrew Stockman
and Lindsey Sharpe, and available at http://www.cvrl.org/.
Anatomical investigation of the retina shows two types of cell that are sensitive
to light, diﬀerentiated by their shape. The light-sensitive region of a cone has a
roughly conical shape, whereas that in a rod is roughly cylindrical. Cones largely
dominate color vision and completely dominate the fovea. Cones are somewhat less
sensitive to light than rods are, meaning that in low light, color vision is poor and
it is impossible to read (one doesn’t have suﬃcient spatial precision, because the
Section 3.2 The Physics of Color 73
fovea isn’t working).
The sensitivities of the three diﬀerent kinds of receptor to diﬀerent wave-
lengths can be obtained by comparing color matching data for normal observers
with color matching data for observers lacking one type of cone. Sensitivities ob-
tained in this fashion are shown in Figure 3.3. The three types of cone are properly
called S cones, M cones, and L cones (for their peak sensitivity being to short-,
medium-, and long-wavelength light, respectively). They are occasionally called
blue, green, and red cones; however, this is bad practice, because the sensation of
red is deﬁnitely not caused by the stimulation of red cones, and so on.
3.2 THE PHYSICS OF COLOR
Several diﬀerent mechanisms result in colored light. First, light sources can produce
diﬀerent amounts of light at diﬀerent wavelengths. This is what makes incandescent
lights look orange or yellow, and ﬂuorescent lights look bluish. Second, for most
diﬀuse surfaces, albedo depends on wavelength, so that some wavelengths may be
largely absorbed and others largely reﬂected. This means that most surfaces will
look colored when lit by a white light. The light reﬂected from a colored surface
is aﬀected by both the color of the light falling on the surface, and by the surface,
and so is profoundly ambiguous. For example, a white surface lit by red light will
reﬂect red light, and a red surface lit by white light will also reﬂect red light.
3.2.1 The Color of Light Sources
The most important natural light source is the sun. The sun is usually modeled as
a distant, bright point. Light from the sun is scattered by the air. In particular,
light can leave the sun, be scattered by the air, strike a surface, and be reﬂected
into the camera or the eye. This means the sky is an important natural light
source. A crude geometrical model of the sky has it as a source consisting of a
hemisphere with constant exitance. The assumption that exitance is constant is
poor, however, because the sky is substantially brighter at the horizon than at the
zenith. A natural model of the sky is to assume that air emits a constant amount
of light per unit volume; this means that the sky is brighter on the horizon than at
the zenith because a viewing ray along the horizon passes through more sky.
A patch of surface outdoors during the day is illuminated both by light that
comes directly from the sun—usually called daylight—and by light from the sun that
has been scattered by the air (sometimes called skylight or airlight; the presence of
clouds or snow can add other, important, phenomena). The color of daylight varies
with time of day (Figure 3.1) and time of year.
For clear air, the intensity of radiation scattered by a unit volume depends on
the fourth power of the frequency; this means that light of a long wavelength can
travel much farther before being scattered than light of a short wavelength (this is
known as Rayleigh scattering). This means that, when the sun is high in the sky,
blue light is scattered out of the ray from the sun to the earth—meaning that the
sun looks yellow—and can scatter from the sky into the eye—meaning that the sky
looks blue. There are standard models of the spectral energy density of the sky at
diﬀerent times of day and latitude, too. Surprising eﬀects occur when there are ﬁne
particles of dust in the sky (the larger particles cause much more complex scattering
Section 3.2 The Physics of Color 74
400
0
0.2
0.4
0.6
0.8
1
1.2
450 500 550
Wavelength in nm
R
e
l
a
t
i
v
e
S
p
e
c
t
r
a
l
P
o
w
e
r
600 650 700
Illuminant D65 Illuminant A
FIGURE 3.4: There is a variety of illuminant models; the graph shows the relative spectral
power distribution of two standard CIE models, illuminant A—which models the light from
a 100W Tungsten ﬁlament light bulb, with color temperature 2800K—and illuminant D-
65—which models daylight. Figure plotted from data available at http://www.cvrl.org/.
eﬀects, usually modeled rather roughly by the Mie scattering model, described in
Lynch and Livingston (2001) or in Minnaert (1993)) One author remembers vivid
sunsets in Johannesburg caused by dust in the air from mine dumps, and there are
records of blue and even green moons caused by volcanic dust in the air.
Artiﬁcial Illumination
Typical artiﬁcial light sources are commonly of a small number of types:
• An incandescent light contains a metal ﬁlament that is heated to a high tem-
perature. The spectrum roughly follows the black-body law (Section 3.2.1),
but the melting temperature of the element limits the color temperature of
the light source, so the light has a reddish tinge.
• A ﬂuorescent light works by generating high-speed electrons that strike gas
within the bulb. The gas releases ultraviolet radiation, which causes phos-
phors coating the inside of the bulb to ﬂuoresce. Typically the coating consists
of three or four phosphors, which ﬂuoresce in quite narrow ranges of wave-
lengths. Most ﬂuorescent bulbs generate light with a bluish tinge, but some
bulbs mimic natural daylight (Figure 3.5).
• In some bulbs, an arc is struck in an atmosphere consisting of gaseous met-
als and inert gases. Light is produced by electrons in metal atoms dropping
Section 3.2 The Physics of Color 75
from an excited state to a lower energy state. Typical of such lamps is strong
radiation at a small number of wavelengths, which correspond to particular
state transitions. The most common cases are sodium arc lamps and mercury
arc lamps. Sodium arc lamps produce a yellow-orange light extremely eﬃ-
ciently and are quite commonly used for freeway lighting. Mercury arc lamps
produce a blue-white light and are often used for security lighting.
Figure 3.5 shows a sample of spectra from diﬀerent light bulbs.
Metal halide
Standard flourescent
Moon white flourescent
Daylight flourescent
350 400 450 500 550 600 650 700 750
Wavelength in nm
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
R
e
l
a
t
i
v
e
e
n
e
r
g
y
FIGURE 3.5: The relative spectral power distribution of four diﬀerent lamps from the Mit-
subishi Electric Corporation. Note the bright, narrow bands that come from the ﬂourescing
phosphors in the ﬂuorescent lamp. The ﬁgure was plotted from data made available by the
Coloring Info Pages at http://www.colorpro.com/info/data/lamps.html; the data was
measured by Hiroaki Sugiura.
Black Body Radiators
One useful abstraction is the black body, a body that reﬂects no light. A heated
black body emits electromagnetic radiation. It is a remarkable fact that the spectral
power distribution of this radiation depends only on the temperature of the body.
If we write T for the temperature of the body in Kelvins, h for Planck’s constant,
k for Boltzmann’s constant, c for the speed of light, and λ for the wavelength, we
have
E(λ) ∝
1
λ
5
1
(exp(hc/kλT) −1)
.
This means that there is one parameter family of light colors corresponding to
black body radiators—the parameter being the temperature—and so we can talk
about the color temperature of a light source. This is the temperature of the black
Section 3.2 The Physics of Color 76
body that looks most similar. At relatively low temperatures, black bodies are red,
passing through orange to a pale yellow-white to white as the temperature increases
(Figure 3.12 shows this locus). When hc ¸ kλT, we have 1/(exp(hc/kλT) − 1) ≈
exp(−hc/kλT), so
E(λ; T) = C
exp(−hc/kλT)
λ
5
where C is the constant of proportionality; this model is somewhat easier to use
than the exact model (Section 3.5.2).
3.2.2 The Color of Surfaces
The color of surfaces is a result of a large variety of mechanisms, including dif-
ferential absorbtion at diﬀerent wavelengths, refraction, diﬀraction, and bulk scat-
tering (for more details, see, for example Lamb and Bourriau (1995), Lynch and
Livingston (2001), Minnaert (1993), or Williamson and Cummins (1983)). If we ig-
nore the physical eﬀects that give rise to the color, we can model surfaces as having
a diﬀuse and a specular component. Each component has a wavelength-dependent
albedo. The wavelength-dependent diﬀuse albedo is sometimes referred to as the
spectral reﬂectance (sometimes abbreviated to reﬂectance or, less commonly, spec-
tral albedo). Figures 3.6 and 3.7 show examples of spectral reﬂectances for a number
of diﬀerent natural objects.
400 450 500 550 600 650 700
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Wavelength in nm
R
e
f
l
e
c
t
a
n
c
e orange flower
yellow
flower
violet
flower
blue flower
white flower
white
petal
orange
berry
FIGURE 3.6: Spectral albedoes for a variety of natural surfaces measured by Esa
Koivisto, Department of Physics, University of Kuopio, Finland, plotted against
wavelength in nanometers. These ﬁgures were plotted from data available at
http://www.it.lut.fi/ip/research/color/database/database.html.
There are two color regimes for specular reﬂection. If the surface is dielectric
Section 3.3 Representing Color 77
(i.e., does not conduct electricity), specularly reﬂected light tends to take the color
of the light source. If the surface is a conductor, the specular albedo may depend
quite strongly on wavelength, so that white light may result in colored specularities.
red leaf
brown dry leaf
brown dry leaf
black dry leaf
yellow leaf
orange leaf
400 450 500 550 600 650 700
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Wavelength in nm
R
e
f
l
e
c
t
a
n
c
e
reddish brown leaf
FIGURE 3.7: Spectral albedoes for a variety of natural surfaces measured by Esa
Koivisto, Department of Physics, University of Kuopio, Finland, plotted against
wavelength in nanometers. These ﬁgures were plotted from data available at
http://www.it.lut.fi/ip/research/color/database/database.html.
3.3 REPRESENTING COLOR
Describing colors accurately is a matter of great commercial importance. Many
products are closely associated with speciﬁc colors—for example, the golden arches,
the color of various popular computers, and the color of photographic ﬁlm boxes—
and manufacturers are willing to go to a great deal of trouble to ensure that diﬀerent
batches have the same color. This requires a standard system for talking about
color. Simple names are insuﬃcient because relatively few people know many color
names, and most people are willing to associate a large variety of colors with a
given name.
3.3.1 Linear Color Spaces
There is a natural mechanism for representing color: agree on a standard set of
primaries, and then describe any colored light by the three values of weights that
people would use to match the light using those primaries. In principle, this is easy
to use. To describe a color, we set up and perform the matching experiment and
transmit the match weights. Of course, this approach extends to give a representa-
Section 3.3 Representing Color 78
tion for surface colors as well if we use a standard light for illuminating the surface
(and if the surfaces are equally clean, etc.).
Performing a matching experiment each time we wish to describe a color can
be practical. For example, this is the technique used by paint stores; you take in
a ﬂake of paint, and they mix paint, adjusting the mixture until a color match is
obtained. Paint stores do this because complicated scattering eﬀects within paints
mean that predicting the color of a mixture can be quite diﬃcult. However, Grass-
man’s laws mean that mixtures of colored lights—at least those seen in a simple
display—mix linearly, which means that a much simpler procedure is available.
Color Matching Functions
When colors mix linearly, we can construct a simple algorithm to determine
which weights would be used to match a source of some known spectral energy
density given a ﬁxed set of primaries. The spectral energy density of the source
can be thought of as a weighted sum of single wavelength sources. Because color
matching is linear, the combination of primaries that matches a weighted sum of
single wavelength sources is obtained by matching the primaries to each of the
single wavelength sources and then adding up these match weights.
For any set of primaries, P
1
, P
2
, and P
3
, we can obtain a set of color matching
functions by experiment. We tune the weight of each primary to match a unit energy
source at every wavelength, and record a table of these weights against wavelengths.
These tables are the color matching functions, which we write as f
1
(λ), f
2
(λ), and
f
3
(λ). Now for some wavelength λ
0
, we have
U(λ
0
) = f
1
(λ
0
)P
1
+f
2
(λ
0
)P
2
+f
3
(λ
0
)P
3
(i.e., f
1
, f
2
, and f
3
give the weights required to match a unit energy source at that
wavelength).
We wish to choose the weights to match a source S(λ). This source is a sum
of a vast number of single wavelength sources, each with a diﬀerent intensity. We
now match the primaries to each of the single wavelength sources and then add up
these match weights, obtaining
S(λ) = w
1
P
1
+w
2
P
2
+w
3
P
3
=
__
Λ
f
1
(λ)S(λ)dλ
_
P
1
+
__
Λ
f
2
(λ)S(λ)dλ
_
P
2
+
__
Λ
f
3
(λ)S(λ)dλ
_
P
3
.
General Issues for Linear Color Spaces
Linear color naming systems can be obtained by specifying primaries, which
imply color matching functions, or by specifying color matching functions, which
imply primaries. It is an inconvenient fact of life that, if the primaries are real lights,
at least one of the color matching functions is negative for some wavelengths. This
is not a violation of natural law; it just implies that subtractive matching is required
to match some lights, whatever set of primaries is used. It is a nuisance, though.
One way to avoid this problem is to specify color matching functions that are
everywhere positive (which guarantees that the primaries are imaginary because for
some wavelengths their spectral energy density is negative). Although this looks like
Section 3.3 Representing Color 79
x
y
z
350 400 450 500 550 600 650 700 750 800 850
2.5
2
1.5
1
0.5
0
Wavelength in nm
FIGURE 3.8: Color matching functions for the CIE X, Y, and Z primaries; the color
matching functions are everywhere positive, so no subtractive matching is required, but
the primaries are not real. Figures plotted from data disseminated by the Color and Vision
Research Laboratories database, compiled by Andrew Stockman and Lindsey Sharpe, and
available at http://www.cvrl.org/.
a problem—how would one create a real color with imaginary primaries?—it isn’t,
because color naming systems are hardly ever used that way. Usually, we would
simply compare weights to tell whether colors are similar, and for that purpose it is
enough to know the color matching functions. A variety of diﬀerent systems have
been standardized by the CIE (the commission international d’´eclairage, which
exists to create standards for such things).
Important Linear Color Spaces
The CIE XYZ color space is one quite popular standard. The color matching
functions were chosen to be everywhere positive (Figure 3.8), so that the coordinates
of any real light are always positive. It is not possible to obtain CIE X, Y, or Z
primaries because for some wavelengths the value of their pectral energy density
is negative. However, given color matching functions alone, one can specify the
XYZ coordinates of a color and hence describe it. Linear color spaces allow a
number of useful graphical constructions that are more diﬃcult to draw in three
dimensions than in two, so it is common to intersect the XYZ space with the plane
X + Y + Z = 1 (as shown in Figure 3.10) and draw the resulting ﬁgure using
coordinates
(x, y) =
_
X
X +Y +Z
,
Y
X + Y +Z
_
.
Section 3.3 Representing Color 80
350 400 450 500 550 600 650 700 750 800 850
3.5
3
2.5
2
1.5
1
0.5
0
Ϫ0.5
r
g
b
wavelength in nm
FIGURE 3.9: Color matching functions for the primaries for the RGB system. The negative
values mean that subtractive matching is required to match lights at that wavelength
with the RGB primaries. Figures plotted from data disseminated by the Color and Vision
Research Laboratories database, compiled by Andrew Stockman and Lindsey Sharpe, and
available at http://www.cvrl.org/.
This space, which is often referred to as the CIE xy color space is shown in Fig-
ure 3.12. CIE xy is widely used in vision and graphics textbooks and in some
applications, but is usually regarded by professional colorimetrists as out of date.
The RGB color space is a linear color space that formally uses single wave-
length primaries (645.16 nm for R, 526.32 nm for G, and 444.44 nm for B; see
Figure 3.9). Informally, RGB uses whatever phosphors a monitor has as primaries.
Available colors are usually represented as a unit cube—usually called the RGB
cube—whose edges represent the R, G, and B weights. The cube is drawn in Fig-
ure 3.13.
The opponent color space is a linear color space derived from RGB. There is
evidence that there are three kinds of color system in primates (e.g., see Mollon
(1982); Hurvich and Jameson (1957)). The oldest responds to intensity (i.e., light-
dark comparisons). A more recent, but still old, color system compares blue with
yellow. The most recent color system compares red with green. In some applica-
tions, it is useful to use a comparable representation. This can be obtained from
RGB coordinates using I = (R + G + B)/3 for intensity, (B − (R + G)/2)/I for
the blue-yellow comparison (sometimes called B-Y), and (R − G)/I for the red-
green comparison (sometimes called R-G). Notice that B-Y (resp. R-G) is positive
for strongly blue (resp. red) colors and negative for strongly yellow (resp. green)
Section 3.3 Representing Color 81
X
Z
Y
X ϩ Y ϩ Z ϭ 1
X Y
Z
y ϭ Y/(X ϩ Y ϩ Z) x ϭ X/(X ϩ Y ϩ Z)
FIGURE 3.10: The volume of all visible colors in the CIE XYZ coordinate space is a cone
whose vertex is at the origin. Usually it is easier to suppress the brightness of a color,
which we can do because, to a good approximation, perception of color is linear, and we
do this by intersecting the cone with the plane X + Y + Z = 1 to get the CIE xy space
shown in Figure 3.12.
colors, and is intensity independent.
There are two useful constructions that work in linear color spaces, but are
most commonly applied in CIE xy. First, because the color spaces are linear, and
color matching is linear, all colors that can be obtained by mixing two primaries
A and B lie on the line segment joining them plotted on the color space. Second,
all colors that can be obtained by mixing three primaries A, B, and C lie in the
triangle formed by the three primaries plotted on the color space. Typically, we
use this construction to determine the set of colors (or gamut) that a set of monitor
phosphors can display.
Subtractive Mixing and Inks
Intuition from one’s ﬁnger-painting days suggests that the primary colors
should be red, yellow, and blue, and that yellow and blue mix to make green. The
reason this intuition doesn’t apply to monitors is that paints involve pigments—
which mix subtractively—rather than lights. Pigments can behave in quite complex
ways, but the simplest model is that pigments remove color from incident light,
which is reﬂected from paper. Thus, red ink is really a dye that absorbs green
and blue light—incident red light passes through this dye and is reﬂected from the
paper. In this case, mixing is subtractive.
Color spaces for this kind of mixing can be quite complicated. In the simplest
case, mixing is linear (or reasonably close to linear), and the CMY space applies.
In this space, there are three primaries: cyan (a blue-green color), magenta (a
purplish color), and yellow. These primaries should be thought of as subtracting
a light primary from white light; cyan is W − R (white − red); magenta is W −G
(white − green), and yellow is W − B (white − blue). Now the appearance of
mixtures can be evaluated by reference to the RGB color space. For example, cyan
Section 3.3 Representing Color 82
x axis
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
y
a
x
i
s
Red
Orange
Yellow
Green
Blue
Indigo
Violet
White
Saturation increases as
one moves out radially
from white
Hue changes as one moves
around the spectral locus
FIGURE 3.11: The ﬁgure shows a constant brightness section of the standard 1931 standard
CIE xy color space, with color names marked on the diagram. Generally, colors that lie
farther away from the neutral point are more saturated—the diﬀerence between deep red
and pale pink—and hue—the diﬀerence between green and red—as one moves around the
neutral point.
and magenta mixed give
(W −R) + (W − G) = R +G+B −R −G = B,
that is, blue. Notice that W + W = W because we assume that ink cannot cause
paper to reﬂect more light than it does when uninked. Practical printing devices
use at least four inks (cyan, magenta, yellow, and black) because mixing color inks
leads to a poor black, it is diﬃcult to ensure good enough registration between the
three color inks to avoid colored haloes around text, and color inks tend to be more
expensive than black inks. Getting really good results from a color printing process
is still diﬃcult: diﬀerent inks have signiﬁcantly diﬀerent spectral properties, diﬀer-
ent papers also have diﬀerent spectral properties, and inks can mix non-linearly.
One reason that ﬁngerpainting is hard is that the color resulting from mixing
paints can be quite hard to predict. This is because the outcome depends very
strongly on details such as the speciﬁc pigment in the paint, the size of pigment
particles, the medium in which the pigment is suspended, the care put into stirring
Section 3.3 Representing Color 83
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
520
600
780
380
Sunlight at
sunrise 1800K
Incandescent
lighting
Sunlight at
noon 4870K
Daylight
7000K and
Illuminant C
Northwest
sky light 25000K
Equal energy
spectrum
x
y
FIGURE 3.12: The ﬁgure shows a constant brightness section of the standard 1931 standard
CIE xy color space. This space has two coordinate axes. The curved boundary of the
ﬁgure is often known as the spectral locus; it represents the colors experienced when lights
of a single wavelength are viewed. The ﬁgure shows a locus of colors due to black-body
radiators at diﬀerent temperatures and a locus of diﬀerent sky colors. Near the center of
the diagram is the neutral point, the color whose weights are equal for all three primaries.
CIE selected the primaries so that this light appears achromatic. Generally, colors that
lie farther away from the neutral point are more saturated—the diﬀerence between deep
red and pale pink—and hue—the diﬀerence between green and red—as one moves around
the neutral point.
the mixture, and similar parameters; usually, we do not have enough detailed infor-
mation to use a full physical model of these eﬀects. A useful study of this diﬃcult
topic is (Berns 2000).
3.3.2 Non-linear Color Spaces
The coordinates of a color in a linear space may not necessarily encode properties
that are common in language or are important in applications. Useful color terms
include: hue, the property of a color that varies in passing from red to green;
saturation, the property of a color that varies in passing from red to pink; and
brightness (sometimes called lightness or value, the property that varies in passing
from black to white. For example, if we are interested in checking whether a color
lies in a particular range of reds, we might wish to encode the hue of the color
Section 3.3 Representing Color 84
directly.
Another diﬃculty with linear color spaces is that the individual coordinates do
not capture human intuitions about the topology of colors; it is a common intuition
that hues form a circle, in the sense that hue changes from red through orange to
yellow, and then green, and from there to cyan, blue, purple, and then red again.
Another way to think of this is to picture local hue relations: red is next to purple
and orange; orange is next to red and yellow; yellow is next to orange and green;
green is next to yellow and cyan; cyan is next to green and blue; blue is next to
cyan and purple; and purple is next to blue and red. Each of these local relations
works, and globally they can be modeled by laying hues out in a circle. This means
that no individual coordinate of a linear color space can model hue, because that
coordinate has a maximum value that is far away from the minimum value.
Hue, Saturation, and Value
A standard method for dealing with this problem is to construct a color space
that reﬂects these relations by applying a non-linear transformation to the RGB
space. There are many such spaces. One, called HSV space (for hue, saturation,
and value), is obtained by looking down the center axis of the RGB cube. Because
RGB is a linear space, brightness—called value in HSV—varies with scale out from
the origin. We can ﬂatten the RGB cube to get a 2D space of constant value and for
neatness deform it to be a hexagon. This gets the structure shown in Figure 3.13,
where hue is given by an angle that changes as one goes round the neutral point
and saturation changes as one moves away from the neutral point.
There are a variety of other possible changes of coordinate from linear color
space to linear color space, or from linear to non-linear color spaces (Fairchild (1998)
is a good reference). There is no obvious advantage to using one set of coordinates
over another (particularly if the diﬀerence between coordinate systems is just a
one–one transformation) unless one is concerned with coding, bit rates, and the
like, or with perceptual uniformity.
Uniform Color Spaces
Usually one cannot reproduce colors exactly. This means it is important to
know whether a color diﬀerence would be noticeable to a human viewer. It is
generally useful to compare the signiﬁcance of small color diﬀerences, but it is
usually dangerous to try and compare large color diﬀerences; consider trying to
answer the question, “Is the blue patch more diﬀerent from the yellow patch than
the red patch is from the green patch?”.
One can determine just noticeable diﬀerences by modifying a color shown to
observers until they can only just tell it has changed in a comparison with the
original color. When these diﬀerences are plotted on a color space, they form the
boundary of a region of colors that are indistinguishable from the original colors.
Usually ellipses are ﬁtted to the just noticeable diﬀerences. It turns out that in CIE
xy space these ellipses depend quite strongly on where in the space the diﬀerence
occurs, as the MacAdam ellipses in Figure 3.14 illustrate.
This means that the size of a diﬀerence in (x, y) coordinates, given by ((Δx)
2
+
(Δy)
2
)
(1/2)
, is a poor indicator of the signiﬁcance of a diﬀerence in color (if it were a
good indicator, the ellipses representing indistinguishable colors would be circles).
Section 3.3 Representing Color 85
R
G
B
Green
Yellow
Red
Magenta
Blue
Cyan
Hue
(angle)
Saturation
Value
Green (120 )
Blue (240 )
Red (0 )
FIGURE 3.13: On the left, we see the RGB cube; this is the space of all colors that can
be obtained by combining three primaries (R, G, and B—usually deﬁned by the color
response of a monitor) with weights between zero and one. It is common to view this cube
along its neutral axis—the axis from the origin to the point (1, 1, 1)—to see a hexagon.
This hexagon codes hue (the property that changes as a color is changed from green to
red) as an angle, which is intuitively satisfying. On the right, we see a cone obtained
from this cross-section, where the distance along a generator of the cone gives the value
(or brightness) of the color, the angle around the cone gives the hue, and the distance out
gives the saturation of the color.
A uniform color space is one in which the distance in coordinate space is a fair
guide to the signiﬁcance of the diﬀerence between two colors—in such a space,
if the distance in coordinate space were below some threshold, a human observer
would not be able to tell the colors apart.
A more uniform space can be obtained from CIE XYZ by using a projective
transformation to skew the ellipses; this yields the CIE u/v/ space CIE u’v’ space,
illustrated in Figure 3.15. The coordinates are:
(u

, v

) =
_
4X
X + 15Y + 3Z
,
9Y
X + 15Y + 3Z
_
.
Generally, the distance between coordinates in u/, v/ space is a fair indicator
of the signiﬁcance of the diﬀerence between two colors. Of course, this omits diﬀer-
ences in brightness. CIE LAB is now almost universally the most popular uniform
color space. Coordinates of a color in LAB are obtained as a non-linear mapping
Section 3.4 A Model of Image Color 86
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
y
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
y
FIGURE 3.14: This ﬁgure shows variations in color matches on a CIE xy space. At the
center of the ellipse is the color of a test light; the size of the ellipse represents the scatter
of lights that the human observers tested would match to the test color; the boundary
shows where the just noticeable diﬀerence is. The ellipses in the ﬁgure on the left have
been magniﬁed 10x for clarity; on the right they are plotted to scale, with color names
on the CIE diagram as a reference. The ellipses are known as MacAdam ellipses after
their inventor. Notice that the ellipses at the top are larger than those at the bottom of
the ﬁgure, and that they rotate as they move up. This means that the magnitude of the
diﬀerence in x, y coordinates is a poor guide to the diﬀerence in color. Ellipses are plotted
using data from MacAdam (1942).
of the XYZ coordinates:
L
∗
= 116
_
Y
Y
n
_1
3
−16
a
∗
= 500
_
_
X
X
n
_1
3
−
_
Y
Y
n
_1
3
_
b
∗
= 200
_
_
Y
Y
n
_1
3
−
_
Z
Z
n
_1
3
_
Here X
n
, Y
n
, and Z
n
are the X, Y , and Z coordinates of a reference white patch.
The reason to care about the LAB space is that it is substantially uniform. In
some problems, it is important to understand how diﬀerent two colors will look to
a human observer, and diﬀerences in LAB coordinates give a good guide.
3.4 A MODEL OF IMAGE COLOR
Assume that an image pixel is the image of some surface patch. Many phenomena
aﬀect the color of this pixel. The main eﬀects are: the camera response to illu-
mination (which might not be linear); the choice of camera receptors; the amount
of light that arrives at the surface; the color of light arriving at the surface; the
dependence of the diﬀuse albedo on wavelength; and specular components. A quite
Section 3.4 A Model of Image Color 87
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
u'
v'
500 nm
600 nm
700 nm
400 nm
FIGURE 3.15: This ﬁgure shows the CIE 1976 u/, v/ space, which is obtained by a projective
transformation of CIE x, y space. The intention is to make the MacAdam ellipses (from
Figure 3.14) uniformly circles. This would yield a uniform color space. A variety of non-
linear transforms can be used to make the space more uniform (see Fairchild (1998) for
details).
simple model can be used to separate some of these eﬀects.
Generally, it is easier to model linear cameras. CCDs are intrinsically linear
devices. However, most users are used to ﬁlm, which tends to compress the incoming
dynamic range (brightness diﬀerences at the top end of the range are reduced, as are
those at the bottom end of the range). The output of a linear device tends to look
too harsh (the darks are too dark and the lights are too light), so that manufacturers
apply various forms of compression to the output. We assume that the camera
response has been calibrated, perhaps using the methods of Section 2.2.1, so that
it is linear.
Assume that the surfaces that we are dealing with can be described by the
diﬀuse+specular model. Write x for a point, λ for wavelength, E(x, λ) for the
spectral energy density of the light leaving a surface, ρ(x, λ) for the albedo of a
surface as a function of wavelength and position, S
d
(x, λ) for the spectral energy
density of the light source (which may vary with position; for example, the intensity
might change), and S
i
(x, λ) for the spectral energy density of interreﬂected light.
Then we have that:
E(x, λ) = [diﬀuse term] + (specular term)
= [(direct term) + (interreﬂected term)] + (specular term)
= (ρ(x, λ)(geometric term))[(S
d
(x, λ) +S
i
(x, λ))] + (specular term).
The geometric terms represent how intensity is aﬀected by surface normal. Notice
that the diﬀuse term is aﬀected both by the color of the surface and by the color
Section 3.4 A Model of Image Color 88
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
y
x
Uniform reflectance
Metal halide
Standard fluorescent
Moon white fluorescent
Daylight fluorescent
Uniform SPD
FIGURE 3.16: Light sources can have quite widely varying colors. This ﬁgure shows the
color of the four light sources of Figure 3.5, compared with the color of a uniform spectral
power distribution, plotted in CIE x, y coordinates.
of the light (examples in Figures 3.16 and 3.17).
Because the camera is linear, the pixel value at x is a sum of terms corre-
sponding to each of the terms in E(x, λ). Write d(x) for the color taken by a ﬂat
patch facing the light source at x with the same albedo as the actual patch there,
g(x) for a geometric term (explained below), i(x) for the contribution of the inter-
reﬂected term, s(x) for the unit intensity color of the specular term, and g
s
(x) for
a geometric term (explained below). Then we have:
C(x) = [(direct term) + (interreﬂected term)] + (specular term)
= g
d
(x)d(x) +i(x) +g
s
(x)s(x).
Generally, to work with this model, we ignore i(x); we identify and remove specu-
larities, using the methods of Section 3.5.1, and so assume that C(x) = g
d
(x)d(x).
3.4.1 The Diﬀuse Term
There are two diﬀuse components. One, i(x), is due to interreﬂections. Interreﬂec-
tions can be a signiﬁcant source of colored light. If a large colored surface reﬂects
light onto another surface, that surface’s color can change quite substantially. This
is an eﬀect that people ﬁnd hard to see, but which is usually fairly easy to spot in
photographs. There are no successful models for removing these color shifts, most
likely because they can be very hard to predict. This is because many diﬀerent
surface reﬂectances can have the same color, so that two surfaces with the same
color (but diﬀerent reﬂectances) can have quite diﬀerently colored interreﬂections.
The interreﬂection term is often small, and usually is simply ignored.
Ignoring the interreﬂected component, the diﬀuse term is
g
d
(x)d(x).
Section 3.4 A Model of Image Color 89
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
y
x
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
y
x
Orange flower
Metal halide
Standard fluorescent
Moon white fluorescent
Daylight fluorescent
Uniform SPD
Violet flower
FIGURE 3.17: The color of a light source aﬀects the color of surfaces lit by the source. The
diﬀerent colors obtained by lighting the violet ﬂower of Figure 3.6 (left) and the orange
ﬂower of Figure 3.6 (right) with the four light sources of Figure 3.5.
Here d(x) is the image color of an equivalent ﬂat surface facing the light source and
viewed under the same light. The geometric term, g
d
(x), varies relatively slowly
over space and accounts for the change in brightness due to the orientation of the
surface.
We can model the dependence of d(x) on the light and on the surface by
assuming we are viewing ﬂat, diﬀuse surfaces, illuminated from inﬁnitely far behind
the camera. In this case, there will be no eﬀects due to specularities or to surface
orientation. The color of light arriving at the camera will be determined by two
factors: ﬁrst, the wavelength-dependent albedo of the surface that the light is
leaving; and second, the wavelength-dependent intensity of the light falling on that
surface. If a patch of perfectly diﬀuse surface with diﬀuse albedo ρ(λ) is illuminated
by a light whose spectrum is S(λ), the spectrum of the reﬂected light is ρ(λ)S(λ).
Assume the camera has linear photoreceptors, and the k’th type of photoreceptor
has sensitivity σ
k
(λ). If a linear photoreceptor of the kth type sees this surface
patch, its response is:
p
k
=
_
Λ
σ
k
(λ)ρ(λ)S(λ)dλ,
where Λ is the range of all relevant wavelengths.
The main engineering parameter here is the photoreceptor sensitivities σ
k
(λ).
For some applications such as shadow removal (Section 3.5.2), it can be quite helpful
to have photoreceptor sensitivities that are “narrow-band” (i.e., the photoreceptors
respond to only one wavelength). Usually, the only practical methods to change the
photoreceptor sensitivities are to either put colored ﬁlters in front of the camera
or to use a diﬀerent camera. Using a diﬀerent camera doesn’t work particularly
Section 3.5 Inference from Color 90
well, because manufacturers try to have sensitivities that are reasonably compat-
ible with human receptor sensitivities. They do this so that cameras give about
the same responses to colored lights that people do; as a result, cameras tend to
have quite similar receptor sensitivities. There are three ways to proceed: install
narrow-band ﬁlters in front of the lens (diﬃcult to do and seldom justiﬁed); apply a
transformation to the receptor outputs that makes them behave more like narrow-
band receptors (often helpful, if the necessary data are available, Finlayson et al.
(1994b);Barnard et al. (2001a)); or assume that they are narrow-band receptors
and tolerate any errors that result (generally quite successful).
3.4.2 The Specular Term
The specular component will have a characteristic color, and its intensity will change
with position. We can model the specular component as
g
s
(x)s(x),
where s(x) is the unit intensity image color of the specular reﬂection at that pixel,
and g
s
(x) is a term that varies from pixel to pixel, and models the amount of energy
specularly reﬂected. We expect g
s
(x) to be zero at most points, and large at some
points.
The color s(x) of the specular component depends on the material. Generally,
metal surfaces have a specular component that is wavelength dependent and so
takes on a characteristic color that depends on the metal (gold is yellow, copper
is orange, platinum is white, and osmium is blue or purple). Surfaces that do
not conduct—dielectric surfaces— have a specular component that is independent
of wavelength (e.g., the specularities on a shiny plastic object are the color of the
light). Section 3.5.1 describes how these properties can be used to ﬁnd specularities,
and to ﬁnd image regions corresponding to metal or plastic objects.
3.5 INFERENCE FROM COLOR
Our color model supports a variety of inferences. It can be used to ﬁnd specular-
ities (Section 3.5.1); to remove shadows (Section 3.5.2); and to infer surface color
(Section 3.5.3).
3.5.1 Finding Specularities Using Color
Specular reﬂections can have strong eﬀects on an object’s appearance. Typically,
they appear as small, bright patches, called highlights or specularities. Highlights
have a substantial eﬀect on human perception of a surface properties; the addition
of small, highlight-like patches to a ﬁgure makes the object depicted look glossy
or shiny. Specularities are often bright enough to saturate the camera, so that the
color of a specularity can be hard to measure. However, because the appearance of
a specularity is quite strongly constrained, there are a number of eﬀective schemes
for marking them, and the results can be used as a shape cue.
The dynamic range of practically available albedoes is relatively small. Sur-
faces with very high or very low albedo are diﬃcult to make. Uniform illumination
is common too, and most cameras are reasonably close to linear within their operat-
ing range. This means that very bright patches cannot be due to diﬀuse reﬂection;
Section 3.5 Inference from Color 91
R
G
B
R
G
B
Diffuse
region
Boundary of
specularity
FIGURE 3.18: The linear clusters produced by specularities on plastic objects can be
found by reasoning about windows of image pixels. In a world of plastic objects on a
black background, a background window produces a region of pixels that are point-like in
color space—all pixels have the same color. A window that lies along the body produces a
line-like cluster of points in color space, because the intensity varies, but the color does not.
At the boundary of a specularity, windows produce plane-like clusters because points are
a weighted combination of two diﬀerent colors (the specular and the body color). Finally,
at the interior of a specular region, the windows can produce volume-like clusters, because
the camera saturates, and the extent of the window can include both the boundary-style
window and saturated points. Whether a region is line-like, plane-like, or volume-like can
be determined easily by looking at the eigenvalues of the covariance of the pixels.
they must be either sources (of one form or another—perhaps a stained glass win-
dow with the light behind it) or specularities. Furthermore, specularities tend to
be small. Thus, looking for small, bright patches can be an eﬀective way to ﬁnd
specularities (Brelstaﬀ and Blake 1988a).
An alternative is to use image color. From our model, the color of specularities
on dielectric objects is the color of the light source. Assume we can ignore the
interreﬂection term, either because we have an isolated object or because the term
doesn’t change much over the object we are viewing. Our model gives the image
color as a sum of a diﬀuse term and a specular term. Now consider a patch of
surface around a specularity. We expect that this patch is small, because we expect
specularities to be small (this will be true on curved surfaces; the approach we are
describing might not work for ﬂat surfaces). Because the patch is small, we expect
that d(x) does not change in the patch; we do not expect to be unlucky, and have
Section 3.5 Inference from Color 92
a specularity on an albedo boundary. We expect that s(x) does not change within
the patch, because the color of the specularity will be the color of the light source,
and this will not change within a small patch.
On a dielectric object, as we move from a patch with no specular reﬂection
to one with a specular component, the image color will change, because the size of
the specular component changes. We can write the image color as
g
d
(x)d +g
s
(x)s,
where s is the color of the source and d is the color of the diﬀuse reﬂected light,
g
d
(x) is the geometric term that depends on the orientation of the surface, and
g
s
(x) is a term that gives the extent of the specular reﬂection.
If the object is curved, then g
s
(x) is small over much of the surface and
large only around specularities; g
d
(x) varies more slowly with the orientation of
the surface. We now map the colors produced by this surface in receptor response
space and look at the structures that appear there.
The term g
d
(x)d produces a line that should extend to pass through the origin
because it represents the same vector of receptor responses multiplied by a constant
that varies over space. If there is a specularity, then we expect to see a second line
due to g
s
(x)s. This does not, in general, pass through the origin (because of the
diﬀuse term). This is a line, rather than a planar region, because g
s
(x) is large
over only a small range of surface normals. We expect that, because the surface
is curved, this corresponds to a small region of surface. The term g
d
(x) should be
approximately constant in this region. We expect a line, rather than an isolated
pixel value, because we expect surfaces to have (possibly narrow) specular lobes,
meaning that the specular coeﬃcient has a range of values. This second line might
collide with a face of the color cube and get clipped.
The resulting dog-leg pattern leads pretty much immediately to a specularity
marking algorithm: ﬁnd the pattern and then ﬁnd the specular line. All the pixels
on this line are specular pixels, and the specular and diﬀuse components can be
estimated easily. For the approach to work eﬀectively, we need to be conﬁdent that
only one object is represented in the collection of pixels. This is helped by using
local image windows, as illustrated by Figure 3.18. The observations underlying the
method hold even if the surface is not monochrome—a coﬀee mug with a picture on
it, for example—but ﬁnding the resulting structures in the color space now becomes
something of a nuisance and, to our knowledge, has not been demonstrated.
3.5.2 Shadow Removal Using Color
Lightness methods make the assumption that “fast” edges in images are due to
changes in albedo (Section 2.2.3). This assumption is usable, but fails badly at
shadows, particularly shadows in sunlight outdoors (Figure 3.20), where there can
be a large and fast change of image brightness. People usually are not fooled into
believing that a shadow is a patch of dark surface, so must have some method to
identify shadow edges. Home users often like editing and improving photographs,
and programs that could remove shadows from images would be valuable. A shadow
removal program would work something like a lightness method: ﬁnd all edges,
identify the shadow edges, remove those, and then integrate to get the picture
Section 3.5 Inference from Color 93
back.
There are some cues for ﬁnding shadow edges that seem natural, but don’t
work well. One might assume that shadow edges have very large dynamic range
(which albedo edges can’t have; see Section 2.1.1), but this is not always the case.
One might assume that, at a shadow edge, there was a change in brightness but not
in color. It turns out that this is not the case for outdoor shadows, because the lit
region is illuminated by yellowish sunlight, and the shadowed region is illuminated
by bluish light from the sky, or sometimes by interreﬂected light from buildings,
and so on. However, a really useful cue can be obtained by modelling the diﬀerent
light sources.
We assume that light sources are black bodies, so that their spectral energy
density is a function of temperature. We assume that surfaces are diﬀuse. We
use the simpliﬁed black-body model of Section 3.2.1, where, writing T for the
temperature of the body in Kelvins, h for Planck’s constant, k for Boltzmann’s
constant, c for the speed of light, and λ for the wavelength, we have
E(λ; T) = C
exp(−hc/kλT)
λ
5
(C is some constant of proportionality). Now assume that the color receptors each
respond only at one wavelength, which we write λ
k
for the k’th receptor, so that
σ
k
(λ) = δ(λ − λ
k
). If we view a surface with spectral albedo ρ(λ) illuminated by
one of these sources at temperature T, the response of the j’th receptor will be
r
j
=
_
σ
j
(λ)ρ(λ)K
exp(−hc/kλT)
λ
5
dλ = Kρ(λ
j
)
exp(−hc/kλ
j
T)
λ
5
j
.
We can form a color space that is very well behaved by taking c
1
= log(r
1
/r
3
),
c
2
= log(r
2
/r
3
), because
_
c
1
c
2
_
=
_
a
1
a
2
_
+
1
T
_
b
1
b
2
_
where a
1
= log ρ(λ
1
) −log ρ(λ
3
) +5 log λ
3
−5 log λ
1
and b
1
= (hc/k)(1/λ
3
−1/λ
1
)
(and a
2
, b
2
follow). Notice that, when one changes the color temperature of the
source, the (c
1
, c
2
) coordinates move along a straight line. The direction of the
line depends on the sensor, but not on the surface. Call this direction the color
temperature direction. The intercept of the line depends on the surface.
Now consider a world of colored surfaces, and map the image colors to this
space. There is a family of parallel lines in this space, whose direction is the color
temperature direction. Diﬀerent surfaces may map to diﬀerent lines. If we change
the color temperature of the illuminant, then each color in this space will move
along the color temperature direction, but colors will not move from line to line.
We now represent a surface color by its line. For example, we could construct a
line through the origin that is perpendicular to color temperature direction, then
represent a surface color by distance along this line (Figure 3.19). We can represent
each pixel in the image in this space, and in this representation the color image
becomes a gray-level image, where the gray level does not change inside shadows
(because a shadow region just has a diﬀerent color temperature to the non-shadowed
Section 3.5 Inference from Color 94
Color
temperature
direction
I
n
v
a
r
i
a
n
t
i
m
a
g
e
v
a
l
u
e
s
1
2
FIGURE 3.19: Changing the color temperature of the light under which a surface is viewed
moves the (c
1
, c
2
) coordinates of that surface along the color temperature direction (left;
the diﬀerent gray patches represent the same surface under diﬀerent lights). If we now
project the coordinates along the (c
1
, c
2
) direction onto some line, we obtain a value that
doesn’t change when the illuminant color temperature changes. This is the invariant value
for that pixel. Generally, we do not know enough about the imaging system to estimate
the color temperature direction. However, we expect to see many diﬀerent surfaces in
each scene; this suggests that the right choice of color temperature direction on the right
is 1 (where there are many diﬀerent types of surface) rather than 2 (where the range of
invariant values is small).
region). Finlayson (1996) calls this the invariant image. Any edge that appears in
the image but not in the invariant image is a shadow edge, so we can now apply
our original formula: ﬁnd all edges, identify the shadow edges, remove those, and
then integrate to get the picture back.
Of course, under practical circumstances, usually we do not know enough
about the sensors to evaluate the as and bs that deﬁne this family of lines, so we
cannot get the invariant image directly. However, we can infer a direction in (c
1
, c
2
)
space that is a good estimate by a form of entropy reasoning. We must choose a
color temperature direction. Assume the world is rich in diﬀerently colored surfaces.
Now consider two surfaces S
1
and S
2
. If c
1
(the (c
1
, c
2
) values for S
1
) and c
2
are
such that c
1
−c
2
is parallel to the color temperature direction, we can choose T
1
and
T
2
so that S
1
viewed under light with color temperature T
1
will look the same as
S
2
viewed under light with color temperature T
2
. We expect this to be uncommon,
because surfaces tend not to mimic one another in this way. This means we expect
that colors will tend to spread out when we project along a good estimate of the
color temperature direction. A reasonable measure of this spreading out is the
entropy of the histogram of projected colors. We can now estimate the invariant
image, without knowing anything about the sensor. We search directions in (c
1
, c
2
)
space, projecting all the image colors along that direction; our estimate of the color
temperature direction is the one where this projection yields the largest entropy.
From this we can compute the invariant image, and so apply our shadow removal
strategy above. In practice, the method works well, though great care is required
with the integration procedure to get the best results (Figure 3.20).
Section 3.5 Inference from Color 95
Invariant image Shadow removed image Image
FIGURE 3.20: The invariant of the text and of Figure 3.19 does not change value when a
surface is shadowed. Finlayson et al. use this to build a shadow removal system that works
by (a) taking image edges; (b) forming an invariant image; then (c) using that invariant
image to identify shadow edges; and ﬁnally (d) integrating only non-shadow edges to form
the result. The results are quite convincing. This ﬁgure was originally published as Figures
2 and 4 of “On the Removal of Shadows From Images,” G. Finlayson, S. Hordley, C. Lu
and M. Drew, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006 c _
IEEE, 2006.
3.5.3 Color Constancy: Surface Color from Image Color
In our model, the image color depends on both light color and on surface color. If
we light a green surface with white light, we get a green image; if we light a white
surface with a green light, we also get a green image. This makes it diﬃcult to
name surface colors from pictures. We would like to have an algorithm that can
take an image, discount the eﬀect of the light, and report the actual color of the
surface being viewed.
This process is called color constancy. Humans have some form of color con-
stancy algorithm. People are often unaware of this, and inexperienced photogra-
phers are sometimes surprised that a scene photographed indoors under ﬂuorescent
lights has a blue cast, whereas the same scene photographed outdoors may have a
warm orange cast. The simple linear models of Section 3.3 can predict the color
an observer will perceive when shown an isolated spot of light of a given power
spectral distribution. But if this spot is part of a larger, more complex scene, these
models can give wildly inaccurate predictions. This is because the human color
constancy algorithm uses various forms of scene information to decide what color
to report. Demonstrations by Land and McCann (1971), which are illustrated in
Figure 3.21, give convincing examples of this eﬀect. It is surprisingly diﬃcult to
predict what colors a human will see in a complex scene (Fairchild (1998); Helson
(1938a); (1938b); (1934); (1940)). This is one of the many diﬃculties that make it
hard to produce really good color reproduction systems.
Human color constancy is not perfectly accurate, and people can choose to
Section 3.5 Inference from Color 96
Photometer reading
(1, .3, .3)
Audience name
"Blue"
Coloured light
Photometer reading
(1, .3, .3)
Audience name
"Red"
White light
FIGURE 3.21: Land showed an audience a quilt of rectangles of ﬂat colored papers—since
known as a Mondrian for a purported resemblance to the work of that artist—illuminated
using three slide projectors, casting red, green and blue light respectively. He used a
photometer to measure the energy leaving a particular spot in three diﬀerent channels,
corresponding to the three classes of receptor in the eye. He recorded the measurement,
and asked the audience to name the patch. Assume the answer was “red” (on the left).
Land then adjusted the slide projectors so that some other patch reﬂected light that gave
the same photometer measurements, and asked the audience to name that patch. The
reply would describe the patch’s color in white light—if the patch looked blue in white
light, the answer would be “blue” (on the right). In later versions of this demonstration,
Land put wedge-shaped neutral density ﬁlters into the slide projectors so that the color
of the light illuminating the quilt of papers would vary slowly across the quilt. Again,
although the photometer readings vary signiﬁcantly from one end of a patch to another,
the audience sees the patch as having a constant color.
disregard information from their color constancy system. As a result, people can
often report:
• the color a surface would have in white light (often called surface color);
• the color of the light arriving at the eye (a useful skill that allows artists to
paint surfaces illuminated by colored lighting); and
• the color of the light falling on the surface.
The model of image color in Section 3.4 is
C(x) = g
d
(x)d(x) +g
s
(x)s(x) +i(x).
We decided to ignore the interreﬂection term i(x). In principle, we could use
the methods of Section 3.5.1 to generate new images without specularities. This
brings us to the term g
d
(x)d(x). Assume that g
d
(x) is a constant, so we are
viewing a ﬂat, frontal surface. The resulting term, d(x), models the world as a
collage of ﬂat, frontal, diﬀuse colored surfaces. Such worlds are sometimes called
Mondrian worlds, after the painter. Notice that, under our assumptions, d(x)
consists of a set of patches of ﬁxed color. We assume that there is a single illuminant
that has a constant color over the whole image. This term is a conglomeration of
Section 3.5 Inference from Color 97
illuminant, receptor, and reﬂectance information. It is impossible to disentangle
these completely in a realistic world. However, current algorithms can make quite
usable estimates of surface color from image colors given a well-populated world of
colored surfaces and a reasonable illuminant.
Recall from Section 3.4 that if a patch of perfectly diﬀuse surface with diﬀuse
spectral reﬂectance ρ(λ) is illuminated by a light whose spectrum is E(λ), the
spectrum of the reﬂected light is ρ(λ)E(λ) (multiplied by some constant to do
with surface orientation, which we have already decided to ignore). If a linear
photoreceptor of the kth type sees this surface patch, its response is:
p
k
=
_
Λ
σ
k
(λ)ρ(λ)E(λ)dλ,
where Λ is the range of all relevant wavelengths, and σ
k
(λ) is the sensitivity of the
kth photoreceptor.
Finite-Dimensional Linear Models
This response is linear in the surface reﬂectance and linear in the illumination,
which suggests using linear models for the families of possible surface reﬂectances
and illuminants. A ﬁnite-dimensional linear model models surface spectral albedoes
and illuminant spectral energy density as a weighted sum of a ﬁnite number of basis
functions. We need not use the same bases for reﬂectances and for illuminants.
If a ﬁnite-dimensional linear model of surface reﬂectance is a reasonable de-
scription of the world, any surface reﬂectance can be written as
ρ(λ) =
n

j=1
r
j
φ
j
(λ),
where the φ
j
(λ) are the basis functions for the model of reﬂectance, and the r
j
vary from surface to surface. Similarly, if a ﬁnite-dimensional linear model of the
illuminant is a reasonable model, any illuminant can be written as
E(λ) =
m

i=1
e
i
ψ
i
(λ),
where the ψ
i
(λ) are the basis functions for the model of illumination.
When both models apply, the response of a receptor of the kth type is
p
k
=
_
σ
k
(λ)
⎛
⎝
n

j=1
r
j
φ
j
(λ)
⎞
⎠
_
m

i=1
e
i
ψ
i
(λ)
_
dλ
=
m,n

i=1,j=1
e
i
r
j
__
σ
k
(λ)φ
j
(λ)ψ
i
(λ)
_
dλ
=
m,n

i=1,j=1
e
i
r
j
g
ijk
,
Section 3.5 Inference from Color 98
where we expect that the
g
ijk
=
_
σ
k
(λ)φ
j
(λ)ψ
i
(λ)dλ
are known, as they are components of the world model (they can be learned from
observations; see the exercises).
Inferring Surface Color
The ﬁnite-dimensional linear model describes the interaction between illumi-
nation color, surface color, and image color. To infer surface color from image color,
we need some sort of assumption. There are several plausible cues that can be used.
Specular reﬂections at dielectric surfaces have uniform specular albedo. We
could ﬁnd the specularities with the methods of that section, then recover surface
color using this information. At a specularity, we have
p
k
=
_
σ
k
(λ)
m

i=1
e
i
ψ
i
(λ)dλ,
and so if we knew the spectral sensitivities of the sensor and the basis functions ψ
i
,
we could solve for e
i
by solving a linear system. Now we know all e
i
, and all p
k
for
each pixel. We can solve the linear system
p
k
=
m,n

i=1,j=1
e
i
r
j
g
ijk
in the unknown r
j
to recover reﬂectance coeﬃcients.
Known average reﬂectance is another plausible cue. In this case, we
assume that the spatial average of reﬂectance in all scenes is constant and known
(e.g., we might assume that all scenes have a spatial average of reﬂectance that is
dull gray). In the ﬁnite-dimensional basis for reﬂectance, we can write this average
as
n

j=1
r
j
φ
j
(λ).
Now if the average reﬂectance is constant, the average of the receptor responses
must be constant too (if the imaging process is linear; see the discussion), and the
average of the response of the kth receptor can be written as:
p
k
=
m,n

i=1,j=1
e
i
g
ijk
r
j
.
We know p
k
and r
j
, and so have a linear system in the unknown light coeﬃcients
e
i
. We solve this, and then recover reﬂectance coeﬃcients at each pixel, as for the
case of specularities. For reasonable choices of reﬂectors and dimension of light and
surface basis, this linear system will have full rank.
The gamut of a color image is revealing. The gamut is the set of diﬀerent
colors that appears in the image. Generally, it is diﬃcult to obtain strongly colored
Section 3.6 Notes 99
pixels under white light with current imaging systems. Furthermore, if the picture
is taken under strongly colored light, that will tend to bias the gamut. One doesn’t
see bright green pixels in images taken under deep red light, for example. As
a result, the image gamut is a source of information about the illumination. If
an image gamut contains two pixel values—call them p
1
and p
2
—then it must
be possible to take an image under the same illuminant that contains the value
tp
1
+(1 −t)p
2
for 0 ≤ t ≤ 1 (because we could mix the colorants on the surfaces).
This means that the illuminant information depends on the convex hull of the
image gamut. There are now various methods to exploit these observations. There
is usually more than one illuminant consistent with a given image gamut, and
geometric methods can be used to identify the consistent illuminants. This set can
be narrowed down using probabilistic methods (for example, images contain lots of
diﬀerent colors (Forsyth 1990)) or physical methods (for example, the main sources
of illumination are the sun and the sky, well modelled as black bodies (Finlayson
and Hordley 2000)).
3.6 NOTES
There are a number of important general resources on the use of color. We rec-
ommend Hardin and Maﬃ (1997), Lamb and Bourriau (1995), Lynch and Liv-
ingston (2001), Minnaert (1993), Trussell et al. (1997), Williamson and Cummins
(1983). Wyszecki and Stiles (1982) contains an enormous amount of helpful infor-
mation. Recent textbooks with an emphasis on color include Velho et al. (2008), Lee
(2009), Reinhard et al. (2008), Gevers et al. (2011) and Burger and Burge (2009).
Trichromacy and Color Spaces
Until quite recently, there was no conclusive explanation of why trichromacy ap-
plied, although it was generally believed to be due to the presence of three diﬀerent
types of color receptor in the eye. Work on the genetics of photoreceptors can be
interpreted as conﬁrming this hunch (see Nathans et al. (1986a) and Nathans et al.
(1986b)), although a full explanation is still far from clear because this work can
also be interpreted as suggesting many individuals have more than three types of
photoreceptor (Mollon 1995).
There is an astonishing number of color spaces and color appearance models
available. The important issue is not in what coordinate system one measures color,
but how one counts the diﬀerence, so color metrics may still bear some thought.
Color metrics are an old topic; usually, one ﬁts a metric tensor to MacAdam el-
lipses. The diﬃculty with this approach is that a metric tensor carries the strong im-
plication that you can measure diﬀerences over large ranges by integration, whereas
it is very hard to see large-range color comparisons as meaningful. Another con-
cern is that the weight observers place on a diﬀerence in a Maxwellian view and the
semantic signiﬁcance of a diﬀerence in image colors are two very diﬀerent things.
Specularity Finding
The specularity ﬁnding method we describe is due to Shafer (1985), with improve-
ments due to Klinker et al. (1987), (1990), and to Maxwell and Shafer (2000).
Section 3.6 Notes 100
Specularities can also be detected because they are small and bright (Brelstaﬀ and
Blake 1988a), because they diﬀer in color and motion from the background (Lee
and Bajcsy 1992a, Lee and Bajcsy 1992b, Zheng and Murata 2000), or because
they distort patterns (Del Pozo and Savarese 2007). Specularities are a prodi-
gious nuisance in reconstruction, because specularities cause matching points in
diﬀerent images to have diﬀerent colors; various motion-based strategies have been
developed to remove them in these applications (Lin et al. 2002, Swaminathan et
al. 2002, Criminisi et al. 2005).
Color Constancy
Land reported a variety of color vision experiments (Land (1959a), (1959b), (1959c),
(1983)). Finite-dimensional linear models for spectral reﬂectances can be supported
by an appeal to surface physics as spectral absorption lines are thickened by solid
state eﬀects. The main experimental justiﬁcations for ﬁnite-dimensional linear
models of surface reﬂectance are measurements, by Cohen (1964), of the surface re-
ﬂectance of a selection of standard reference surfaces known as Munsell chips, and
measurements of a selection of natural objects by Krinov (1947). Cohen (1964)
performed a principal axis decomposition of his data to obtain a set of basis func-
tions, and Maloney (1984) ﬁtted weighted sums of these functions to Krinov’s date
to get good ﬁts with patterned deviations. The ﬁrst three principal axes explained
in each case a high percentage of the sample variance (near 99 %), and hence a
linear combination of these functions ﬁtted all the sampled functions rather well.
More recently, Maloney (1986) ﬁtted Cohen’s (1964) basis vectors to a large set of
data, including Krinov’s (1947) data, and further data on the surface reﬂectances
of Munsell chips, and concluded that the dimension of an accurate model of surface
reﬂectance was on the order of ﬁve or six.
Finite-dimensional linear models are an important tool in color constancy.
There is a large collection of algorithms that follow rather naturally from the ap-
proach. Some algorithms exploit the properties of the linear spaces involved (Mal-
oney (1984); Maloney and Wandell (1986); Wandell (1987)). Illumination can be
inferred from: reference objects (Abdellatif et al. 2000); specular reﬂections (Judd
(Judd 1960) writing in 1960 about early German work in surface color perception
refers to this as “a more usual view”; recent work includes (D’Zmura and Lennie
1986, Flock 1984, Klinker et al. 1987, Lee 1986)); the average color (Buchsbaum
1980, Gershon 1987, Gershon et al. 1986); and the gamut (Forsyth (1990), Barnard
(2000), Finlayson and Hordley (1999), (2000)).
The structure of the family of maps associated with a change in illumination
has been studied quite extensively. The ﬁrst work is due to Von Kries (who didn’t
think about it quite the way we do). He assumed that color constancy was, in
essence, the result of independent lightness calculations in each channel, meaning
that one can rectify an image by scaling each channel independently. This practice
is known as Von Kries’ law. The law boils down to assuming that the family of
maps consists of diagonal matrices. Von Kries’ law has proved to be a remarkably
good law (Finlayson et al. 1994a). Current best practice involves applying a linear
transformation to the channels and then scaling the result using diagonal maps
(Finlayson et al. (1994a), (1994b)).
Section 3.6 Notes 101
Reference datasets are available for testing methods (Barnard et al. 2002c).
Color constancy methods seem to work quite well in practice (Barnard et al.
2002a, Barnard et al. 2002b); whether this is good enough is debated (Funt et
al. 1998, Hordley and Finlayson 2006). Probabilistic methods can be applied to
color constancy (Freeman and Brainard 1997). Prior models on illumination are a
signiﬁcant cue (Kawakami et al. 2007).
There is surprisingly little work on color constancy that uniﬁes a study of the
spatial variation in illumination with solutions for surface color, which is why we
were reduced to ignoring a number of terms in our color model. Ideally, one would
work in shadows and surface orientation, too. Again, the whole thing looks like an
inference problem to us, but a subtle one. The main papers on this extremely im-
portant topic are Barnard et al. (1997), Funt and Drew (1988). There is substantial
room for research here, too.
Interreﬂections between colored surfaces lead to a phenomenon called color
bleeding, where each surface reﬂects colored light onto the other. The phenomenon
can be surprisingly large in practice. People seem to be quite good at ignoring it
entirely, to the extent that most people don’t realize that the phenomenon occurs
at all. Discounting color bleeding probably uses spatial cues. Some skill is required
to spot really compelling examples. The best known to the authors is occasionally
seen in southern California, where there are many large hedges of white oleander
by the roadside. White oleander has dark leaves and white ﬂowers. Occasionally, in
bright sunlight, one sees a hedge with yellow oleander ﬂowers; a moment’s thought
attributes the color to the yellow service truck parked by the road reﬂecting yellow
light onto the white ﬂowers. One’s ability to discount color bleeding eﬀects seems
to have been disrupted by the dark leaves of the plant breaking up the spatial
pattern. Color bleeding contains cues to surface color that are quite diﬃcult to
disentangle (see Drew and Funt (1990), Funt and Drew (1993), and Funt et al.
(1991) for studies).
It is possible to formulate and attack color constancy as an inference prob-
lem (Forsyth 1999, Freeman and Brainard 1997). The advantage of this approach
is that, for given data, the algorithm could report a range of possible surface colors,
with posterior weights.
PROBLEMS
3.1. Sit down with a friend and a packet of colored papers, and compare the color
names that you use. You need a large packet of papers—one can very often
get collections of colored swatches for paint, or for the Pantone color system
very cheaply. The best names to try are basic color names—the terms red,
pink, orange, yellow, green, blue, purple, brown, white, gray and black, which
(with a small number of other terms) have remarkable canonical properties
that apply widely across diﬀerent languages (the papers in Hardin and Maﬃ
(1997) give a good summary of current thought on this issue). You will ﬁnd it
surprisingly easy to disagree on which colors should be called blue and which
green, for example.
3.2. Derive the equations for transforming from RGB to CIE XYZ and back. This
is a linear transformation. It is suﬃcient to write out the expressions for the
elements of the linear transformation—you don’t have to look up the actual
numerical values of the color matching functions.
Section 3.6 Notes 102
3.3. Linear color spaces are obtained by choosing primaries and then construct-
ing color matching functions for those primaries. Show that there is a linear
transformation that takes the coordinates of a color in one linear color space to
those in another; the easiest way to do this is to write out the transformation
in terms of the color matching functions.
3.4. Exercise 3 means that, in setting up a linear color space, it is possible to choose
primaries arbitrarily, but there are constraints on the choice of color matching
functions. Why? What are these constraints?
3.5. Two surfaces that have the same color under one light and diﬀerent colors
under another are often referred to as metamers. An optimal color is a spectral
reﬂectance or radiance that has value 0 at some wavelengths and 1 at others.
Although optimal colors don’t occur in practice, they are a useful device (due
to Ostwald) for explaining various eﬀects.
(a) Use optimal colors to explain how metamerism occurs.
(b) Given a particular spectral albedo, show that there are an inﬁnite number
of metameric spectral albedoes.
(c) Use optimal colors to construct an example of surfaces that look diﬀerent
under one light (say, red and green) and the same under another.
(d) Use optimal colors to construct an example of surfaces that swop apparent
color when the light is changed (i.e., surface one looks red and surface two
looks green under light one, and surface one looks green and surface two
looks red under light two).
3.6. You have to map the gamut for a printer to that of a monitor. There are colors
in each gamut that do not appear in the other. Given a monitor color that
can’t be reproduced exactly, you could choose the printer color that is closest.
Why is this a bad idea for reproducing images? Would it work for reproducing
“business graphics” (bar charts, pie charts, and the like, which all consist of
many diﬀerernt large blocks of a single color)?
3.7. Volume color is a phenomenon associated with translucent materials that are
colored—the most attractive example is a glass of wine. The coloring comes
from diﬀerent absorption coeﬃcients at diﬀerent wavelengths. Explain (a)
why a small glass of suﬃciently deeply colored red wine (a good Cahors or
Gigondas) looks black (b) why a big glass of lightly colored red wine also looks
black. Experimental work is optional.
3.8. Read the book Colour: Art and Science, by Lamb and Bourriau, Cambridge
University Press, 1995.
3.9. In section 3.5.3, we described the gamut as a possible cue to illuminant color.
Write G for the convex hull of the gamut of the given image, W for the convex
hull of the gamut of an image of many diﬀerent surfaces under white light, and
/
e
for the map that takes an image seen under illuminant e to an image seen
under white light.
(a) Show the only illuminants we need to consider are those such that /
e
(G) ∈
W.
(b) Show that, for the case of ﬁnite dimensional linear models, /
e
depends
linearly on e.
(c) Show that, for the case of ﬁnite dimensional linear models, the set of /
e
such that /
e
(G) ∈ W is convex.
(d) How would you represent this set?
PROGRAMMING EXERCISES
3.10. Spectra for illuminants and for surfaces are available on the web
Section 3.6 Notes 103
(try http://www.it.lut.fi/ip/research/color/database/database.html).
Fit a ﬁnite-dimensional linear model to a set of illuminants and surface re-
ﬂectances using principal components analysis, render the resulting models,
and compare your rendering with an exact rendering. Where do you get the
most signiﬁcant errors? Why?
3.11. Print a colored image on a color inkjet printer using diﬀerent papers and
compare the result. It is particularly informative to (a) ensure that the driver
knows what paper the printer will be printing on, and compare the variations
in colors (which are ideally imperceptible), and (b) deceive the driver about
what paper it is printing on (i.e., print on plain paper and tell the driver it
is printing on photographic paper). Can you explain the variations you see?
Why is photographic paper glossy?
3.12. Fitting a ﬁnite-dimensional linear model to illuminants and reﬂectances sepa-
rately is somewhat ill-advised because there is no guarantee that the interac-
tions will be represented well (they’re not accounted for in the ﬁtting error).
It turns out that one can obtain g
ijk
by a ﬁtting process that sidesteps the
use of basis functions. Implement this procedure (which is described in de-
tail in Marimont and Wandell (1992)), and compare the results with those
obtained from the previous assignment.
3.13. Build a color constancy algorithm that uses the assumption that the spatial
average of reﬂectance is constant. Use ﬁnite-dimensional linear models. You
can get values of g
ijk
from your solution to the previous exercise.
3.14. We ignore color interreﬂections in our surface color model. Do an experiment
to get some idea of the size of color shifts possible from color interreﬂections
(which are astonishingly big). Humans seldom interpret color interreﬂections
as surface color. Speculate as to why this might be the case, using the discus-
sion of the lightness algorithm as a guide.
3.15. Build a specularity ﬁnder along the lines described in Section 3.5.1.
3.16. Build a shadow remover along the lines described in Section 3.5.2 (this is much
easier than it sounds, and the results are usually rather good).
This page intentionally left blank
P A R T T W O
EARLY VISION: JUST ONE IMAGE
This page intentionally left blank
C H A P T E R 4
Linear Filters
Pictures of zebras and of dalmatians have black and white pixels, and in about the
same number, too. The diﬀerences between the two have to do with the character-
istic appearance of small groups of pixels, rather than individual pixel values. In
this chapter, we introduce methods for obtaining descriptions of the appearance of
a small group of pixels.
Our main strategy is to use weighted sums of pixel values using diﬀerent
patterns of weights to ﬁnd diﬀerent image patterns. Despite its simplicity, this
process is extremely useful. It allows us to smooth noise in images, and to ﬁnd
edges and other image patterns.
4.1 LINEAR FILTERS AND CONVOLUTION
Many important eﬀects can be modeled with a simple model. Construct a new
array, the same size as the image. Fill each location of this new array with a
weighted sum of the pixel values from the locations surrounding the corresponding
location in the image using the same set of weights each time. Diﬀerent sets of
weights could be used to represent diﬀerent processes. One example is computing
a local average taken over a ﬁxed region. We could average all pixels within a
2k + 1 2k + 1 block of the pixel of interest. For an input image T, this gives an
output
!
ij
=
1
(2k + 1)
2
u=i+k

u=i−k
v=j+k

v=j−k
T
uv
.
The weights in this example are simple (each pixel is weighted by the same con-
stant), but we could use a more interesting set of weights. For example, we could
use a set of weights that was large at the center and fell oﬀ sharply as the distance
from the center increased to model the kind of smoothing that occurs in a defocused
lens system.
Whatever the weights chosen, the output of this procedure is shift invari-
ant—meaning that the value of the output depends on the pattern in an image
neighborhood, rather than the position of the neighborhood—and linear—meaning
that the output for the sum of two images is the same as the sum of the outputs
obtained for the images separately. The procedure is known as linear ﬁltering.
4.1.1 Convolution
We introduce some notation at this point. The pattern of weights used for a linear
ﬁlter is usually referred to as the kernel of the ﬁlter. The process of applying the
ﬁlter is usually referred to as convolution. There is a catch: For reasons that will
appear later (Section 4.2.1), it is convenient to write the process in a non-obvious
way. In particular, given a ﬁlter kernel 1, the convolution of the kernel with image
107
Section 4.1 Linear Filters and Convolution 108
T is an image !. The i, jth component of ! is given by
R
ij
=

u,v
H
i−u,j−v
F
u,v
.
This process deﬁnes convolution: we say that 1 has been convolved with T to yield
!. You should look closely at this expression; the “direction” of the dummy variable
u (resp. v) has been reversed compared with correlation. This is important because
if you forget that it is there, you compute the wrong answer. The reason for the
reversal emerges from the derivation of Section 4.2.1. We carefully avoid inserting
the range of the sum; in eﬀect, we assume that the sum is over a large enough range
of u and v that all nonzero values are taken into account. Furthermore, we assume
that any values that haven’t been speciﬁed are zero; this means that we can model
the kernel as a small block of nonzero values in a sea of zeros. We use this common
convention regularly in what follows.
Example: Smoothing by Averaging
Images typically have the property that the value of a pixel usually is similar
to that of its neighbor. Assume that the image is aﬀected by noise of a form where
we can reasonably expect that this property is preserved. For example, there might
be occasional dead pixels, or small random numbers with zero mean might have
been added to the pixel values. It is natural to attempt to reduce the eﬀects of this
noise by replacing each pixel with a weighted average of its neighbors, a process
often referred to as smoothing or blurring.
FIGURE 4.1: Although a uniform local average may seem to give a good blurring model,
it generates eﬀects not usually seen in defocusing a lens. The images above compare the
eﬀects of a uniform local average with weighted average. The image on the left shows a
view of grass; in the center, the result of blurring this image using a uniform local model;
and on the right, the result of blurring this image using a set of Gaussian weights. The
degree of blurring in each case is about the same, but the uniform average produces a
set of narrow vertical and horizontal bars—an eﬀect often known as ringing. The small
insets show the weights used to blur the image, themselves rendered as an image; bright
points represent large values and dark points represent small values (in this example, the
smallest values are zero).
Replacing each pixel with an unweighted average computed over some ﬁxed
region centered at the pixel is the same as convolution with a kernel that is a block
Section 4.1 Linear Filters and Convolution 109
of ones multiplied by a constant. You can (and should) establish this point by close
attention to the range of the sum. This process is a poor model of blurring; its
output does not look like that of a defocused camera (Figure 4.1). The reason is
clear. Assume that we have an image in which every point but the center point is
zero, and the center point is one. If we blur this image by forming an unweighted
average at each point, the result looks like a small, bright box, but this is not what
defocused cameras do. We want a blurring process that takes a small bright dot
to a circularly symmetric region of blur, brighter at the center than at the edges
and fading slowly to darkness. As Figure 4.1 suggests, a set of weights of this form
produces a much more convincing defocus model.
-5
0
5
-5
0
5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
FIGURE 4.2: The symmetric Gaussian kernel in 2D. This view shows a kernel scaled so
that its sum is equal to one; this scaling is quite often omitted. The kernel shown has
σ = 1. Convolution with this kernel forms a weighted average that stresses the point at
the center of the convolution window and incorporates little contribution from those at
the boundary. Notice how the Gaussian is qualitatively similar to our description of the
point spread function of image blur: it is circularly symmetric, has strongest response in
the center, and dies away near the boundaries.
Example: Smoothing with a Gaussian
A good formal model for this fuzzy blob is the symmetric Gaussian kernel
G
σ
(x, y) =
1
2πσ
2
exp
_
−
(x
2
+y
2
)
2σ
2
_
illustrated in Figure 4.2. σ is referred to as the standard deviation of the Gaussian
(or its “sigma”!); the units are interpixel spaces, usually referred to as pixels. The
Section 4.1 Linear Filters and Convolution 110
constant term makes the integral over the whole plane equal to one and is often
ignored in smoothing applications. The name comes from the fact that this kernel
has the form of the probability density for a 2D normal (or Gaussian) random
variable with a particular covariance.
This smoothing kernel forms a weighted average that weights pixels at its
center much more strongly than at its boundaries. One can justify this approach
qualitatively: Smoothing suppresses noise by enforcing the requirement that pixels
should look like their neighbors. By downweighting distant neighbors in the average,
we can ensure that the requirement that a pixel looks like its neighbors is less
strongly imposed for distant neighbors. A qualitative analysis gives the following:
• If the standard deviation of the Gaussian is very small—say, smaller than one
pixel—the smoothing will have little eﬀect because the weights for all pixels
oﬀ the center will be very small.
• For a larger standard deviation, the neighboring pixels will have larger weights
in the weighted average, which in turn means that the average will be strongly
biased toward a consensus of the neighbors. This will be a good estimate of a
pixel’s value, and the noise will largely disappear at the cost of some blurring.
• Finally, a kernel that has a large standard deviation will cause much of the
image detail to disappear, along with the noise.
Figure 4.3 illustrates these phenomena. You should notice that Gaussian smoothing
can be eﬀective at suppressing noise.
In applications, a discrete smoothing kernel is obtained by constructing a
2k + 1 2k + 1 array whose i, jth value is
H
ij
=
1
2πσ
2
exp
_
−
((i −k −1)
2
+ (j −k −1)
2
)
2σ
2
_
.
Notice that some care must be exercised with σ. If σ is too small, then only one
element of the array will have a nonzero value. If σ is large, then k must be large,
too; otherwise, we are ignoring contributions from pixels that should contribute
with substantial weight.
Example: Derivatives and Finite Diﬀerences
Image derivatives can be approximated using another example of a convolution
process. Because
∂f
∂x
= lim
→0
f(x +, y) −f(x, y)

,
we might estimate a partial derivative as a symmetric ﬁnite diﬀerence:
∂h
∂x
≈ h
i+1,j
−h
i−1,j
.
This is the same as a convolution, where the convolution kernel is
1 =
⎧
⎨
⎩
0 0 0
1 0 −1
0 0 0
⎫
⎬
⎭
.
Section 4.1 Linear Filters and Convolution 111
σ=0.05 σ=0.1 σ=0.2
no
smoothing
σ=1 pixel
σ=2 pixels
FIGURE 4.3: The top row shows images of a constant mid-gray level corrupted by additive
Gaussian noise. In this noise model, each pixel has a zero-mean normal random variable
added to it. The range of pixel values is from zero to one, so that the standard deviation
of the noise in the ﬁrst column is about 1/20 of full range. The center row shows the
eﬀect of smoothing the corresponding image in the top row with a Gaussian ﬁlter of σ
one pixel. Notice the annoying overloading of notation here; there is Gaussian noise and
Gaussian ﬁlters, and both have σ’s. One uses context to keep these two straight, although
this is not always as helpful as it could be, because Gaussian ﬁlters are particularly good at
suppressing Gaussian noise. This is because the noise values at each pixel are independent,
meaning that the expected value of their average is going to be the noise mean. The
bottom row shows the eﬀect of smoothing the corresponding image in the top row with
a Gaussian ﬁlter of σ two pixels.
Notice that this kernel could be interpreted as a template: it gives a large positive
response to an image conﬁguration that is positive on one side and negative on the
other, and a large negative response to the mirror image.
As Figure 4.4 suggests, ﬁnite diﬀerences give a most unsatisfactory estimate
of the derivative. This is because ﬁnite diﬀerences respond strongly (i.e., have an
output with large magnitude) at fast changes, and fast changes are characteristic of
noise. Roughly, this is because image pixels tend to look like one another. For exam-
ple, if we had bought a discount camera with some pixels that were stuck at either
black or white, the output of the ﬁnite diﬀerence process would be large at those
Section 4.2 Shift Invariant Linear Systems 112
FIGURE 4.4: The top row shows estimates of derivatives obtained by ﬁnite diﬀerences.
The image at the left shows a detail from a picture of a zebra. The center image shows
the partial derivative in the y-direction—which responds strongly to horizontal stripes
and weakly to vertical stripes—and the right image shows the partial derivative in the
x-direction—which responds strongly to vertical stripes and weakly to horizontal stripes.
However, ﬁnite diﬀerences respond strongly to noise. The image at center left shows
a detail from a picture of a zebra; the next image in the row is obtained by adding a
random number with zero mean and normal distribution (σ = 0.03; the darkest value in
the image is 0, and the lightest 1) to each pixel; and the third image is obtained by adding
a random number with zero mean and normal distribution (σ = 0.09) to each pixel. The
bottom row shows the partial derivative in the x-direction of the image at the head
of the row. Notice how strongly the diﬀerentiation process emphasizes image noise; the
derivative ﬁgures look increasingly grainy. In the derivative ﬁgures, a mid-gray level is a
zero value, a dark gray level is a negative value, and a light gray level is a positive value.
pixels because they are, in general, substantially diﬀerent from their neighbors. All
this suggests that some form of smoothing is appropriate before diﬀerentiation; the
details appear in Section 5.1.
4.2 SHIFT INVARIANT LINEAR SYSTEMS
Convolution represents the eﬀect of a large class of system. In particular, most
imaging systems have, to a good approximation, three signiﬁcant properties:
• Superposition: We expect that
R(f +g) = R(f) +R(g);
Section 4.2 Shift Invariant Linear Systems 113
that is, the response to the sum of stimuli is the sum of the individual re-
sponses.
• Scaling: The response to a zero input is zero. Taken with superposition, we
have that the response to a scaled stimulus is a scaled version of the response
to the original stimulus; that is,
R(kf) = kR(f).
A device that exihibits superposition and scaling is linear.
• Shift invariance: In a shift invariant system, the response to a translated
stimulus is just a translation of the response to the stimulus. This means
that, for example, if a view of a small light aimed at the center of the camera
is a small, bright blob, then if the light is moved to the periphery, we should
see the same small, bright blob, only translated.
A device that is linear and shift invariant is known as a shift invariant linear system,
or often just as a system.
The response of a shift invariant linear system to a stimulus is obtained by
convolution. We demonstrate this ﬁrst for systems that take discrete inputs—say,
vectors or arrays—and produce discrete outputs. We then use this to describe the
behavior of systems that operate on continuous functions of the line or the plane,
and from this analysis we obtain some useful facts about convolution.
4.2.1 Discrete Convolution
In the 1D case, we have a shift invariant linear system that takes a vector and
responds with a vector. This case is the easiest to handle because there are fewer
indices to look after. The 2D case—a system that takes an array and responds with
an array—follows easily. In each case, we assume that the input and output are
inﬁnite dimensional. This allows us to ignore some minor issues that arise at the
boundaries of the input. We deal with these in Section 4.2.3.
Discrete Convolution in One Dimension
We have an input vector f. For convenience, we assume that the vector has
inﬁnite length and its elements are indexed by the integers (i.e., there is an element
with index −1, say). The ith component of this vector is f
i
. Now f is a weighted
sum of basis elements. A convenient basis is a set of elements that have a one in a
single component and zeros elsewhere. We write
e
0
= . . . 0, 0, 0, 1, 0, 0, 0, . . .
This is a data vector that has a 1 in the zeroth place, and zeros elsewhere. Deﬁne
a shift operation, which takes a vector to a shifted version of that vector. In
particular, the vector Shift(f, i) has, as its jth component, the j −ith component
of f. For example, Shift(e
0
, 1) has a zero in the ﬁrst component. Now, we can
write
f =

i
f
i
Shift(e
0
, i).
Section 4.2 Shift Invariant Linear Systems 114
We write the response of our system to a vector f as
R(f).
Now, because the system is shift invariant, we have
R(Shift(f, k)) = Shift(R(f), k).
Furthermore, because it is linear, we have
R(kf) = kR(f).
This means that
R(f) = R
_

i
f
i
Shift(e
0
, i)
_
=

i
R(f
i
Shift(e
0
, i))
=

i
f
i
R(Shift(e
0
, i))
=

i
f
i
Shift(R(e
0
), i)).
This means that to obtain the system’s response to any data vector, we need to
know only its response to e
0
. This is usually called the system’s impulse response.
Assume that the impulse response can be written as g. We have
R(f) =

i
f
i
Shift(g, i) = g ∗ f.
This deﬁnes an operation—the 1D, discrete version of convolution—which we write
with a ∗.
This is all very well, but it doesn’t give us a particularly easy expression for
the output. If we consider the jth element of R(f), which we write as R
i
, we must
have
R
j
=

u,v
G
i−u,j−v
F
uv
,
which we write as
! = ( ∗ ∗1.
4.2.2 Continuous Convolution
There are shift invariant linear systems that produce a continuous response to a
continuous input; for example, a camera lens takes a set of radiances and produces
another set, and many lenses are approximately shift invariant. A brief study of
these systems allows us to study the information lost by approximating a contin-
uous function—the incoming radiance values across an image plane—by a discrete
function—the value at each pixel.
The natural description is in terms of the system’s response to a rather un-
natural function, the δ-function, which is not a function in formal terms. We do
the derivation ﬁrst in one dimension to make the notation easier.
Convolution in One Dimension
We obtain an expression for the response of a continuous shift invariant linear
system from our expression for a discrete system. We can take a discrete input and
replace each value with a box straddling the value; this gives a continuous input
function. We then make the boxes narrower and consider what happens in the
limit.
Our system takes a function of one dimension and returns a function of one
dimension. Again, we write the response of the system to some input f(x) as R(f);
when we need to emphasize that f is a function, we write R(f(x)). The response
is also a function; occasionally, when we need to emphasize this fact, we write
R(f)(u). We can express the linearity property in this notation by writing
R(kf) = kR(f)
(for k some constant) and the shift invariance property by introducing a Shift
operator, which takes functions to functions:
Shift(f, c) = f(u −c).
With this Shift operator, we can write the shift invariance property as
R(Shift(f, c)) = Shift(R(f), c).
We deﬁne the box function as:
box

(x) =
_
0 abs(x) >

2
1 abs(x) <

2
.
Section 4.2 Shift Invariant Linear Systems 116
The value of box

(/2) does not matter for our purposes. The input function is f(x).
We construct an even grid of points x
i
, where x
i+1
− x
i
= . We now construct
a vector f whose ith component (written f
i
) is f(x
i
). This vector can be used to
represent the function.
We obtain an approximate representation of f by

i
f
i
Shift(box

, x
i
). We
apply this input to a shift invariant linear system; the response is a weighted sum
of shifted responses to box functions. This means that
R
_

i
f
i
Shift(box

, x
i
)
_
=

i
R(f
i
Shift(box

, x
i
))
=

i
f
i
R(Shift(box

, x
i
))
=

i
f
i
Shift(R(
box

), x
i
)
=

i
f
i
Shift(R(
box

), x
i
).
So far, everything has followed our derivation for discrete functions. We now have
something that looks like an approximate integral if →0.
We introduce a new device, called a δ-function, to deal with the term box

/.
Deﬁne
d

(x) =
box

(x)

.
The δ-function is:
δ(x) = lim
→0
d

(x).
We don’t attempt to evaluate this limit, so we need not discuss the value of
δ(0). One interesting feature of this function is that, for practical shift invariant
linear systems, the response of the system to a δ-function exists and has compact
support (i.e., is zero except on a ﬁnite number of intervals of ﬁnite length). For
example, a good model of a δ-function in 2D is an extremely small, extremely
bright light. If we make the light smaller and brighter while ensuring the total
energy is constant, we expect to see a small but ﬁnite spot due to the defocus of
the lens. The δ-function is the natural analogue for e
0
in the continuous case.
This means that the expression for the response of the system,

i
f
i
Shift(R(
box

), x
i
),
turns into an integral as limits to zero. We obtain
R(f) =
_
¦R(δ)(u −x

)¦ f(x

)dx

=
_
g(u −x

)f(x

)dx

,
Section 4.2 Shift Invariant Linear Systems 117
where we have written R(δ)—which is usually called the impulse response of
the system—as g and have omitted the limits of the integral. These integrals could
be from −∞ to ∞, but more stringent limits could apply if g and h have compact
support. This operation is called convolution (again), and we write the foregoing
expression as
R(f) = (g ∗ f).
Convolution is commutative, meaning
(g ∗ h)(x) = (h ∗ g)(x).
Convolution is associative, meaning that
(f ∗ (g ∗ h)) = ((f ∗ g) ∗ h).
This latter property means that we can ﬁnd a single shift invariant linear system
that behaves like the composition of two diﬀerent systems. This will be useful when
we discuss sampling.
Convolution in Two Dimensions
The derivation of convolution in two dimensions requires more notation. A
box function is now given by box

2(x, y) = box

(x)box

(y); we now have
d

(x, y) =
box

2(x, y)

2
.
The δ-function is the limit of d

(x, y) function as → 0. Finally, there are more
terms in the sum. All this activity results in the expression
R(h)(x, y) =
_ _
g(x −x

, y −y

)h(x

, y

)dxdy
= (g ∗ ∗h)(x, y),
where we have used two ∗s to indicate a two-dimensional convolution. Convolution
in 2D is commutative, meaning that
(g ∗ ∗h) = (h ∗ ∗g),
and associative, meaning that
((f ∗ ∗g) ∗ ∗h) = (f ∗ ∗(g ∗ ∗h)).
A natural model for the impulse response of a two-dimensional system is to think
of the pattern seen in a camera viewing a very small, distant light source (which
subtends a very small viewing angle). In practical lenses, this view results in some
form of fuzzy blob, justifying the name point spread function, which is often used
for the impulse response of a 2D system. The point spread function of a linear
system is often known as its kernel.
Section 4.3 Spatial Frequency and Fourier Transforms 118
4.2.3 Edge Eﬀects in Discrete Convolutions
In practical systems, we cannot have inﬁnite arrays of data. This means that when
we compute the convolution, we need to contend with the edges of the image; at
the edges, there are pixel locations where computing the value of the convolved
image requires image values that don’t exist. There are a variety of strategies we
can adopt:
• Ignore these locations, which means that we report only values for which
every required image location exists. This has the advantage of probity, but
the disadvantage that the output is smaller than the input. Repeated convo-
lutions can cause the image to shrink quite drastically.
• Pad the image with constant values, which means that, as we look at
output values closer to the edge of the image, the extent to which the output
of the convolution depends on the image goes down. This is a convenient
trick because we can ensure that the image doesn’t shrink, but it has the
disadvantage that it can create the appearance of substantial gradients near
the boundary.
• Pad the image in some other way. For example, we might think of
the image as a doubly periodic function so that if we have an n m image,
then column m+ 1—required for the purposes of convolution—would be the
same as column m−1. This can create the appearance of substantial second
derivative values near the boundary.
4.3 SPATIAL FREQUENCY AND FOURIER TRANSFORMS
We have used the trick of thinking of a signal g(x, y) as a weighted sum of a large (or
inﬁnite) number of small (or inﬁnitely small) box functions. This model emphasizes
that a signal is an element of a vector space. The box functions form a convenient
basis, and the weights are coeﬃcients on this basis. We need a new technique to
deal with two related problems so far left open:
• Although it is clear that a discrete image version cannot represent the full
information in a signal, we have not yet indicated what is lost.
• It is clear that we cannot shrink an image simply by taking every kth pixel—
this could turn a checkerboard image all white or all black—and we would
like to know how to shrink an image safely.
All of these problems are related to the presence of fast changes in an image. For
example, shrinking an image is most likely to miss fast eﬀects because they could
slip between samples; similarly, the derivative is large at fast changes.
These eﬀects can be studied by a change of basis. We change the basis to be
a set of sinusoids and represent the signal as an inﬁnite weighted sum of an inﬁnite
number of sinusoids. This means that fast changes in the signal are obvious, because
they correspond to large amounts of high-frequency sinusoids in the new basis.
Section 4.3 Spatial Frequency and Fourier Transforms 119
FIGURE 4.5: The real component of Fourier basis elements shown as intensity images.
The brightest point has value one, and the darkest point has value zero. The domain is
[−1, 1] [−1, 1], with the origin at the center of the image. On the left, (u, v) = (0, 0.4);
in the center, (u, v) = (1, 2); and on the right (u, v) = (10, −5). These are sinusoids of
various frequencies and orientations described in the text.
4.3.1 Fourier Transforms
The change of basis is eﬀected by a Fourier transform. We deﬁne the Fourier
transform of a signal g(x, y) to be
T(g(x, y))(u, v) =
_
∞
_
−∞
g(x, y)e
−i2π(ux+vy)
dxdy.
Assume that appropriate technical conditions are true to make this integral
exist. It is suﬃcient for all moments of g to be ﬁnite; a variety of other possible
conditions are available (Bracewell 1995). The process takes a complex valued
function of x, y and returns a complex valued function of u, v (images are complex
valued functions with zero imaginary component).
For the moment, ﬁx u and v, and let us consider the meaning of the value of
the transform at that point. The exponential can be rewritten
e
−i2π(ux+vy)
= cos(2π(ux +vy)) +i sin(2π(ux +vy)).
These terms are sinusoids on the x, y plane, whose orientation and frequency are
given by u, v. For example, consider the real term, which is constant when ux+vy
is constant (i.e., along a straight line in the x, y plane whose orientation is given by
tanθ = v/u). The gradient of this term is perpendicular to lines where ux + vy is
constant, and the frequency of the sinusoid is
√
u
2
+v
2
. These sinusoids are often
referred to as spatial frequency components; a variety are illustrated in Figure 4.5.
The integral should be seen as a dot product. If we ﬁx u and v, the value
of the integral is the dot product between a sinusoid in x and y and the original
function. This is a useful analogy because dot products measure the amount of one
vector in the direction of another.
In the same way, the value of the transform at a particular u and v can be
seen as measuring the amount of the sinusoid with given frequency and orientation
Section 4.3 Spatial Frequency and Fourier Transforms 120
in the signal. The transform takes a function of x and y to the function of u and v
whose value at any particular (u, v) is the amount of that particular sinusoid in the
original function. This view justiﬁes the model of a Fourier transform as a change
of basis.
Linearity
The Fourier transform is linear:
T(g(x, y) +h(x, y)) = T(g(x, y)) +T(h(x, y))
and
T(kg(x, y)) = kT(g(x, y)).
The Inverse Fourier Transform It is useful to recover a signal from its
Fourier transform. This is another change of basis with the form
g(x, y) =
_
∞
_
−∞
T(g(x, y))(u, v)e
i2π(ux+vy)
dudv.
Fourier Transform Pairs Fourier transforms are known in closed form
for a variety of useful cases; a large set of examples appears in Bracewell (1995).
We list a few in Table 4.1 for reference. The last line of Table 4.1 contains the
convolution theorem; convolution in the signal domain is the same as multiplication
in the Fourier domain.
Phase and Magnitude The Fourier transform consists of a real and a
complex component:
T(g(x, y))(u, v) =
_ _
∞
−∞
g(x, y) cos(2π(ux +vy))dxdy +
i
_ _
∞
−∞
g(x, y) sin(2π(ux +vy))dxdy
= 1(T(g)) +i ∗ ·(T(g))
= T
R
(g) +i ∗ T
I
(g).
It is usually inconvenient to draw complex functions of the plane. One solution
is to plot T
R
(g) and T
I
(g) separately; another is to consider the magnitude and
phase of the complex functions, and to plot these instead. These are then called
the magnitude spectrum and phase spectrum, respectively.
The value of the Fourier transform of a function at a particular u, v point
depends on the whole function. This is obvious from the deﬁnition because the
domain of the integral is the whole domain of the function. It leads to some subtle
properties, however. First, a local change in the function (e.g., zeroing out a block
of points) is going to lead to a change at every point in the Fourier transform. This
means that the Fourier transform is quite diﬃcult to use as a representation (e.g.,
Section 4.4 Sampling and Aliasing 121
TABLE 4.1: A variety of functions of two dimensions and their Fourier transforms. This
table can be used in two directions (with appropriate substitutions for u, v and x, y)
because the Fourier transform of the Fourier transform of a function is the function.
Observant readers might suspect that the results on inﬁnite sums of δ functions contradict
the linearity of Fourier transforms. By careful inspection of limits, it is possible to show
that they do not (see, for example, Bracewell (1995)). Observant readers also might have
noted that an expression for T(
∂f
∂y
) can be obtained by combining two lines of this table.
Function Fourier transform
g(x, y)
_
∞
_
−∞
g(x, y)e
−i2π(ux+vy)
dxdy
_
∞
_
−∞
T(g(x, y))(u, v)e
i2π(ux+vy)
dudv T(g(x, y))(u, v)
δ(x, y) 1
∂f
∂x
(x, y) uT(f)(u, v)
0.5δ(x +a, y) + 0.5δ(x −a, y) cos 2πau
e
−π(x
2
+y
2
)
e
−π(u
2
+v
2
)
box
1
(x, y)
sin u
u
sin v
v
f(ax, by)
F(f)(u/a,v/b)
ab

∞
i=−∞

∞
j=−∞
δ(x −i, y −j)

∞
i=−∞

∞
j=−∞
δ(u −i, v −j)
(f ∗ ∗g)(x, y) T(f)T(g)(u, v)
f(x −a, y −b) e
−i2π(au+bv)
T(f)
f(x cos θ −y sin θ, xsin θ +y cos θ) T(f)(u cos θ −v sin θ, usin θ +v cos θ)
it might be very diﬃcult to tell whether a pattern was present in an image just by
looking at the Fourier transform). Second, the magnitude spectra of images tends
to be similar. This appears to be a fact of nature, rather than something that
can be proven axiomatically. As a result, the magnitude spectrum of an image is
surprisingly uninformative (see Figure 4.6 for an example).
4.4 SAMPLING AND ALIASING
The crucial reason to discuss Fourier transforms is to get some insight into the
diﬀerence between discrete and continuous images. In particular, it is clear that
some information has been lost when we work on a discrete pixel grid, but what?
A good, simple example comes from an image of a checkerboard, and is given in
Section 4.4 Sampling and Aliasing 122
FIGURE 4.6: The second image in each row shows the log of the magnitude spectrum for
the ﬁrst image in the row; the third image shows the phase spectrum scaled so that −π
is dark and π is light. The ﬁnal images are obtained by swapping the magnitude spectra.
Although this swap leads to substantial image noise, it doesn’t substantially aﬀect the
interpretation of the image, suggesting that the phase spectrum is more important for
perception than the magnitude spectrum.
Figure 4.7. The problem has to do with the number of samples relative to the
function; we can formalize this rather precisely given a suﬃciently powerful model.
4.4.1 Sampling
Passing from a continuous function—like the irradiance at the back of a camera
system—to a collection of values on a discrete grid —like the pixel values reported
by a camera—is referred to as sampling. We construct a model that allows us to
obtain a precise notion of what is lost in sampling.
Sampling in One Dimension
Sampling in one dimension takes a function and returns a discrete set of
values. The most important case involves sampling on a uniform discrete grid, and
we assume that the samples are deﬁned at integer points. This means we have a
process that takes some function and returns a vector of values:
sample
1D
(f(x)) = f.
We model this sampling process by assuming that the elements of this vector
are the values of the function f(x) at the sample points and allowing negative
indices to the vector (Figure 4.8). This means that the ith component of f is
f(x
i
).
Sampling in Two Dimensions
Section 4.4 Sampling and Aliasing 123
FIGURE 4.7: The two checkerboards on the top illustrate a sampling procedure that
appears to be successful (whether it is or not depends on some details that we will deal
with later). The gray circles represent the samples; if there are suﬃcient samples, then
the samples represent the detail in the underlying function. The sampling procedures
shown on the bottom are unequivocally unsuccessful; the samples suggest that there are
fewer checks than there are. This illustrates two important phenomena: ﬁrst, successful
sampling schemes sample data often enough; and second, unsuccessful sampling schemes
cause high-frequency information to appear as lower-frequency information.
Sampling in 2D is very similar to sampling in 1D. Although sampling can
occur on nonregular grids (the best example being the human retina), we proceed
on the assumption that samples are drawn at points with integer coordinates. This
yields a uniform rectangular grid, which is a good model of most cameras. Our
sampled images are then rectangular arrays of ﬁnite size (all values outside the grid
being zero).
In the formal model, we sample a function of two dimensions, instead of one,
yielding an array (Figure 4.9). We allow this array to have negative indices in both
dimensions, and can then write
sample
2D
(F(x, y)) = T,
where the i, jth element of the array T is F(x
i
, y
j
) = F(i, j).
Samples are not always evenly spaced in practical systems. This is quite often
Section 4.4 Sampling and Aliasing 124
Sample
1D
FIGURE 4.8: Sampling in 1D takes a function and returns a vector whose elements are
values of that function at the sample points. For our purposes, it is enough that the sample
points be integer values of the argument. We allow the vector to be inﬁnite dimensional
and have negative as well as positive indices.
Sample
2D
FIGURE 4.9: Sampling in 2D takes a function and returns an array; again, we allow the
array to be inﬁnite dimensional and to have negative as well as positive indices.
due to the pervasive eﬀect of television; television screens have an aspect ratio of
4:3 (width:height). Cameras quite often accommodate this eﬀect by spacing sample
points slightly farther apart horizontally than vertically (in jargon, they have non-
square pixels).
Section 4.4 Sampling and Aliasing 125
A Continuous Model of a Sampled Signal
We need a continuous model of a sampled signal. Generally, this model is used
to evaluate integrals; in particular, taking a Fourier transform involves integrating
the product of our model with a complex exponential. It is clear how this integral
should behave: the value of the integral should be obtained by adding up values
at each integer point. This means we cannot model a sampled signal as a function
that is zero everywhere except at integer points (where it takes the value of the
signal), because this model has a zero integral.
An appropriate continuous model of a sampled signal relies on an important
property of the δ function:
_
∞
−∞
aδ(x)f(x)dx = a lim
→0
_
∞
−∞
d(x; )f(x)dx
= a lim
→0
_
∞
−∞
bar(x; )

(f(x))dx
= a lim
→0
∞

i=−∞
bar(x; )

(f(i)bar(x −i; ))
= af(0).
Here we have used the idea of an integral as the limit of a sum of small strips.
An appropriate continuous model of a sampled signal consists of a δ-function
at each sample point weighted by the value of the sample at that point. We can
obtain this model by multiplying the sampled signal by a set of δ-functions, one
at each sample point. In one dimension, a function of this form is called a comb
function (because that’s what the graph looks like). In two dimensions, a function
of this form is called a bed-of-nails function (for the same reason).
Working in 2D and assuming that the samples are at integer points, this
procedure gets
sample
2D
(f) =
∞

i=−∞
∞

j=−∞
f(i, j)δ(x −i, y −j)
= f(x, y)
⎧
⎨
⎩
∞

i=−∞
∞

j=−∞
δ(x −i, y −j)
⎫
⎬
⎭
.
This function is zero except at integer points (because the δ-function is zero except
at integer points), and its integral is the sum of the function values at the integer
points.
4.4.2 Aliasing
Sampling involves a loss of information. As this section shows, a signal sampled
too slowly is misrepresented by the samples; high spatial frequency components
of the original signal appear as low spatial frequency components in the sampled
signal—an eﬀect known as aliasing.
Section 4.4 Sampling and Aliasing 126
The Fourier Transform of a Sampled Signal
A sampled signal is given by a product of the original signal with a bed-of-
nails function. By the convolution theorem, the Fourier transform of this product
is the convolution of the Fourier transforms of the two functions. This means that
the Fourier transform of a sampled signal is obtained by convolving the Fourier
transform of the signal with another bed-of-nails function.
Now convolving a function with a shifted δ-function merely shifts the function
(see exercises). This means that the Fourier transform of the sampled signal is the
sum of a collection of shifted versions of the Fourier transforms of the signal, that
is,
T(sample
2D
(f(x, y))) = T
⎛
⎝
f(x, y)
⎧
⎨
⎩
∞

i=−∞
∞

j=−∞
δ(x −i, y −j)
⎫
⎬
⎭
⎞
⎠
= T(f(x, y)) ∗ ∗T
⎛
⎝
⎧
⎨
⎩
∞

i=−∞
∞

j=−∞
δ(x −i, y −j)
⎫
⎬
⎭
⎞
⎠
=
∞

i=−∞
F(u −i, v −j),
where we have written the Fourier transform of f(x, y) as F(u, v).
If the support of these shifted versions of the Fourier transform of the signal
does not intersect, we can easily reconstruct the signal from the sampled version.
We take the sampled signal, Fourier transform it, and cut out one copy of the
Fourier transform of the signal and Fourier transform this back (Figure 4.10).
However, if the support regions do overlap, we are not able to reconstruct the
signal because we can’t determine the Fourier transform of the signal in the regions
of overlap, where diﬀerent copies of the Fourier transform will add. This results in
a characteristic eﬀect, usually called aliasing, where high spatial frequencies appear
to be low spatial frequencies (see Figure 4.12 and exercises). Our argument also
yields Nyquist’s theorem: the sampling frequency must be at least twice the highest
frequency present for a signal to be reconstructed from a sampled version. By the
same argument, if we happen to have a signal that has frequencies present only in
the range [2k −1Ω, 2k+1Ω], then we can represent that signal exactly if we sample
at a frequency of at least 2Ω.
4.4.3 Smoothing and Resampling
Nyquist’s theorem means it is dangerous to shrink an image by simply taking every
kth pixel (as Figure 4.12 conﬁrms). Instead, we need to ﬁlter the image so that
spatial frequencies above the new sampling frequency are removed. We could do
this exactly by multiplying the image Fourier transform by a scaled 2D bar function,
which would act as a low-pass ﬁlter. Equivalently, we would convolve the image
with a kernel of the form (sin xsin y)/(xy). This is a diﬃcult and expensive (a polite
way of saying impossible) convolution because this function has inﬁnite support.
The most interesting case occurs when we want to halve the width and height
of the image. We assume that the sampled image has no aliasing (because if it
Section 4.4 Sampling and Aliasing 127
Signal Magnitude
Spectrum
Sampled
Signal Magnitude
Spectrum
Sample
Fourier
Transform
Fourier
Transform
Copy and
Shift
Magnitude
Spectrum
Cut out by
multiplication
with box filter
Inverse
Fourier
Transform
Accurately
Reconstructed
Signal
FIGURE 4.10: The Fourier transform of the sampled signal consists of a sum of copies
of the Fourier transform of the original signal, shifted with respect to each other by the
sampling frequency. Two possibilities occur. If the shifted copies do not intersect with
each other (as in this case), the original signal can be reconstructed from the sampled
signal (we just cut out one copy of the Fourier transform and inverse transform it). If
they do intersect (as in Figure 4.11), the intersection region is added, and so we cannot
obtain a separate copy of the Fourier transform, and the signal has aliased.
did, there would be nothing we could do about it anyway; once an image has been
sampled, any aliasing that is going to occur has happened, and there’s not much we
can do about it without an image model). This means that the Fourier transform
of the sampled image is going to consist of a set of copies of some Fourier transform,
with centers shifted to integer points in u, v space.
If we resample this signal, the copies now have centers on the half-integer
points in u, v space. This means that, to avoid aliasing, we need to apply a ﬁlter
that strongly reduces the content of the original Fourier transform outside the range
[u[ < 1/2, [v[ < 1/2. Of course, if we reduce the content of the signal inside this
range, we might lose information, too. Now the Fourier transform of a Gaussian is
a Gaussian, and Gaussians die away fairly quickly. Thus, if we were to convolve the
image with a Gaussian—or multiply its Fourier transform by a Gaussian, which is
Section 4.4 Sampling and Aliasing 128
Signal Magnitude
Spectrum
Sampled
Signal Magnitude
Spectrum
Sample
Fourier
Transform
Fourier
Transform
Copy and
Shift
Magnitude
Spectrum
Cut out by
multiplication
with box filter
Inverse
Fourier
Transform
Inaccurately
Reconstructed
Signal
FIGURE 4.11: The Fourier transform of the sampled signal consists of a sum of copies
of the Fourier transform of the original signal, shifted with respect to each other by the
sampling frequency. Two possibilities occur. If the shifted copies do not intersect with
each other (as in Figure 4.10), the original signal can be reconstructed from the sampled
signal (we just cut out one copy of the Fourier transform and inverse transform it). If they
do intersect (as in this ﬁgure), the intersection region is added, and so we cannot obtain a
separate copy of the Fourier transform, and the signal has aliased. This also explains the
tendency of high spatial frequencies to alias to lower spatial frequencies.
the same thing—we could achieve what we want.
The choice of Gaussian depends on the application. If σ is large, there is
less aliasing (because the value of the kernel outside our range is very small), but
information is lost because the kernel is not ﬂat within our range; similarly, if σ is
small, less information is lost within the range, but aliasing can be more substantial.
Figures 4.13 and 4.14 illustrate the eﬀects of diﬀerent choices of σ.
We have been using a Gaussian as a low-pass ﬁlter because its response at
high spatial frequencies is low and its response at low spatial frequencies is high.
In fact, the Gaussian is not a particularly good low-pass ﬁlter. What one wants
is a ﬁlter whose response is pretty close to constant for some range of low spatial
frequencies—the pass band—and whose response is also pretty close to zero—for
Section 4.4 Sampling and Aliasing 129
256x256 128x128 64x64 32x32 16x16
FIGURE 4.12: The top row shows sampled versions of an image of a grid obtained by
multiplying two sinusoids with linearly increasing frequency—one in x and one in y. The
other images in the series are obtained by resampling by factors of two without smoothing
(i.e., the next is a 128x128, then a 64x64, etc., all scaled to the same size). Note the
substantial aliasing; high spatial frequencies alias down to low spatial frequencies, and
the smallest image is an extremely poor representation of the large image. The bottom
row shows the magnitude of the Fourier transform of each image displayed as a log to
compress the intensity scale. The constant component is at the center. Notice that the
Fourier transform of a resampled image is obtained by scaling the Fourier transform of the
original image and then tiling the plane. Interference between copies of the original Fourier
transform means that we cannot recover its value at some points; this is the mechanism
underlying aliasing.
higher spatial frequencies—the stop band. It is possible to design low-pass ﬁlters
that are signiﬁcantly better than Gaussians. The design process involves a detailed
compromise between criteria of ripple—how ﬂat is the response in the pass band
and the stop band?—and roll-oﬀ—how quickly does the response fall to zero and
stay there? The basic steps for resampling an image are given in Algorithm 4.1.
Apply a low-pass ﬁlter to the original image
(a Gaussian with a σ of between one
and two pixels is usually an acceptable choice).
Create a new image whose dimensions on edge are half
those of the old image
Set the value of the i, jth pixel of the new image to the value
of the 2i, 2jth pixel of the ﬁltered image
Algorithm 4.1: Subsampling an Image by a Factor of Two.
Section 4.4 Sampling and Aliasing 130
256x256 128x128 64x64 32x32 16x16
FIGURE 4.13: Top: Resampled versions of the image of Figure 4.12, again by factors of
two, but this time each image is smoothed with a Gaussian of σ one pixel before resam-
pling. This ﬁlter is a low-pass ﬁlter, and so suppresses high spatial frequency components,
reducing aliasing. Bottom: The eﬀect of the low-pass ﬁlter is easily seen in these log-
magnitude images; the low-pass ﬁlter suppresses the high spatial frequency components
so that components interfere less, to reduce aliasing.
256x256 128x128 64x64 32x32 16x16
FIGURE 4.14: Top: Resampled versions of the image of Figure 4.12, again by factors
of two, but this time each image is smoothed with a Gaussian of σ two pixels before
resampling. This ﬁlter suppresses high spatial frequency components more aggressively
than that of Figure 4.13. Bottom: The eﬀect of the low-pass ﬁlter is easily seen in these
log-magnitude images; the low-pass ﬁlter suppresses the high spatial frequency components
so that components interfere less, to reduce aliasing.
Section 4.5 Filters as Templates 131
4.5 FILTERS AS TEMPLATES
It turns out that ﬁlters oﬀer a natural mechanism for ﬁnding simple patterns be-
cause ﬁlters respond most strongly to pattern elements that look like the ﬁlter. For
example, smoothed derivative ﬁlters are intended to give a strong response at a
point where the derivative is large. At these points, the kernel of the ﬁlter looks
like the eﬀect it is intended to detect. The x-derivative ﬁlters look like a verti-
cal light blob next to a vertical dark blob (an arrangement where there is a large
x-derivative), and so on.
FIGURE 4.15: Filter kernels look like the eﬀects they are intended to detect. On the left, a
smoothed derivative of Gaussian ﬁlter that looks for large changes in the x-direction (such
as a dark blob next to a light blob); on the right, a smoothed derivative of Gaussian ﬁlter
that looks for large changes in the y-direction.
It is generally the case that ﬁlters intended to give a strong response to a
pattern look like that pattern (Figure 4.15). This is a simple geometric result.
4.5.1 Convolution as a Dot Product
Recall from Section 4.1.1 that, for (, the kernel of some linear ﬁlter, the response
of this ﬁlter to an image 1 is given by
R
ij
=

u,v
G
i−u,j−v
H
uv
.
Now consider the response of a ﬁlter at the point where i and j are zero. This is
R =

u,v
G
−u,−v
H
u,v
.
This response is obtained by associating image elements with ﬁlter kernel
elements, multiplying the associated elements, and summing. We could scan the
image into a vector and the ﬁlter kernel into another vector in such a way that
Section 4.6 Technique: Normalized Correlation and Finding Patterns 132
associated elements are in the same component. By inserting zeros as needed, we
can ensure that these two vectors have the same dimension. Once this is done, the
process of multiplying associated elements and summing is precisely the same as
taking a dot product.
This is a powerful analogy because this dot product, like any other, achieves
its largest value when the vector representing the image is parallel to the vector
representing the ﬁlter kernel. This means that a ﬁlter responds most strongly when
it encounters an image pattern that looks like the ﬁlter. The response of a ﬁlter
gets stronger as a region gets brighter, too.
Now consider the response of the image to a ﬁlter at some other point. Nothing
signiﬁcant about our model has changed. Again, we can scan the image into one
vector and the ﬁlter kernel into another vector, such that associated elements lie
in the same components. Again, the result of applying this ﬁlter is a dot product.
There are two useful ways to think about this dot product.
4.5.2 Changing Basis
We can think of convolution as a dot product between the image and a diﬀerent
vector (because we have moved the ﬁlter kernel to lie over some other point in
the image). The new vector is obtained by rearranging the old one so that the
elements lie in the right components to make the sum work out. This means that,
by convolving an image with a ﬁlter, we are representing the image on a new basis
of the vector space of images—the basis given by the diﬀerent shifted versions of
the ﬁlter. The original basis elements were vectors with a zero in all slots except
one. The new basis elements are shifted versions of a single pattern.
For many of the kernels discussed, we expect that this process will lose
information—for the same reason that smoothing suppresses noise—so that the
coeﬃcients on this basis are redundant. This basis transformation is valuable in
texture analysis. Typically, we choose a basis that consists of small, useful pattern
components. Large values of the basis coeﬃcients suggest that a pattern compo-
nent is present, and texture can be represented by representing the relationships
between these pattern components, usually with some form of probability model.
4.6 TECHNIQUE: NORMALIZED CORRELATION AND FINDING PATTERNS
We can think of convolution as comparing a ﬁlter with a patch of image centered at
the point whose response we are looking at. In this view, the image neighborhood
corresponding to the ﬁlter kernel is scanned into a vector that is compared with the
ﬁlter kernel. By itself, this dot product is a poor way to ﬁnd features because the
value might be large simply because the image region is bright. By analogy with
vectors, we are interested in the cosine of the angle between the ﬁlter vector and
the image neighborhood vector; this suggests computing the root sum of squares of
the relevant image region (the image elements that would lie under the ﬁlter kernel)
and dividing the response by that value.
This yields a value that is large and positive when the image region looks like
the ﬁlter kernel, and small and negative when the image region looks like a contrast-
reversed version of the ﬁlter kernel. This value could be squared if contrast reversal
doesn’t matter. This is a cheap and eﬀective method for ﬁnding patterns, often
Section 4.6 Technique: Normalized Correlation and Finding Patterns 133
called normalized correlation.
4.6.1 Controlling the Television by Finding Hands by Normalized Correlation
It would be nice to have systems that could respond to human gestures. For ex-
ample, you might wave at the light to turn the room lights on, point at the air
conditioning to change the room temperature, or make an appropriate gesture at
an annoying politician on television to change the channel. In typical consumer
applications, there are quite strict limits to the amount of computation available,
meaning that it is essential that the gesture recognition system be simple. However,
such systems are usually quite limited in what they need to do, too.
Controlling the Television
Typically, a user interface is in some state—perhaps a menu is displayed—and
then an event occurs—perhaps a button is pressed on a remote control. This event
causes the interface to change state—a new menu item is highlighted, say—and the
whole process continues. In some states, some events cause the system to perform
some action, such as changing the channel. All this means that a state machine is
a natural model for a user interface.
One way for vision to ﬁt into this model is to provide events. This is good
because there are generally few diﬀerent kinds of event, and we know what kinds of
event the system should care about in any particular state. As a result, the vision
system needs to determine only whether either nothing or one of a small number of
known kinds of event has occurred. It is quite often possible to build systems that
meet these constraints.
A relatively small set of events is required to simulate a remote control; one
needs events that look like button presses (e.g., to turn the television on or oﬀ),
and events that look like pointer motion (e.g., to increase the volume; it is possible
to do this with buttons, too). With these events, the television can be turned on,
and an on-screen menu system can be navigated.
Finding Hands
Freeman et al. (1998) produced an interface where an open hand turns the
television on. This can be robust because all the system needs to do is determine
whether there is a hand in view. Furthermore, the user will cooperate by holding
the hand up and open. Because the user is expected to be a fairly constant distance
from the camera—so the size of the hand is roughly known, and there is no need
to search over scales—and in front of the television, the image region that needs to
be searched to determine whether there is a hand is quite small.
The hand is held up in a fairly standard conﬁguration and orientation to turn
the television set on, and it usually appears at about the same distance from the
television (so we know what it looks like). This means that a normalized correlation
score is suﬃcient to ﬁnd the hand. Any points in the correlation image where the
score is high enough correspond to hands. This approach can also be used to control
volume and so on. To do so, we need some notion of where the hand is going—to
one side turns the volume up, to the other turns it down—and this can be obtained
by comparing the position in the previous frame with that in the current frame.
Section 4.7 Technique: Scale and Image Pyramids 134
a
b c
d e
FIGURE 4.16: Examples of Freeman et al.’s system controlling a television set. Each state
is illustrated with what the television sees on the left and what the user sees on the right.
In (a), the television is asleep, but a process is watching the user. An open hand causes
the television to come on and show its user interface panel (b). Focus on the panel tracks
the movement of the user’s open hand in (c), and the user can change channels by using
this tracking to move an icon on the screen in (d). Finally, the user displays a closed hand
in (e) to turn oﬀ the set. This ﬁgure was originally published as Figure 12 of “Computer
Vision for Interactive Computer Graphics,” W. Freeman et al., IEEE Computer Graphics
and Applications, 1998 c _ IEEE, 1998.
The system displays an iconic representation of its interpretation of hand position
so the user has some feedback as to what the system is doing (Figure 4.16). Notice
that an attractive feature of this approach is that it could be self-calibrating. In
this approach, when you install your television set, you sit in front of it and show
it your hand a few times to allow it to get an estimate of the scale at which the
hand appears.
4.7 TECHNIQUE: SCALE AND IMAGE PYRAMIDS
Images look quite diﬀerent at diﬀerent scales. For example, the zebra’s muzzle in
Figure 4.17 can be described in terms of individual hairs—which might be coded in
terms of the response of oriented ﬁlters that operate at a scale of a small number of
pixels—or in terms of the stripes on the zebra. In the case of the zebra, we would
not want to apply large ﬁlters to ﬁnd the stripes. This is because these ﬁlters are
inclined to spurious precision—we don’t wish to represent the disposition of each
hair on the stripe—inconvenient to build, and slow to apply. A more practical
approach than applying large ﬁlters is to apply smaller ﬁlters to smoothed and
resampled versions of the image.
Section 4.7 Technique: Scale and Image Pyramids 135
512 256 128 64 32 16 8
FIGURE 4.17: A Gaussian pyramid of images running from 512x512 to 8x8. On the top
row, we have shown each image at the same size (so that some have bigger pixels than
others), and the lower part of the ﬁgure shows the images to scale. Notice that if we
convolve each image with a ﬁxed-size ﬁlter, it responds to quite diﬀerent phenomena. An
8x8 pixel block at the ﬁnest scale might contain a few hairs; at a coarser scale, it might
contain an entire stripe; and at the coarsest scale, it contains the animal’s muzzle.
4.7.1 The Gaussian Pyramid
An image pyramid is a collection of representations of an image. The name comes
from a visual analogy. Typically, each layer of the pyramid is half the width and
half the height of the previous layer; if we were to stack the layers on top of each
other, a pyramid would result. In a Gaussian pyramid, each layer is smoothed by
a symmetric Gaussian kernel and resampled to get the next layer (Figure 4.17).
These pyramids are most convenient if the image dimensions are a power of two or
a multiple of a power of two. The smallest image is the most heavily smoothed; the
layers are often referred to as coarse scale versions of the image.
With a little notation, we can write simple expressions for the layers of a
Gaussian pyramid. The operator S
↓
downsamples an image; in particular, the j,
kth element of S
↓
(J) is the 2j, 2kth element of J. The nth level of a pyramid P(J)
Section 4.7 Technique: Scale and Image Pyramids 136
is denoted P(J)
n
. With this notation, we have
P
Gaussian
(J)
n+1
= S
↓
(G
σ
∗ ∗P
Gaussian
(J)
n
)
= (S
↓
G
σ
)P
Gaussian
(J)
n
)
(where we have written G
σ
for the linear operator that takes an image to the
convolution of that image with a Gaussian). The ﬁnest scale layer is the original
image:
P
Gaussian
(J)
1
= J.
Set the ﬁnest scale layer to the image
For each layer, going from next to ﬁnest to coarsest
Obtain this layer by smoothing the next ﬁnest
layer with a Gaussian, and then subsampling it
end
Algorithm 4.2: Forming a Gaussian Pyramid.
4.7.2 Applications of Scaled Representations
Gaussian pyramids are useful because they make it possible to extract representa-
tions of diﬀerent types of structure in an image. We give three applications here;
in section 5.3.2, we describe another method that can be sped up using a Gaussian
pyramid.
Search over Scale
Numerous objects can be represented as small image patterns. A standard
example is a frontal view of a face. Typically, at low resolution, frontal views of
faces have a quite distinctive pattern: the eyes form dark pools, under a dark bar
(the eyebrows), separated by a lighter bar (specular reﬂections from the nose), and
above a dark bar (the mouth). There are various methods for ﬁnding faces that
exploit these properties (see Chapter 17.1.1). These methods all assume that the
face lies in a small range of scales. All other faces are found by searching a pyramid.
To ﬁnd bigger faces, we look at coarser scale layers, and to ﬁnd smaller faces we
look at ﬁner scale layers. This useful trick applies to many diﬀerent kinds of feature,
as we see in the chapters that follow.
Spatial Search
One application is spatial search, a common theme in computer vision. Typi-
cally, we have a point in one image and are trying to ﬁnd a point in a second image
that corresponds to it. This problem occurs in stereopsis—where the point has
moved because the two images are obtained from diﬀerent viewing positions—and
in motion analysis—where the image point has moved, either because the camera
moved or because it is on a moving object.
Searching for a match in the original pairs of images is ineﬃcient because we
might have to wade through a great deal of detail. A better approach, which is now
Section 4.8 Notes 137
pretty much universal, is to look for a match in a heavily smoothed and resampled
image and then reﬁne that match by looking at increasingly detailed versions of the
image. For example, we might reduce 1024 1024 images down to 4 4 versions,
match those, and then look at 8 8 versions (because we know a rough match, it
is easy to reﬁne it); we then look at 16 16 versions, and so on, all the way up
to 1024 1024. This gives an extremely eﬃcient search because a step of a single
pixel in the 4 4 version is equivalent to a step of 256 pixels in the 1024 1024
version. This strategy is known as coarse-to-ﬁne matching.
Feature Tracking
Most features found at coarse levels of smoothing are associated with large,
high-contrast image events because for a feature to be marked at a coarse scale, a
large pool of pixels need to agree that it is there. Typically, ﬁnding coarse-scale
phenomena misestimates both the size and location of a feature. For example,
a single pixel error in a coarse-scale image represents a multiple pixel error in a
ﬁne-scale image.
At ﬁne scales, there are many features, some of which are associated with
smaller, low-contrast events. One strategy for improving a set of features obtained
at a ﬁne scale is to track features across scales to a coarser scale and accept only the
ﬁne-scale features that have identiﬁable parents at a coarser scale. This strategy,
known as feature tracking in principle, can suppress features resulting from textured
regions (often referred to as noise) and features resulting from real noise.
4.8 NOTES
We don’t claim to be exhaustive in our treatment of linear systems, but it wouldn’t
be possible to read the literature on ﬁlters in vision without a grasp of the ideas in
this chapter. We have given a fairly straightforward account here; more details on
these topics can be found in the excellent books by Bracewell (1995), (2000).
Real Imaging Systems versus Shift Invariant Linear Systems
Imaging systems are only approximately linear. Film is not linear—it does not
respond to weak stimuli, and it saturates for bright stimuli—but one can usually
get away with a linear model within a reasonable range. CCD cameras are linear
within a working range. They give a small, but nonzero response to a zero input
as a result of thermal noise (which is why astronomers cool their cameras) and
they saturate for very bright stimuli. CCD cameras often contain electronics that
transforms their output to make them behave more like ﬁlm because consumers
are used to ﬁlm. Shift invariance is approximate as well because lenses tend to
distort responses near the image boundary. Some lenses—ﬁsh-eye lenses are a good
example—are not shift invariant.
Scale
There is a large body of work on scale space and scaled representations. The origins
appear to lie with Witkin (1983) and the idea was developed by Koenderink and
van Doorn (1986). Since then, a huge literature has sprung up (one might start
Section 4.8 Notes 138
with ter Haar Romeny et al. (1997) or Nielsen et al. (1999)). We have given only the
briefest picture here because the analysis tends to be quite tricky. The usefulness
of the techniques is currently hotly debated, too.
Image pyramids are useful. The next step from a Gaussian pyramid, which
is a highly redundant, is the Laplacian pyramid, originally described by Burt and
Adelson (1983). This is an eﬃcient representation. Instead of storing each image
level of a Gaussian pyramid, one stores the diﬀerence between the observed level
of the Gaussian pyramid and that predicted by the upsampling the coarser scale
level. Because coarse scale images are moderately good representations of ﬁner
scale images, this diﬀerence is small. As a result, there are numerous zeros in the
pyramid, and it is a convenient image code.
Anisotropic Scaling
One important diﬃculty with scale space models is that the symmetric Gaussian
smoothing process tends to blur out edges rather too aggressively for comfort. For
example, if we have two trees near one another on a skyline, the large-scale blobs
corresponding to each tree might start merging before all the small-scale blobs
have ﬁnished. This suggests that we should smooth diﬀerently at edge points than
at other points. For example, we might make an estimate of the magnitude and
orientation of the gradient. For large gradients, we would then use an oriented
smoothing operator that smoothed aggressively perpendicular to the gradient and
little along the gradient; for small gradients, we might use a symmetric smoothing
operator. This idea used to be known as edge-preserving smoothing.
In the modern, more formal version, due to Perona and Malik (1990b), we
notice the scale space representation family is a solution to the diﬀusion equation
∂Φ
∂σ
=
∂
2
Φ
∂x
2
+
∂
2
Φ
∂y
2
= ∇
2
Φ,
with the initial condition
Φ(x, y, 0) = J(x, y)
If this equation is modiﬁed to have the form
∂Φ
∂σ
= ∇ (c(x, y, σ)∇Φ)
= c(x, y, σ)∇
2
Φ + (∇c(x, y, σ)) (∇Φ)
with the same initial condition, then if c(x, y, σ) = 1, we have the diﬀusion equation
we started with, and if c(x, y, σ) = 0, there is no smoothing. We assume that c
does not depend on σ. If we knew where the edges were in the image, we could
construct a mask that consisted of regions where c(x, y) = 1, isolated by patches
along the edges where c(x, y) = 0; in this case, a solution would smooth inside each
separate region, but not over the edge. Although we do not know where the edges
are—the exercise would be empty if we did—we can obtain reasonable choices of
Section 4.8 Notes 139
c(x, y) from the magnitude of the image gradient. If the gradient is large, then c
should be small and vice versa. There is a substantial literature dealing with this
approach; a good place to start is ter Haar Romeny (1994).
PROBLEMS
4.1. Show that forming unweighted local averages, which yields an operation of the
form
1
ij
=
1
(2k + 1)
2
u=i+k

u=i−k
v=j+k

v=j−k
T
uv
,
is a convolution. What is the kernel of this convolution?
4.2. Write c
0
for an image that consists of all zeros with a single one at the center.
Show that convolving this image with the kernel
H
ij
=
1
2πσ
2
exp
_
−
((i −k −1)
2
+ (j −k −1)
2
)
2σ
2
_
(which is a discretised Gaussian) yields a circularly symmetric fuzzy blob.
4.3. Show that convolving an image with a discrete, separable 2D ﬁlter kernel is
equivalent to convolving with two 1D ﬁlter kernels. Estimate the number of
operations saved for an NxN image and a 2k + 1 2k + 1 kernel.
4.4. Show that convolving a function with a δ function simply reproduces the orig-
inal function. Now show that convolving a function with a shifted δ function
shifts the function.
4.5. We said that convolving the image with a kernel of the form (sin xsin y)/(xy)
is impossible because this function has inﬁnite support. Why would it be
impossible to Fourier transform the image, multiply the Fourier transform by
a box function, and then inverse-Fourier transform the result? (Hint: Think
support.)
4.6. Aliasing takes high spatial frequencies to low spatial frequencies. Explain why
the following eﬀects occur:
(a) In old cowboy ﬁlms that show wagons moving, the wheel often seems to be
stationary or moving in the wrong direction (i.e., the wagon moves from
left to right, and the wheel seems to be turning counterclockwise).
(b) White shirts with thin, dark pinstripes often generate a shimmering array
of colors on television.
(c) In ray-traced pictures, soft shadows generated by area sources look blocky.
PROGRAMMING EXERCISES
4.7. One way to obtain a Gaussian kernel is to convolve a constant kernel with
itself many times. Compare this strategy with evaluating a Gaussian kernel.
(a) How many repeated convolutions do you need to get a reasonable ap-
proximation? (You need to establish what a reasonable approximation is;
you might plot the quality of the approximation against the number of
repeated convolutions).
(b) Are there any beneﬁts that can be obtained like this? (Hint: Not every
computer comes with an FPU.)
4.8. Write a program that produces a Gaussian pyramid from an image.
4.9. A sampled Gaussian kernel must alias because the kernel contains components
at arbitrarily high spatial frequencies. Assume that the kernel is sampled on
Section 4.8 Notes 140
an inﬁnite grid. As the standard deviation gets smaller, the aliased energy
must increase. Plot the energy that aliases against the standard deviation of
the Gaussian kernel in pixels. Now assume that the Gaussian kernel is given
on a 7 7 grid. If the aliased energy must be of the same order of magnitude
as the error due to truncating the Gaussian, what is the smallest standard
deviation that can be expressed on this grid?
C H A P T E R 5
Local Image Features
An object is separated from its background in an image by an occluding contour.
Draw a path in the image that crosses such a contour. On one side, pixels lie
on the object, and on the other, the background. Finding occluding contours is
an important challenge, because the outline of an object—which is one cue to its
shape—is formed by occluding contours. We can expect that, at occluding contours,
there are often substantial changes in image brightness. There are other important
causes of sharp changes in image brightness, including sharp changes in albedo, in
surface orientation, or in illumination. Each can provide interesting information
about the objects in the world. Occluding contours carry shape information; sharp
changes in albedo carry texture information; sharp changes in surface orientation
tell us about shape; and illumination changes might tell us where the sun is. All
this means it is useful to ﬁnd and reason about sharp changes in image intensity.
Sharp changes in brightness cause large image gradients. Section 5.1 describes
methods to extract image gradients. One important use of gradients is to ﬁnd edges
or edge points, where the brightness changes particularly sharply (Section 5.2.1).
The edge points produced tend to be sensitive to changes in contrast (i.e., the size
of the diﬀerence in brightness across the edge), which can result from changes in
lighting. Often, it is helpful to use the orientation of the gradient vector (Sec-
tion 5.2.2), which does not depend on contrast. For example, at corners, the image
gradient vector swings sharply in orientation.
Corners are important, because they are easy to match from image to image.
At a corner, we expect to see strong image gradients that turn fast locally, and this
cue yields a corner detector (Section 5.3). If we can describe a neighborhood around
a corner, we can match descriptions across images. Such matching is an important
basic subroutine in computer vision. Applications include: estimating a homog-
raphy that will cause images to overlap (and so form a mosaic), Section 12.1.3;
estimating the fundamental matrix, Section 7.1; reconstructing points in 3D from
multiple views, Section 8.2.3; registering a 3D model with one or more images,
Chapter 19. We must ﬁrst ﬁnd a natural size for a neighborhood around a corner,
which we do by looking for the blob that best describes the local gray levels (Sec-
tion 5.3.2). Once we have that neighborhood, there are two natural constructions
that build representations of the orientation ﬁeld in the neighborhood; the resulting
features yield very well-behaved matchers (Section 5.4).
5.1 COMPUTING THE IMAGE GRADIENT
For an image J, the gradient is
∇J = (
∂J
∂x
,
∂J
∂y
)
T
,
141
Section 5.1 Computing the Image Gradient 142
which we could estimate by observing that
∂J
∂x
= lim
δx→0
J(x +δx, y) −J(x, y)
δx
≈ J
i+1,j
−J
i,j
.
By the same argument, ∂J/∂y ≈ J
i,j+1
−J
i,j
. These kinds of derivative estimates
are known as ﬁnite diﬀerences. Image noise tends to result in pixels not looking
like their neighbors, so that simple ﬁnite diﬀerences tend to give strong responses
to noise. As a result, just taking one ﬁnite diﬀerence for x and one for y gives noisy
gradient estimates. The way to deal with this problem is to smooth the image and
then diﬀerentiate it (we could also smooth the derivative).
The most usual noise model is the additive stationary Gaussian noise model,
where each pixel has added to it a value chosen independently from the same
Gaussian probability distribution. This distribution almost always has zero mean.
The standard deviation is a parameter of the model. The model is intended to
describe thermal noise in cameras and is illustrated in Figure 5.1.
Smoothing works because, in general, any image gradient of signiﬁcance to
us has eﬀects over a pool of pixels. For example, the contour of an object can
result in a long chain of points where the image derivative is large. As another
example, a corner typically involves many tens of pixels. If the noise at each pixel
is independent and additive, then large image derivatives caused by noise are a local
event. Smoothing the image before we diﬀerentiate will tend to suppress noise at
the scale of individual pixels, because it will tend to make pixels look like their
neighbors. However, gradients that are supported by evidence over multiple pixels
will tend not to be smoothed out. This suggests diﬀerentiating a smoothed image
(Figure 5.2).
5.1.1 Derivative of Gaussian Filters
Smoothing an image and then diﬀerentiating it is the same as convolving it with the
derivative of a smoothing kernel. This fact is most easily seen by thinking about
continuous convolution.
First, diﬀerentiation is linear and shift invariant. This means that there is
some kernel—we dodge the question of what it looks like—that diﬀerentiates. That
is, given a function I(x, y),
∂I
∂x
= K
(∂/∂x)
∗ ∗I.
Now we want the derivative of a smoothed function. We write the convolution
kernel for the smoothing as S. Recalling that convolution is associative, we have
(K
(∂/∂x)
∗ ∗(S ∗ ∗I)) = (K
(∂/∂x)
∗ ∗S) ∗ ∗I = (
∂S
∂x
) ∗ ∗I.
This fact appears in its most commonly used form when the smoothing function is
a Gaussian; we can then write
∂ (G
σ
∗ ∗I)
∂x
= (
∂G
σ
∂x
) ∗ ∗I,
that is, we need only convolve with the derivative of the Gaussian, rather than
convolve and then diﬀerentiate. As discussed in Section 4.5, smoothed derivative
Section 5.1 Computing the Image Gradient 143
FIGURE 5.1: The top row shows three realizations of a stationary additive Gaussian
noise process. We have added half the range of brightnesses to these images to show both
negative and positive values of noise. From left to right, the noise has standard deviation
1/256, 4/256, and 16/256 of the full range of brightness, respectively. This corresponds
roughly to bits zero, two, and ﬁve of a camera that has an output range of eight bits per
pixel. The lower row shows this noise added to an image. In each case, values below zero
or above the full range have been adjusted to zero or the maximum value accordingly.
ﬁlters look like the eﬀects they are intended to detect. The x-derivative ﬁlters look
like a vertical light blob next to a vertical dark blob (an arrangement where there
is a large x-derivative), and so on (Figure 4.15). Smoothing results in much smaller
noise responses from the derivative estimates (Figure 5.2).
The choice of σ used in estimating the derivative is often called the scale of
the smoothing. Scale has a substantial eﬀect on the response of a derivative ﬁlter.
Assume we have a narrow bar on a constant background, rather like the zebra’s
whisker. Smoothing on a scale smaller than the width of the bar means that the
ﬁlter responds on each side of the bar, and we are able to resolve the rising and
falling edges of the bar. If the ﬁlter width is much greater, the bar is smoothed into
the background and the bar generates little or no response (Figure 5.3).
Section 5.2 Representing the Image Gradient 144
d/dx
FIGURE 5.2: Derivative of Gaussian ﬁlters are less extroverted in their response to noise
than ﬁnite diﬀerence ﬁlters. The image at top left shows a detail from a picture of a
zebra; top center shows the same image corrupted by zero mean stationary additive
Gaussian noise, with σ = 0.03 (pixel values range from 0 to 1). Top right shows the
same image corrupted by zero mean stationary additive Gaussian noise, with σ = 0.09.
The second row shows the ﬁnite diﬀerence in the x-direction of each image. These images
are scaled so that zero is mid-gray, the most negative pixel is dark, and the most positive
pixel is light; we used a diﬀerent scaling for each image. Notice how the noise results in
occasional strong derivatives, shown by a graininess in the derivative maps for the noisy
images. The ﬁnal row shows the partial derivative in the x-direction of each image, in each
case estimated by a derivative of Gaussian ﬁlter with σ one pixel. Again, these images are
scaled so that zero is mid-gray, the most negative pixel is dark, and the most positive pixel
is light; we used a diﬀerent scaling for each image. The images are smaller than the input
image, because we used a 1313 pixel discrete kernel. This means that the six rows (resp.
columns) on the top and bottom of the image (resp. left and right) cannot be evaluated
exactly, because for these rows the kernel covers some points outside the image; we have
omitted these values. Notice how the smoothing helps reduce the impact of the noise; this
is emphasized by the detail images (between the second and ﬁnal row), which are doubled
in size. The details show patches that correspond from the ﬁnite diﬀerence image and the
smoothed derivative estimate. We show a derivative of Gaussian ﬁlter kernel, which (as
we expect) looks like the structure it is supposed to ﬁnd. This is not to scale (it’d be
extremely small if it were).
5.2 REPRESENTING THE IMAGE GRADIENT
There are two important representations of the image gradient. The ﬁrst is to
compute edges, where there are very fast changes in brightness. These are usually
seen as points where the magnitude of the gradient is extremal (Section 5.2.1). The
second is to use gradient orientations, which are largely independent of illumination
intensity (Section 5.7).
Section 5.2 Representing the Image Gradient 145
FIGURE 5.3: The scale (i.e., σ) of the Gaussian used in a derivative of Gaussian ﬁlter has
signiﬁcant eﬀects on the results. The three images show estimates of the derivative in the
x direction of an image of the head of a zebra obtained using a derivative of Gaussian
ﬁlter with σ one pixel, three pixels, and seven pixels (left to right). Note how images at
a ﬁner scale show some hair, the animal’s whiskers disappear at a medium scale, and the
ﬁne stripes at the top of the muzzle disappear at the coarser scale.
FIGURE 5.4: The gradient magnitude can be estimated by smoothing an image and then
diﬀerentiating it. This is equivalent to convolving with the derivative of a smoothing
kernel. The extent of the smoothing aﬀects the gradient magnitude; in this ﬁgure, we
show the gradient magnitude for the ﬁgure of a zebra at diﬀerent scales. At the center,
gradient magnitude estimated using the derivatives of a Gaussian with σ = 1 pixel; and on
the right, gradient magnitude estimated using the derivatives of a Gaussian with σ = 2
pixel. Notice that large values of the gradient magnitude form thick trails.
5.2.1 Gradient-Based Edge Detectors
We think of sharp changes in image intensity as lying on curves in the image, which
are known as edges; the curves are made up of edge points. Many eﬀects can cause
edges; worse, each eﬀect that can cause an edge is not guaranteed to cause an edge.
For example, an object may happen to be the same intensity as the background, and
so the occluding contour will not result in an edge. This means that interpreting
edge points can be very diﬃcult. Nonetheless, they are worth ﬁnding.
In the most common method for ﬁnding edges, we start by computing an
Section 5.2 Representing the Image Gradient 146
Form an estimate of the image gradient
Compute the gradient magnitude
While there are points with high gradient
magnitude that have not been visited
Find a start point that is a local maximum in the
direction perpendicular to the gradient
erasing points that have been checked
While possible, expand a chain through
the current point by:
1) predicting a set of next points, using
the direction perpendicular to the gradient
2) ﬁnding which (if any) is a local maximum
in the gradient direction
3) testing if the gradient magnitude at the
maximum is suﬃciently large
4) leaving a record that the point and
neighbors have been visited
record the next point, which becomes the current point
end
end
Algorithm 5.1: Gradient-Based Edge Detection.
estimate of the gradient magnitude. The gradient magnitude is large along a thick
trail in the image (Figure 5.4), but occluding contours are curves, so we must obtain
a curve of the most distinctive points on this trail.
There is clearly no objective deﬁnition, and we can proceed by reasonable
intuition. The gradient magnitude can be thought of as a chain of low hills. Marking
local maxima would mark isolated points—the hilltops in the analogy. A better
criterion is to slice the gradient magnitude along the gradient direction, which
should be perpendicular to the edge, and mark the points along the slice where the
magnitude is maximal. This would get a chain of points along the crown of the hills
in our chain. Each point in the chain can be used to predict the location of the next
point, which will be in a direction roughly at right angles to the gradient at the
edge point (Figure 5.5). Forming these chains is called nonmaximum suppression.
It is relatively straightforward to identify the location of these chains at a resolution
ﬁner than that of the pixel grid (Figure 5.5).
There are too many of these chains to come close to being a reasonable repre-
sentation of object boundaries. In part, this is because we have marked maxima of
the gradient magnitude without regard to how large these maxima are. It is more
usual to apply a threshold test to ensure that the maxima are greater than some
lower bound. This in turn leads to broken edge curves. The usual trick for dealing
with this is to use hysteresis; we have two thresholds and refer to the larger when
starting an edge chain and the smaller while following it. The trick often results in
an improvement in edge outputs. These considerations yield Algorithm 5.1. Most
Section 5.3 Finding Corners and Building Neighborhoods 147
Gradient
p
q
r
r
s
Gradient
FIGURE 5.5: Nonmaximum suppression obtains points where the gradient magnitude is
at a maximum along the direction of the gradient. The ﬁgure on the left shows how
we reconstruct the gradient magnitude. The dots are the pixel grid. We are at pixel q,
attempting to determine whether the gradient is at a maximum; the gradient direction
through q does not pass through any convenient pixels in the forward or backward direc-
tion, so we must interpolate to obtain the values of the gradient magnitude at p and r.
If the value at q is larger than both, q is an edge point. Typically, the magnitude values
are reconstructed with a linear interpolate, which in this case would use the pixels to the
left and right of p and r, respectively, to interpolate values at those points. On the right,
we sketch how to ﬁnd candidates for the next edge point given that q is an edge point;
an appropriate search direction is perpendicular to the gradient, so that points s and t
should be considered for the next edge point. Notice that, in principle, we don’t need to
restrict ourselves to pixel points on the image grid, because we know where the predicted
position lies between s and t. Hence, we could again interpolate to obtain gradient values
for points oﬀ the grid.
current edgeﬁnders follow these lines.
5.2.2 Orientations
As the light gets brighter or darker (or as the camera aperture opens or closes), the
image will get brighter or darker, which we can represent as a scaling of the image
value. The image J will be replaced with sJ for some value s. The magnitude of
the gradient scales with the image, i.e., [[ ∇J [[ will be replaced with s[[ ∇J [[. This
creates problems for edge detectors, because edge points may appear and disappear
as the image gradient values go above and below thresholds with the scaling. One
solution is to represent the orientation of image gradient, which is unaﬀected by
scaling (Figure 5.7). The gradient orientation ﬁeld depends on the smoothing scale
at which the gradient was computed. Orientation ﬁelds can be quite characteristic
of particular textures (Figure 5.9), and we will use this important property to come
up with more complex features below.
Section 5.3 Finding Corners and Building Neighborhoods 148
FIGURE 5.6: Edge points marked on the pixel grid for the image shown on the top. The
edge points on the left are obtained using a Gaussian smoothing ﬁlter at σ one pixel,
and gradient magnitude has been tested against a high threshold to determine whether
a point is an edge point. The edge points at the center are obtained using a Gaussian
smoothing ﬁlter at σ four pixels, and gradient magnitude has been tested against a high
threshold to determine whether a point is an edge point. The edge points on the right
are obtained using a Gaussian smoothing ﬁlter at σ four pixels, and gradient magnitude
has been tested against a low threshold to determine whether a point is an edge point.
At a ﬁne scale, ﬁne detail at high contrast generates edge points, which disappear at the
coarser scale. When the threshold is high, curves of edge points are often broken because
the gradient magnitude dips below the threshold; for the low threshold, a variety of new
edge points of dubious signiﬁcance are introduced.
5.3 FINDING CORNERS AND BUILDING NEIGHBORHOODS
Points worth matching are corners, because a corner can be localized, which means
we can tell where a corner is. This motivates the more general term interest point
often used to describe a corner. In this view, corners are interesting because we can
tell where they are. Place a small window over a patch of constant image value. If
you translate the window in any direction, the image in the window will not change
signiﬁcantly. This means you cannot give a reliable estimate of the location of the
window from its gray levels. Similarly, if you translate a window up and down an
edge, the image in the window doesn’t change, so you cannot estimate location
along the edge (this observation used to be known as the aperture problem). But
Section 5.3 Finding Corners and Building Neighborhoods 149
FIGURE 5.7: The magnitude of the image gradient changes when one increases or decreases
the intensity. The orientation of the image gradient does not change; we have plotted every
10th orientation arrow, to make the ﬁgure easier to read. Note how the directions of the
gradient arrows are ﬁxed, whereas the size changes. Philip Gatward c _Dorling Kindersley,
used with permission.
with a corner, any movement of the window changes the image in the window (i.e.,
the patch of image around the corner is not self-similar), so you can estimate the
location of the corner. Corners are not the only type of local image structure with
this property (Section 5.3.2)
There are many ways of representing a neighborhood around an interesting
corner. Methods vary depending on what might happen to the neighborhood. In
what follows, we will assume that neighborhoods are only translated, rotated, and
scaled (rather than, say, subjected to an aﬃne or projective transformation), and
so without loss of generality we can assume that the patches are circular. We
must estimate the radius of this circle. There is technical machinery available for
the neighborhoods that result from more complex transformations, but it is more
intricate; see Section 5.6.
5.3.1 Finding Corners
One way to ﬁnd corners is to ﬁnd edges, and then walk the edges looking for
a corner. This approach can work poorly, because edge detectors often fail at
corners. At sharp corners or unfortunately oriented corners, gradient estimates are
poor because the smoothing region covers the corner.
At a corner, we expect two important eﬀects. First, there should be large
gradients. Second, in a small neighborhood, the gradient orientation should swing
sharply. We can identify corners by looking at variations in orientation within a
Section 5.3 Finding Corners and Building Neighborhoods 150
5000
10000
15000
20000
25000
30
210
60
240
90
270
120
300
150
330
180 0
5000
10000
15000
30
210
60
240
90
270
120
300
150
330
180 0
FIGURE 5.8: The scale at which one takes the gradient aﬀects the orientation ﬁeld. We
show the overall trend of the orientation ﬁeld by plotting a rose plot, where the size of a
wedge represents the relative frequency of that range of orientations. Left shows an image
of artists pastels at a fairly ﬁne scale; here the edges are sharp, and so only a small set of
orientations occurs. In the heavily smoothed version on the right, all edges are blurred
and corners become smooth and blobby; as a result, more orientations appear in the rose
plot. Philip Gatward c _ Dorling Kindersley, used with permission.
window. In particular, the matrix
1 =

window
_
(∇I)(∇I)
T
_
≈

window
_
(
∂G
σ
∂x
∗ ∗J)(
∂G
σ
∂x
∗ ∗J) (
∂G
σ
∂x
∗ ∗J)(
∂G
σ
∂y
∗ ∗J)
(
∂G
σ
∂x
∗ ∗J)(
∂G
σ
∂y
∗ ∗J) (
∂G
σ
∂y
∗ ∗J)(
∂G
σ
∂y
∗ ∗J)
_
gives a good idea of the behavior of the orientation in a window. In a window of
constant gray level, both eigenvalues of this matrix are small because all the terms
are small. In an edge window, we expect to see one large eigenvalue associated with
gradients at the edge and one small eigenvalue because few gradients run in other
directions. But in a corner window, both eigenvalues should be large.
The Harris corner detector looks for local maxima of
det(1) −k(
trace(1)
2
)
2
where k is some constant (Harris and Stephens 1988); we used 0.5 for Figure 5.10.
These local maxima are then tested against a threshold. This tests whether the
product of the eigenvalues (which is det(1)) is larger than the square of the average
(which is (trace(1)/2)
2
). Large, locally maximal values of this test function imply
the eigenvalues are both big, which is what we want. Figure 5.10 illustrates corners
found with the Harris detector. This detector is unaﬀected by translation and
rotation (Figure 5.11).
Section 5.3 Finding Corners and Building Neighborhoods 151
1000
2000
3000
4000
5000
30
210
60
240
90
270
120
300
150
330
180 0
5000
10000
15000
30
210
60
240
90
270
120
300
150
330
180 0
5000
10000
15000
20000
25000
30
210
60
240
90
270
120
300
150
330
180 0
2000
4000
6000
8000
10000
30
210
60
240
90
270
120
300
150
330
180 0
FIGURE 5.9: Diﬀerent patterns have quite diﬀerent orientation histograms. The left shows
rose plots and images for a picture of artists pastels at two diﬀerent scales; the right shows
rose plots and images for a set of pastels arranged into a circular pattern. Notice how the
pattern of orientations at a particular scale, and also the changes across scales, are quite
diﬀerent for these two very diﬀerent patterns. Philip Gatward c _ Dorling Kindersley, used
with permission.
5.3.2 Using Scale and Orientation to Build a Neighborhood
To turn a corner into an image neighborhood, we must estimate the radius of
the circular patch (equivalently, its scale). The radius estimate should get larger
proportionally when the image gets bigger. For example, in a 2x scaled version
of the original image, our method should double its estimate of the patch radius.
This property helps choose a method. We could center a blob of ﬁxed appearance
(say, dark on a light background) on the corner, and then choose the scale to be
the radius of the best ﬁtting blob. An eﬃcient way to do this is to use a Laplacian
of Gaussian ﬁlter.
The Laplacian of a function in 2D is deﬁned as
(∇
2
f)(x, y) =
∂
2
f
∂x
2
+
∂
2
f
∂y
2
.
It is natural to smooth the image before applying a Laplacian. Notice that the
Laplacian is a linear operator (if you’re not sure about this, you should check),
meaning that we could represent taking the Laplacian as convolving the image
with some kernel (which we write as K
∇
2 ). Because convolution is associative, we
have that
(K
∇
2 ∗ ∗(G
σ
∗ ∗I)) = (K
∇
2 ∗ ∗G
σ
) ∗ ∗I = (∇
2
G
σ
) ∗ ∗I.
The reason this is important is that, just as for ﬁrst derivatives, smoothing an
image and then applying the Laplacian is the same as convolving the image with
the Laplacian of the kernel used for smoothing. Figure 5.12 shows the resulting
kernel for Gaussian smoothing; notice that this looks like a dark blob on a light
background.
Section 5.3 Finding Corners and Building Neighborhoods 152
FIGURE 5.10: The response of the Harris corner detector visualized for two detail regions
of an image of a box of colored pencils (center). Top left, a detail from the pencil points;
top center, the response of the Harris corner detector, where more positive values are
lighter. The top right shows these overlaid on the original image. To overlay this map,
we added the images, so that areas where the overlap is notably dark come from places
where the Harris statistic is negative (which means that one eigenvalue of ¹ is large, the
other small). Note that the detector is aﬀected by contrast, so that, for example, the point
of the mid-gray pencil at the top of this ﬁgure generates a very strong corner response, but
the points of the darker pencils do not, because they have little contrast with the tray. For
the darker pencils, the strong, contrasty corners occur where the lead of the pencil meets
the wood. The bottom sequence shows corners for a detail of pencil ends. Notice that
responses are quite local, and there are a relatively small number of very strong corners.
Steve Gorton c _ Dorling Kindersley, used with permission.
Imagine applying a smoothed Laplacian operator to the image at the center
of the patch. Write J for the image, ∇
2
σ
for the smoothed Laplacian operator with
smoothing constant σ, ↑
k
J for the the image with size scaled by k, (x
c
, y
c
) for
the coordinates of the patch center, and (x
kc
, y
kc
) for the coordinates of the patch
center in the scaled image. Assume that upscaling is perfect, and there are no
eﬀects resulting from the image grid. This is fair because eﬀects will be small for
the scales of interest for us. Then, we have
(∇
2
kσ
↑
k
J)(x
c
, y
c
) = (∇
2
σ
J)(x
kc
, y
kc
)
(this is most easily demonstrated by reasoning about the image as a continuous
function, the operator as a convolution, and then using the change of variables
formula for integrals). Now choose a radius r for the circular patch centered at
Section 5.3 Finding Corners and Building Neighborhoods 153
FIGURE 5.11: The response of the Harris corner detector is unaﬀected by rotation and
translation. The top row shows the response of the detector on a detail of the image on
the far left. The bottom row shows the response of the detector on a corresponding
detail from a rotated version of the image. For each row, we show the detail window
(left); the response of the Harris corner detector, where more positive values are lighter
(center); and the responses overlaid on the image (right). Notice that responses are quite
local, and there are a relatively small number of very strong corners. To overlay this map,
we added the images, so that areas where the overlap is notably dark come from places
where the Harris statistic is negative (which means that one eigenvalue of ¹ is large, the
other small). The arm and hammer in the top row match those in the bottom row; notice
how well the maps of Harris corner detector responses match, too. c _ Dorling Kindersley,
used with permission.
0 1 2 3 4 5 6 7 8 9 10
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
FIGURE 5.12: The scale of a neighborhood around a corner can be estimated by ﬁnding a
local extremum, in scale of the response at that point to a smoothed Laplacian of Gaussian
kernel. On the left, a detail of a piece of fencing. In the center, a corner identiﬁed by
an arrow (which points to the corner, given by a white spot surrounded by a black ring).
Overlaid on this image is a Laplacian of Gaussian kernel, in the top right corner; dark
values are negative, mid gray is zero, and light values are positive. Notice that, using the
reasoning of Section 4.5, this ﬁlter will give a strong positive response for a dark blob on a
light background, and a strong negative response for a light blob on a dark background, so
by searching for the strongest response at this point as a function of scale, we are looking
for the size of the best-ﬁtting blob. On the right, the response of a Laplacian of Gaussian
at the location of the corner, as a function of the smoothing parameter (which is plotted
in pixels). There is one extremal scale, at approximately 2 pixels. This means that there
is one scale at which the image neighborhood looks most like a blob (some corners have
more than one scale). c _ Dorling Kindersley, used with permission.
Section 5.3 Finding Corners and Building Neighborhoods 154
Assume a ﬁxed scale parameter k
Apply a corner detector to the image J
Initialize a list of patches
For each corner detected
Write (x
c
, y
c
) for the location of the corner
Compute the radius r for the patch at (x
c
, y
c
) as
r(x
c
, y
c
) =
argmax
σ
∇
2
σ
J(x
c
, y
c
)
by computing ∇
2
σ
J(x
c
, y
c
) for a variety of values of σ,
interpolating these values, and maximizing
Compute an orientation histogram H(θ) for gradient orientations within
a radius kr of (x
c
, y
c
).
Compute the orientation of the patch θ
p
as
θ
p
=
argmax
θ
H(θ). If there is more than
one theta that maximizes this histogram, make one copy of the
patch for each.
Attach (x
c
, y
c
, r, θ
p
) to the list of patches for each copy
Algorithm 5.2: Obtaining Location, Radius and Orientation of Pattern Elements Using
a Corner Detector.
(x
c
, y
c
), such that
r(x
c
, y
c
) =
argmax
σ
∇
2
σ
J(x
c
, y
c
)
(Figure 5.12). If the image is scaled by k, then this value of r will be scaled by
k too, which is the property we wanted. This procedure looks for the scale of the
best approximating blob. Notice that a Gaussian pyramid could be helpful here; we
could apply the same smoothed Laplacian operator to diﬀerent levels of a pyramid
to get estimates of the scale.
We can generalize this method, too, to detect interest points. Write (x, σ) for
a triple consisting of a point and a scale around that point. We would like to detect
such triples in a way that (a) when the image is translated, the triples translate,
too and (b) when the image is scaled, the triples scale. This can be given a formal
meaning. If J

(x) = J(λx+c) is a scaled and translated image, then for each point
(x, σ) in the list of neighborhoods for J, we want to have (λx + c, λσ) in the list
of neighborhoods for J

. This property is referred to as covariance (although the
term invariance is widely but incorrectly used).
We have already established that, at a particular point (given by our corner
detector), we get a covariant scale estimate by choosing the local maximum in
scale of the response of the Laplacian of Gaussian. We can build an interest point
detector directly out of a Laplacian of Gaussian, by identifying local extrema in
position and scale of the operator (if this looks slow to you, keep in mind that a
Gaussian pyramid could speed up the process). Each such extremum is a triple
(x, σ) with the properties we want. These points are diﬀerent from the points
Section 5.4 Describing Neighborhoods with SIFT and HOG Features 155
Assume a ﬁxed scale parameter k
Find all locations and scales which are local extrema of
∇
2
σ
J(x, y) in location (x, y) and scale σ forming a list of triples (x
c
, y
c
, r)
For each such triple
Compute an orientation histogram H(θ) for gradient orientations within
a radius kr of (x
c
, y
c
).
Compute the orientation of the patch θ
p
as
θ
p
=
argmax
θ
H(θ). If there is more than one θ that
maximizes this histogram, make one copy of the patch for each.
Attach (x
c
, y
c
, r, θ
p
) to the list of patches for each copy
Algorithm 5.3: Obtaining Location, Radius, and Orientation of Pattern Elements
Using the Laplacian of Gaussian.
obtained by using a corner detector and then estimating scale. Corner detectors
respond to a corner structure at the point of interest; the Laplacian of Gaussian
looks for structures that look like a circular blob of a particular scale centered at
the point of interest. Corner detectors tend to produce neighborhoods where the
estimate of the center is very accurate, but the scale estimate is poor. These are
most useful in matching problems where we don’t expect the scale to change much.
Laplacian of Gaussian methods produce neighborhoods where the estimate of the
center is less accurate, but the scale estimate is better. These are most useful in
matching problems where large changes of scale might appear.
As we have seen, orientation histograms are a natural representation of im-
age patches. However, we cannot represent orientations in image coordinates (for
example, using the angle to the horizontal image axis), because the patch we are
matching to might have been rotated. We need a reference orientation so all angles
can be measured with respect to that reference. A natural reference orientation is
the most common orientation in the patch. We compute a histogram of the gradi-
ent orientations in this patch, and ﬁnd the largest peak. This peak is the reference
orientation for the patch. If there are two or more peaks of the same magnitude, we
make multiple copies of the patch, one at each peak orientation. The whole process
is summarized in Algorithms 5.2 and 5.3. These estimates of patch neighborhoods
are remarkably well behaved (Figure 5.13).
5.4 DESCRIBING NEIGHBORHOODS WITH SIFT AND HOG FEATURES
We know the center, radius, and orientation of a set of an image patch, and must
now represent it. Orientations should provide a good representation. They are
unaﬀected by changes in image brightness, and diﬀerent textures tend to have
diﬀerent orientation ﬁelds. The pattern of orientations in diﬀerent parts of the
patch is likely to be quite distinctive. Our representation should be robust to small
errors in the center, radius, or orientation of the patch, because we are unlikely to
estimate these exactly right.
Section 5.4 Describing Neighborhoods with SIFT and HOG Features 156
FIGURE 5.13: This ﬁgure shows local patches recovered using a method similar to that
described in the text (the details of the corner detector were diﬀerent). These patches
are plotted as squares, rather than as circles. The location of the patch is the center of
the square. The reference orientation of the patch is given by the line segment in the
square, and the scale is the size of the square. The image on the right has been scaled,
rotated, and translated to produce the image on the left. Notice that (a) most of the
patches on the right have corresponding patches on the left and (b) the corresponding
patches are translated, rotated, and scaled versions of the original patches. You can check
this by looking at the grayscale version of the image. We have shown some of the many
corresponding pairs of patches (below; the large white arrows). This ﬁgure was originally
published as Figure 1 of “Object recognition from local scale-invariant features” D.G. Lowe,
Proc. IEEE ICCV, 1999 c _ IEEE 1999.
You should think of these neighborhoods as being made up of pattern ele-
ments. In this case, the pattern elements will be orientations, but we will use this
trick again for other kinds of pattern element. These elements move around some-
what inside the neighborhood (because we might not get the center right), but if
most elements are there and are in about the right place, then the neighborhood
has the right properties. We must build features that can make it obvious whether
the pattern elements are present, and whether they are in about the right place,
but are not aﬀected by some rearrangement.
The most obvious approach is to represent the neighborhood with a histogram
of the elements that appear there. This will tell us what is present, but it confuses
too many patterns with one another. For example, all neighborhoods with vertical
stripes will get mixed up, however wide the stripe. The natural approach is to take
histograms locally, within subpatches of the neighborhood. This leads to a very
Section 5.4 Describing Neighborhoods with SIFT and HOG Features 157
N
o
r
m
a
l
i
z
e
,
t
h
r
e
s
h
o
l
d
,
t
h
e
n
r
e
n
o
r
m
a
l
i
z
e
Neighborhood
Grid
Subgrid
Subgrid
element
Gradient
estimate
Grid element
histograms
FIGURE 5.14: To construct a SIFT descriptor for a neighborhood, we place a grid over
the rectiﬁed neighborhood. Each grid is divided into a subgrid, and a gradient estimate
is computed at the center of each subgrid element. This gradient estimate is a weighted
average of nearby gradients, with weights chosen so that gradients outside the subgrid
cell contribute. The gradient estimates in each subgrid element are accumulated into
an orientation histogram. Each gradient votes for its orientation, with a vote weighted
by its magnitude and by its distance to the center of the neighborhood. The resulting
orientation histograms are stacked to give a single feature vector. This is normalized to
have unit norm; then terms in the normalized feature vector are thresholded, and the
vector is normalized again.
important feature construction.
5.4.1 SIFT Features
We can now compute a representation that is not aﬀected by translation, rotation,
or scale. For each patch, we rectify the patch by translating the center to the
origin, rotating so the orientation direction lies along (say) the x-axis, and scaling
so the radius is one. Any representation we compute for this rectiﬁed patch will be
invariant to translations, rotations, and scale. Although we do not need to rectify
in practice—instead, we can work the rectiﬁcation into each step of computing the
description—it helps to think about computing descriptions for a rectiﬁed patch.
A SIFT descriptor (for Scale Invariant Feature Transform) is constructed
out of image gradients, and uses both magnitude and orientation. The descriptor is
normalized to suppress the eﬀects of change in illumination intensity. The descriptor
is a set of histograms of image gradients that are then normalized. These histograms
expose general spatial trends in the image gradients in the patch but suppress detail.
For example, if we estimate the center, scale, or orientation of the patch slightly
wrong, then the rectiﬁed patch will shift slightly. As a result, simply recording the
gradient at each point yields a representation that changes between instances of
the patch. A histogram of gradients will be robust to these changes. Rather than
histogramming the gradient at a set of sample points, we histogram local averages
of image gradients; this helps avoid noise.
The standard SIFT descriptor is obtained by ﬁrst dividing the rectiﬁed patch
Section 5.4 Describing Neighborhoods with SIFT and HOG Features 158
into an n n grid. We then subdivide each grid element into an m m subgrid
of subcells. At the center of each subcell, we compute a gradient estimate. The
gradient estimate is obtained as a weighted average of gradients around the center
of the cell, weighting each by (1 −d
x
/s
x
)(1 −d
y
/s
y
)/N, where d
x
(resp. d
y
) is the
x (resp. y) distance from the gradient to the center of the subcell, and s
x
(resp. s
y
)
is the x (resp. y) spacing between the subcell centers. This means that gradients
make contributions to more than one subcell, so that a small error in the location
of the center of the patch leads to a small change in the descriptor.
We now use these gradient estimates to produce histograms. Each grid ele-
ment has a q-cell orientation histogram. The magnitude of each gradient estimate
is accumulated into the histogram cell corresponding to its orientation; the magni-
tude is weighted by a Gaussian in distance from the center of the patch, using a
standard deviation of half the patch.
We concatenate each histogram into a vector of nnq entries. If the image
intensity were doubled, this vector’s length would double (because the histogram
entries are sums of gradient magnitudes). To avoid this eﬀect, we normalize this
vector to have unit length. Very large gradient magnitude estimates tend to be
unstable (for example, they might result from a lucky arrangement of surfaces in
3D so that one faces the light directly and another points away from the light).
This means that large entries in the normalized vector are untrustworthy. To avoid
diﬃculties with large gradient magnitudes, each value in the normalized vector is
thresholded with threshold t, and the resulting vector is renormalized. The whole
process is summarized in Algorithm 5.4 and Figure 5.14. Standard parameter values
are n = 4, m = 4, q = 8, and t = 0.2.
Given an image J, and a patch with center (x
c
, y
c
),
radius r, orientation θ, and parameters n, m, q, k and t.
For each element of the n n grid centered at (x
c
, y
c
) with spacing kr
Compute a weighted q element histogram of the averaged
gradient samples at each point of the mm subgrid,
as in Algorithm 5.5.
Form an n n q vector v by concatenating the histograms.
Compute u = v/
√
v v.
Form w whose i’th element w
i
is min(u
i
, t).
The descriptor is d = w/
√
w w.
Algorithm 5.4: Computing a SIFT Descriptor in a Patch Using Location, Orientation
and Scale.
There is now extensive experimental evidence that image patches that match
one another will have similar SIFT feature representations, and patches that do
not will tend not to. SIFT features can be used to represent the local color pattern
around a sample point, too. The natural procedure is to apply SIFT feature code
to a color representation. For example, one could compute SIFT features for each
of the hue, saturation, and value channels (HSV-SIFT; see Bosch et al. (2008)); for
the opponent color channels (OpponentSIFT, which uses R-G and B-Y; see van de
Section 5.4 Describing Neighborhoods with SIFT and HOG Features 159
Given a grid cell ( for patch with center c = (x
c
, y
c
) and radius r
Create an orientation histogram
For each point p in an mm subgrid spanning (
Compute a gradient estimate ∇J [
p
estimate at p
as a weighted average of ∇J, using bilinear weights centered at p.
Add a vote with weight [[ ∇J [[
1
r
√
2π
exp
_
−
||p−c||
2
r
2
_
to the orientation histogram cell for the orientation of ∇J.
Algorithm 5.5: Computing a Weighted q Element Histogram for a SIFT Feature.
FIGURE 5.15: The HOG features for each the two images shown here have been visualized
by a version of the rose diagram of Figures 5.7–5.9. Here each of the cells in which the
histogram is taken is plotted with a little rose in it; the direction plotted is at right angles
to the gradient, so you should visualize the overlaid line segments as edge directions.
Notice that in the textured regions the edge directions are fairly uniformly distributed,
but strong contours (the gardener, the fence on the left; the vertical edges of the french
windows on the right) are very clear. This ﬁgure was plotted using the toolbox of Doll´ar
and Rabaud. Left: c _ Dorling Kindersley, used with permission. Right: Geoﬀ Brightling
c _ Dorling Kindersley, used with permission.
Sande et al. (2010)); for normalised opponent color channels (C-SIFT, which uses
(R − G)/(R + G + B) and (B − Y )/(R + G + B); see Abdel Hakim and Farag
(2006); Geusebroek et al. (2001); or Burghouts and Geusebroek (2009)); and for
normalized color channels (rgSIFT, which uses R/(R+G+B) and G/(R+G+B);
see van de Sande et al. (2010)). Each of these features will behave slightly diﬀerently
when the light falling on an object changes, and each can be used in place of, or in
addition to, SIFT features.
5.4.2 HOG Features
The HOG feature (for Histogram Of Gradient orientations) is an important variant
of the SIFT feature. Again, we histogram gradient orientations in cells, but now
adjust the process to try and identify high-contrast edges. We can recover con-
trast information by counting gradient orientations with weights that reﬂect how
signiﬁcant a gradient is compared to other gradients in the same cell. This means
that, rather than normalize gradient contributions over the whole neighborhood,
we normalize with respect to nearby gradients only. Normalization could occur on
a grid of cells that is diﬀerent from the orientation subgrid, too. A single gradient
Section 5.5 Computing Local Features in Practice 160
location might contribute to several diﬀerent histograms, normalized in somewhat
diﬀerent ways; this means we will be relatively unlikely to miss boundaries that
have low contrast.
Write [[ ∇I
x
[[ for the gradient magnitude at point x in the image. Write ( for
the cell whose histogram we wish to compute and w
x,C
for the weight that we will
use for the orientation at x for this cell. A natural choice of weight is
w
x,C
=
[[ ∇I
x
[[

u∈C
[[ ∇I
u
[[
.
This compares the gradient magnitude to others in the cell, so that gradients that
are large compared to their neighbors get a large weight. This normalization process
means that HOG features are quite good at picking outline curves out of confusing
backgrounds (Figure 5.15).
5.5 COMPUTING LOCAL FEATURES IN PRACTICE
We have sketched the most important feature constructions, but there is a huge
range of variants. Performance is aﬀected by quite detailed questions, such as the
extent of smoothing when evaluating orientations. Space doesn’t allow a detailed
survey of these questions (though there’s some material in Section 5.6), and the
answers seem to change fairly frequently, too. This means we simply can’t supply
accurate recipes for building each of these features.
Fortunately, at time of writing, there are several software packages that pro-
vide good implementations of each of these feature types, and of other variations.
Piotr Doll´ ar and Vincent Rabaud publish a toolbox at http://vision.ucsd.
edu/
~
pdollar/toolbox/doc/index.html; we used this to generate several ﬁgures.
VLFeat is a comprehensive open-source package that provides SIFT features, vec-
tor quantization by a variety of methods, and a variety of other representations.
At time of writing, it could be obtained from http://www.vlfeat.org/. SIFT
features are patented (Lowe 2004), but David Lowe (the inventor) provides a refer-
ence object code implementation at http://www.cs.ubc.ca/
~
lowe/keypoints/.
Navneet Dalal, one of the authors of the original HOG feature paper, provides
an implementation at http://www.navneetdalal.com/software/. One variant
of SIFT is PCA-SIFT, where one uses principal components to reduce the di-
mension of the SIFT representation (Ke and Sukthankar 2004). Yan Ke, one
of the authors of the original PCA-SIFT paper, provides an implementation at
http://www.cs.cmu.edu/
~
yke/pcasift/. Color descriptor code, which computes
visual words based on various color SIFT features, is published by van de Sande et
al. at http://koen.me/research/colordescriptors/.
5.6 NOTES
Edges
There is a huge edge detection literature. The earliest paper of which we are aware
is Julez (1959) (yes, 1959!). Those wishing to be acquainted with the early literature
in detail should start with a 1975 survey by Davis (1975); Herskovits and Binford
(1970); Horn (1971); and Hueckel (1971), who models edges and then detects the
model. There are many optimality criteria for edge detectors, and rather more
Section 5.6 Notes 161
“optimal” edge detectors. The key paper in this literature is by Canny (1986);
signiﬁcant variants are due to Deriche (1987) and to Spacek (1986). Faugeras’
textbook contains a detailed and accessible exposition of the main issues Faugeras
(1993). At the end of the day, most variants boil down to smoothing the image
with something that looks a lot like a Gaussian before measuring the gradient. All
edge detectors behave badly at corners; only the details vary.
Object boundaries are not the same as sharp changes in image values. There
is a vast literature seeking to build boundary detectors; we can provide only some
pointers. The reader could start with Bergholm (1987), Deriche (1990), Elder and
Zucker (1998), Fleck (1992), Kube and Perona (1996), Olson (1998), Perona and
Malik (1990b), or Torre and Poggio (1986). The best current boundary detector
takes quite a lot of local information into account, and is described in Section 17.1.3.
The edges that our edge detectors respond to are sometimes called step edges
because they consist of a sharp, “discontinuous” change in value that is sometimes
modeled as a step. A variety of other forms of edge have been studied. The
most commonly cited example is the roof edge, which consists of a rising segment
meeting a falling segment, rather like some of the reﬂexes that can result from the
eﬀects of interreﬂections. Another example that also results from interreﬂections
is a composite of a step and a roof. It is possible to ﬁnd these phenomena by
using essentially the same steps as outlined before (ﬁnd an “optimal” ﬁlter, and do
nonmaximum suppression on its outputs) (Canny 1986, Perona and Malik 1990a).
In practice, this is seldom done. There appear to be two reasons. First, there is no
comfortable basis in theory (or practice) for the models that are adopted. What
particular composite edges are worth looking for? The easy answer—those for which
optimal ﬁlters are reasonably easy to derive—is most unsatisfactory. Second, the
semantics of roof edges and more complex composite edges is even vaguer than that
of step edges. There is little notion of what one would do with roof edge once it
had been found.
Corners, Neighborhoods, and Interest Points
The ﬁrst corner detector we know of is due to Moravec (1980). Corner detectors
are now very well studied (there is an excellent Wikipedia page that describes the
various detectors and their relations at http://en.wikipedia.org/wiki/Corner_
detection). The Harris and Stephens detector we described remains competitive.
Important variants look at diﬀerent eigenvalue criteria (Tomasi and Shi 1994);
diﬀerential geometric criteria (Wang and Brady 1994); multiple scales (Lindeberg
1993); local self-similarity measures (Smith and Brady 1997, Trajkovic and Hedley
1998); and machine learning (Rosten et al. 2010).
For simplicity of exposition, we have elided corners and interest points (the
other name under which corners are often studied). Interest points are usually
thought of as a corner (or something like it) together with a neighborhood, covariant
under some form of transformation. We like to see detecting the points and estimat-
ing their neighborhoods as distinct processes, though for strict covariance both the
detector and the neighborhood estimator must be covariant. Various detectors are
scale covariant (Mikolajczyk and Schmid 2002); aﬃne covariant (Mikolajczyk and
Schmid 2002); and illumination robust (Gevrekci and Gunturk 2009). The idea can
Section 5.6 Notes 162
be extended to spatio-temporal representations (Willems et al. 2008, Laptev 2005).
There are now detailed experimental studies of the performance of interest point
detectors (Schmid et al. 2000, Privitera and Stark 1998, Mikolajczyk et al. 2005).
Descriptors
The tricks to describing neighborhoods seem to be: describe a local texture pattern
within a covariant neighborhood; work with orientations, because they’re illumi-
nation invariant; and use histograms to suppress spatial detail, working with more
detail at the center than at the boundary. These tricks appear in numerous papers
in a variety of forms (e.g., Schmid and Mohr (1997); Belongie et al. (2001); Berg et
al. (2005)), but SIFT and Hog features now dominate. Comparisons between local
descriptors seem to support this dominance (Mikolajczyk and Schmid 2005).
PROBLEMS
5.1. Each pixel value in 500 500 pixel image 1 is an independent, normally dis-
tributed random variable with zero mean and standard deviation one. Esti-
mate the number of pixels that, where the absolute value of the x derivative,
estimated by forward diﬀerences (i.e., [I
i+1,j
−I
i,j
[, is greater than 3.
5.2. Each pixel value in 500 500 pixel image 1 is an independent, normally dis-
tributed random variable with zero mean and standard deviation one. 1 is
convolved with the 2k + 1 2k + 1 kernel (. What is the covariance of pixel
values in the result? There are two ways to do this; on a case-by-case basis
(e.g., at points that are greater than 2k+1 apart in either the x or y direction,
the values are clearly independent) or in one fell swoop. Don’t worry about
the pixel values at the boundary.
5.3. We have a camera that can produce output values that are integers in the range
from 0 to 255. Its spatial resolution is 1024 by 768 pixels, and it produces 30
frames a second. We point it at a scene that, in the absence of noise, would
produce the constant value 128. The output of the camera is subject to noise
that we model as zero mean stationary additive Gaussian noise with a standard
deviation of 1. How long must we wait before the noise model predicts that
we should see a pixel with a negative value? (Hint: You may ﬁnd it helpful
to use logarithms to compute the answer as a straightforward evaluation of
exp(−128
2
/2) will yield 0; the trick is to get the large positive and large
negative logarithms to cancel.)
5.4. Show that for a 2 2 matrix ¹, with eigenvalues λ
1
, λ
2
(a) det¹ = λ
1
λ
2
(b) trace¹ = λ
1
+λ
2
PROGRAMMING EXERCISES
5.5. The Laplacian of a Gaussian looks similar to the diﬀerence between two Gaus-
sians at diﬀerent scales. Compare these two kernels for various values of the
two scales. Which choices give a good approximation? How signiﬁcant is the
approximation error in edge ﬁnding using a zero-crossing approach?
5.6. Obtain an implementation of Canny’s edge detector (you could try the vision
home page; MATLAB also has an implementation in the image processing
toolbox), and make a series of images indicating the eﬀects of scale and contrast
thresholds on the edges that are detected. How easy is it to set up the edge
detector to mark only object boundaries? Can you think of applications where
Section 5.6 Notes 163
this would be easy?
5.7. It is quite easy to defeat hysteresis in edge detectors that implement it; essen-
tially, one sets the lower and higher thresholds to have the same value. Use this
trick to compare the behavior of an edge detector with and without hysteresis.
There are a variety of issues to look at:
(a) What are you trying to do with the edge detector output? It is some-
times helpful to have linked chains of edge points. Does hysteresis help
signiﬁcantly here?
(b) Noise suppression: We often wish to force edge detectors to ignore some
edge points and mark others. One diagnostic that an edge is useful is high
contrast (it is by no means reliable). How reliably can you use hysteresis
to suppress low-contrast edges without breaking high-contrast edges?
5.8. Build a Harris corner detector; for each corner, estimate scale and orientation
as we have described. Now test how well your list of neighborhoods behaves
under rotation, translation, and scale of the image. You can do this by a simple
exercise in matching. For each test image, prepare a rotated, translated, and
scaled version of that image. Now you know where each neighborhood should
appear in the new version of the image — check how often something of the
right size and orientation appears in the right place. You should ﬁnd that
rotation and translation cause no signiﬁcant problems, but large scale changes
can be an issue.
C H A P T E R 6
Texture
Texture is a phenomenon that is widespread, easy to recognise, and hard to deﬁne.
Typically, whether an eﬀect is referred to as texture or not depends on the scale
at which it is viewed. A leaf that occupies most of an image is an object, but the
foliage of a tree is a texture. Views of large numbers of small objects are often best
thought of as textures. Examples include grass, foliage, brush, pebbles, and hair.
Many surfaces are marked with orderly patterns that look like large numbers of
small objects. Examples include the spots of animals such as leopards or cheetahs;
the stripes of animals such as tigers or zebras; the patterns on bark, wood, and
skin. Textures tend to show repetition: (roughly!) the same local patch appears
again and again, though it may be distorted by a viewing transformation.
Texture is important, because texture appears to be a very strong cue to
object identity. Most modern object recognition programs are built around texture
representation machinery of one form or another. This may be because texture
is also a strong cue to material properties: what the material that makes up an
object is like. For example, texture cues can be used to tell tree bark (which is
moderately hard and rough) from bare metal (which is hard, smooth, and shiny).
People seem to be able to predict some mechanical properties of materials from their
appearance. For example, often you can distinguish somewhat viscous materials,
like hand cream, from highly viscous materials, like cream cheese, by eye (Adelson
2001). Material properties are correlated to the identity of objects, but they are
not the same thing. For example, although hammers are commonly made of metal,
a plastic hammer, a metal hammer, and a wooden hammer are all still hammers.
There are three main kinds of texture representation. Local texture represen-
tations encode the texture very close to a point in an image. These representations
can’t be comprehensive, because they look at a small piece of the image. However,
they are very useful in image segmentation, where we must break an image into
large, useful components, usually called regions (the details of what makes a re-
gion useful are deferred to Chapter 9). One reasonable requirement is that points
inside a region look similar to one another, and diﬀerent from points outside the
region, and segmentation algorithms need a description of the appearance close to
the point to impose this requirement. Local texture representations are described
in Section 6.1.
Other problems require a description of the texture within an image domain.
We refer to such representations as pooled texture representations. For example,
texture recognition is the problem of determining what texture is represented
by a patch in an image. Here we have a domain (the patch) and we want a repre-
sentation of the overall texture in the domain. Similarly, in material recognition,
one must decide what material is represented by a patch in the image. Section 6.2
describes methods for building pooled texture representations.
Data-driven texture representations model a texture by a procedure that can
164
165
FIGURE 6.1: Although texture is diﬃcult to deﬁne, it has some important and valuable
properties. In this image, there are many repeated elements (some leaves form repeated
“spots”; others, and branches, form “bars” at various scales; and so on). Our perception
of the material is quite intimately related to the texture (what would the surface feel like
if you ran your ﬁngers over it? what is soggy? what is prickly? what is smooth?). Notice
how much information you are getting about the type of plants, their shape, the shape
of free space, and so on, from the textures. Geoﬀ Brightling c _ Dorling Kindersley, used
with permission.
generate a textured region from an example. These representations are not appro-
priate for segmentation or recognition applications, but are tremendously valuable
for texture synthesis. In this problem, we must create regions of texture, for exam-
ple, to ﬁll holes in images (Section 6.3).
The texture on a surface can be a strong cue to its shape. If the texture is
“the same” over the surface, then deformation of the texture from point to point
can be a cue to the shape of the surface. For example, if we have a perspective view
of an inclined plane with spots on it, the spots will be smaller closer to the horizon
in the image. This can be used to recover the inclination of the plane. Similarly,
on a curved surface, the foreshortening of texture elements gives some information
about the local inclination of the surface. Recovering surface orientation or surface
shape from an image texture is known as shape from texture; solutions to this
Section 6.1 Local Texture Representations Using Filters 166
Fabric
Stone
FIGURE 6.2: Typically, diﬀerent materials display diﬀerent image textures. These are
example images from a collection of 1,000 material images, described in by Sharan et al.
(2009); there are 100 images in each of the ten categories, including the two categories
shown here (fabric and stone). Notice how (a) the textures vary widely, even within a
material category; and (b) diﬀerent materials seem to display quite diﬀerent textures.
This ﬁgure shows elements of a database collected by C. Liu, L. Sharan, E. Adelson, and
R. Rosenholtz, and published at http: // people. csail. mit. edu/ lavanya/ research_
sharan. html . Figure by kind permission of the collectors.
problem tend to use straightforward representations of texture together with strong
constraints on the overall structure of the texture (Section 6.5).
6.1 LOCAL TEXTURE REPRESENTATIONS USING FILTERS
Image textures generally consist of repeated elements; an element is sometimes
called a texton. For example, some of the fabric textures in Figure 6.2 consist of
triangles of wool formed by the knit pattern. Similarly, some stone textures in that
ﬁgure consist of numerous, near-circular, gray blobs. It is natural to represent a
texture with some description of (a) what the textons are and (b) how they repeat.
Notice that it is diﬃcult to be precise about what a texton is, because if a large
pattern repeats frequently, then so do its parts. This presents no major problems,
because we do not need to extract textons accurately. Instead, what we need are
representations that diﬀer in ways that are easy to observe when two textures are
signiﬁcantly diﬀerent. We can do this by assuming that all textons are made of
Section 6.1 Local Texture Representations Using Filters 167
Image
Filter responses
at various orientations
and scales
Rectified images
Summaries
Summarize
Rectify
FIGURE 6.3: Local texture representations can be obtained by ﬁltering an image with a
set of ﬁlters at various scales, and then preparing a summary. Summaries ensure that, at
a pixel, we have a representation of what texture appears near that pixel. The ﬁlters are
typically spots and bars (see Figure 6.4). Filter outputs can be enhanced by rectifying
them (so that positive and negative responses do not cancel), then computing a local
summary of the rectiﬁed ﬁlter outputs. Rectifying by taking the absolute value means
that we do not distinguish between light spots on a dark background and dark spots
on a light background; the alternative, half-wave rectiﬁcation (described in the text),
preserves this distinction at the cost of a fuller representation. One can summarize either
by smoothing (which will tend to suppress noise, as in the schematic example above) or
by taking the maximum over a neighborhood. Compare this ﬁgure to Figure 6.7, which
shows a representation for a real image.
generic subelements, such as spots and bars. We ﬁnd subelements with ﬁlters, then
represent each point in the image with a summary of the pattern of subelements
nearby. This will work because the parts of a texton repeat in the same way that
the texton does.
This suggests representing image textures in terms of the response of a collec-
tion of ﬁlters. Each ﬁlter is a detector for a subelement. The collection of diﬀerent
ﬁlters would represent subelements—spots and bars are usual—at a collection of
scales (to identify bigger or smaller subelements). We can now represent each point
in an image by the vector of ﬁlter outputs at that point. This vector gives a sense
of how much the neighborhood around that point looks like each subelement at
each scale (Figure 6.3).
6.1.1 Spots and Bars
But what ﬁlters should we use? There is no canonical answer. A variety of answers
have been tried. By analogy with the human visual cortex, one could use some
spot ﬁlters, some oriented edge ﬁlters, and some oriented bar ﬁlters at diﬀerent
orientations and scales (Figure 6.4). This seems like a natural choice, because these
are in some sense “minimal” subelements. It would be hard to have subelements
of patterns with less structure than a spot, and it would be hard to have oriented
subelements with less structure than an edge.
Section 6.1 Local Texture Representations Using Filters 168
FIGURE 6.4: Left shows a set of 48 oriented ﬁlters used for expanding images into a series
of responses for texture representation. Each ﬁlter is shown on its own scale, with zero
represented by a mid-gray level, lighter values being positive, and darker values being
negative. The left three columns represent edges at three scales and six orientations; the
center three columns represent stripes; and the right two represent two classes of spots
(with and without contrast at the boundary) at diﬀerent scales. This is the set of ﬁlters
used by Leung and Malik (2001). Right shows a set of orientation-independent ﬁlters,
used by Schmid (2001), using the same representation (there are only 13 ﬁlters in this set,
so there are ﬁve empty slots in the image). The orientation-independence property means
that these ﬁlters look like complicated spots.
In some applications, we would like texture recognition performance to be
unaﬀected if the texture sample is rotated. This is diﬃcult to achieve with oriented
ﬁlters, because one might need to sample the orientations very ﬁnely. An alternative
to using oriented ﬁlters is to use ﬁlters that are orientation-independent, all of which
must look like complicated spots (Figure 6.4).
6.1.2 From Filter Outputs to Texture Representation
Assume we have an image J. A set of ﬁlter output maps (which would have the
form T
i
∗∗J for diﬀerent ﬁlters T
i
) is not, in itself, a representation of texture. The
representation tells us what the window around a pixel looks like; but in a texture,
what counts is not only what’s at a pixel, but also what’s nearby. For example, a
ﬁeld of yellow ﬂowers may consist of many small yellow spots with some vertical
green bars. What’s important is not just the fact that a particular pixel looks a
lot like a spot, but also that near the pixel there are no other spots, but some
bars. This means the texture representation at a point should involve some kind of
summary of nearby ﬁlter outputs, rather than just the ﬁlter outputs themselves.
The ﬁrst step in building a reasonable summary is to notice that the summary
must represent a neighborhood around a pixel that is rather bigger than the scale
of the ﬁlter. To know whether the neighborhood around a pixel is “spotty,” it is
not enough to know that there is one strong spot in it; there should be many spots,
each quite small compared to the size of the patch. However, it is important not to
look at too large a neighborhood, or else the representation will not change much
as we move across the image (because the neighborhoods overlap). The particular
Section 6.1 Local Texture Representations Using Filters 169
FIGURE 6.5: Filter responses for the oriented ﬁlters of Figure 6.4, applied to an image of
a wall. At the center, we show the ﬁlters for reference (but not to scale, because they
would be too small to resolve). The responses are laid out in the same way that the ﬁlters
are (i.e., the response map on the top left corresponds to the ﬁlter on the top left, and
so on). For reference, we show the image at the left. The image of the wall is small, so
that the ﬁlters respond to structures that are relatively large; compare with Figure 6.6,
which shows responses to a larger image of the wall, where the ﬁlters respond to smaller
structures. These are ﬁlters of a ﬁxed size, applied to a small version of the image, and
so are equivalent to large-scale ﬁlters applied to the original version. Notice the strong
response to the vertical and horizontal lines of mortar between the bricks, which are at
about the scale of the bar ﬁlters. All response values are shown on the same intensity
scale: lighter is positive, darker is negative, and mid-gray is zero.
arrangement of these spots within a neighborhood doesn’t matter all that much,
because the patch is small. This suggests that some form of average could give a
fair description of what is going on; an alternative is to take the strongest response.
We must process the responses before we summarize them. For example, a light
spot ﬁlter will give a positive response to a light spot on a dark background, and a
negative response to a dark spot on a light background. As a result, if we simply
average ﬁlter responses over a patch, then a patch containing dark and light spots
might record the same near-zero average as a patch containing no spots. This would
be misleading.
We could compute the absolute value for each output map, to get [ T
i
∗ ∗J [.
This does not distinguish between light spots on a dark background and dark spots
on a light background. An alternative, which does preserve this distinction, is to
report both max(0, T
i
∗∗J(x, y)) and max(0, −T
i
∗∗J(x, y)) (this is half-wave rectiﬁ-
Section 6.1 Local Texture Representations Using Filters 170
FIGURE 6.6: Filter responses for the oriented ﬁlters of Figure 6.4, applied to an image
of a wall. At the center, we show the ﬁlters for reference (not to scale). The responses
are laid out in the same way that the ﬁlters are (i.e., the response map on the top left
corresponds to the ﬁlter on the top left, and so on). For reference, we show the image at
the left. Although there is some response to the vertical and horizontal lines of mortar
between the bricks, it is not as strong as the coarse scale (Figure 6.5); there are also quite
strong responses to texture on individual bricks. All response values are shown on the
same intensity scale: lighter is positive, darker is negative, and mid-gray is zero.
cation), which yields two maps per ﬁlter. We can now summarize the neighborhood
around a pixel by computing a Gaussian weighted average (equivalently, convolving
with a Gaussian). The scale of this Gaussian depends on the scale of the ﬁlter for
the map; typically, it is around twice the scale of the ﬁlter.
6.1.3 Local Texture Representations in Practice
Several diﬀerent sets of ﬁlters have been used for texture representation. The Visual
Geometry Group at Oxford publishes code for diﬀerent sets of ﬁlters, written by
Manik Varma and by Jan-Mark Guesebroek, at http://www.robots.ox.ac.uk/
~
vgg/research/texclass/filters.html; this is part of an excellent web page on
texture classiﬁcation (http://www.robots.ox.ac.uk/
~
vgg/research/texclass/
index.html). One important part of ﬁltering an image with a large number
of ﬁlters is doing so quickly; recent code for this purpose, by Jan-Mark Guese-
broek, can be found at http://www.science.uva.nl/research/publications/
2003/GeusebroekTIP2003/. Some sets of oriented ﬁlters allow fast, eﬃcient rep-
resentations and have good translation and rotation properties. One such set is
Section 6.2 Pooled Texture Representations by Discovering Textons 171
Rectified Summarized
Light stripes
Dark background
Dark stripes
Light background
Positive response
Negative response
Vertical stripes
Image
Horizontal stripes
Light stripes
Dark background
Dark stripes
Light background
Positive response
Negative response
FIGURE 6.7: Filter-based texture representations look for pattern subelements such as
oriented bars. The brick image on the left is ﬁltered with an oriented bar ﬁlter (shown as
a tiny inset on the top left of the image at full scale) to detect bars, yielding stripe responses
(center left; negative is dark, positive is light, mid-gray is zero). These are rectiﬁed (here
we use half-wave rectiﬁcation) to yield response maps (center right; dark is zero, light
is positive). In turn, these are summarized (here we smoothed over a neighborhood twice
the ﬁlter width) to yield the texture representation on the right. In this, pixels that have
strong vertical bars nearby are light, and others are dark; there is not much diﬀerence
between the dark and light vertical structure for this image, but there is a real diﬀerence
between dark and light horizontal structure.
the steerable pyramid of Simoncelli and Freeman (1995a). Code for these ﬁlters is
available at http://www.cns.nyu.edu/
~
eero/steerpyr/.
6.2 POOLED TEXTURE REPRESENTATIONS BY DISCOVERING TEXTONS
A texture is a set of textons that repeat in some way. We could ﬁnd these textons
by looking for image patches that are common. An alternative is to ﬁnd sets of
texton subelements—that is, vectors of ﬁlter outputs—that are common (if textons
are repeated, then so are their subelements). There are two important diﬃculties
in ﬁnding image patches or vectors of ﬁlter outputs that commonly occur together.
First, these representations of the image are continuous. We cannot simply count
how many times a particular pattern occurs, because each vector is slightly diﬀerent.
Second, the representation is high dimensional in either case. A patch around a
Section 6.2 Pooled Texture Representations by Discovering Textons 172
Obtain a set of n ﬁlters representing subelements, at multiple scales
Apply each ﬁlter T
i
to the image
For each ﬁlter response map T
i
∗ ∗J, compute
max(0, T
i
∗ ∗J(x, y)) and max(0, −T
i
∗ ∗J(x, y))
For each of the 2n rectiﬁed maps, compute local summaries
either by convolving with a Gaussian of scale approximately twice the
scale of the base ﬁlter, or by taking the maximum value over that radius.
Algorithm 6.1: Local Texture Representation Using Filters.
pixel might need hundreds of pixels to represent it well; similarly, hundreds of
diﬀerent ﬁlters might be needed to represent the image at a pixel. This means we
cannot build a histogram directly, either, because it will have an unmanageable
number of cells.
6.2.1 Vector Quantization and Textons
Vector quantization is a strategy to deal with these diﬃculties. Vector quantization
is a way of representing vectors in a continuous space with numbers from a set
of ﬁxed size. We ﬁrst build a set of clusters out of a training set of vectors; this
set of clusters is often thought of as a dictionary. We now replace any new vector
with the cluster center closest to that vector. This strategy applies to vectors quite
generally, though we will use it for texture representation. Many diﬀerent clusterers
can be used for vector quantization, but it is most common to use k-means or one
of its variants. For concreteness, we describe this algorithm in Section 6.2.2, but
the other clusterers of Chapter 9 would apply.
We can now represent a collection of vectors as a histogram of cluster centers.
This general recipe can be applied to texture representation by describing each
pixel in the domain with some vector, then vector quantizing and describing the
domain with the histogram of cluster centers. Natural vectors to use are: the local
summary representation described in Section 6.1; a vector of unprocessed ﬁlter
outputs, using ﬁlters appropriate for a local texture representation (Figure 6.9); or
even just a vector obtained by reshaping the pixels from a ﬁxed-size patch around
the image pixel (Figure 6.10). In each case, we are building a representation in
terms of commonly repeated pattern elements.
6.2.2 K-means Clustering for Vector Quantization
We could use any clustering method to vector quantize (Chapter 9 describes a
number of diﬀerent clustering methods in the context of segmentation). However,
by far the most common method used is k-means clustering. Assume we have a set
of data items that we wish to cluster. We now assume that we know how many
clusters there are in the data, which we write k. This is equivalent to ﬁxing the
number of values we wish to quantize to. Each cluster is assumed to have a center;
we write the center of the ith cluster as c
i
. The jth data item to be clustered
is described by a feature vector x
j
. In our case, these items are vectors of ﬁlter
Section 6.2 Pooled Texture Representations by Discovering Textons 173
Dictionary
Cluster
1
2
3
2
3
1
3
Replace
with
closest
cluster
center
Histogram
L
e
a
r
n
i
n
g
a
d
i
c
t
i
o
n
a
r
y
R
e
p
r
e
s
e
n
t
i
n
g
a
r
e
g
i
o
n
FIGURE 6.8: There are two steps to building a pooled texture representation for a texture
in an image domain. First, one builds a dictionary representing the range of possible pat-
tern elements, using a large number of texture patches. This is usually done in advance, us-
ing a training data set of some form. Second, one takes the patches inside the domain, vec-
tor quantizes them by identifying the number of the closest cluster center, then computes
a histogram of the diﬀerent cluster center numbers that occur within a region. This his-
togram might appear to contain no spatial information, but this is a misperception. Some
frequent elements in the histogram are likely to be textons, but others describe common
ways in which textons lie close to one another; this is a rough spatial cue. This ﬁgure shows
elements of a database collected by C. Liu, L. Sharan, E. Adelson, and R. Rosenholtz, and
published at http: // people. csail. mit. edu/ lavanya/ research_ sharan. html . Fig-
ure by kind permission of the collectors.
Build a dictionary:
Collect many training example textures
Construct the vectors x for relevant pixels; these could be
a reshaping of a patch around the pixel, a vector of ﬁlter outputs
computed at the pixel, or the representation of Section 6.1.
Obtain k cluster centers c for these examples
Represent an image domain:
For each relevant pixel i in the image
Compute the vector representation x
i
of that pixel
Obtain j, the index of the cluster center c
j
closest to that pixel
Insert j into a histogram for that domain
Algorithm 6.2: Texture Representation Using Vector Quantization.
Section 6.2 Pooled Texture Representations by Discovering Textons 174
FIGURE 6.9: Pattern elements can be identiﬁed by vector quantizing vectors of ﬁlter
outputs, using k-means. Here we show the top 50 pattern elements (or textons), obtained
from all 1,000 images of the collection of material images described in Figure 6.2. These
were ﬁltered with the complete set of oriented ﬁlters from Figure 6.4. Each subimage
here illustrates a cluster center. For each cluster center, we show the linear combination
of ﬁlter kernels that would result in the set of ﬁlter responses represented by the cluster
center. For some cluster centers, we show the 25 image patches in the training set whose
ﬁlter representation is closest to the cluster center. This ﬁgure shows elements of a
database collected by C. Liu, L. Sharan, E. Adelson, and R. Rosenholtz, and published
at http: // people. csail. mit. edu/ lavanya/ research_ sharan. html . Figure by kind
permission of the collectors.
responses observed at image locations.
Because pattern elements repeat, and so are common, we can assume that
most data items are close to the center of their cluster. This suggests that we
cluster the data by minimizing the the objective function
Φ(clusters, data) =

i∈clusters
⎧
⎨
⎩

j∈ith cluster
(x
j
−c
i
)
T
(x
j
−c
i
)
⎫
⎬
⎭
.
Notice that if we know the center for each cluster, it is easy to determine which
cluster is the best choice for each point. Similarly, if the allocation of points to
clusters is known, it is easy to compute the best center for each cluster. However,
there are far too many possible allocations of points to clusters to search this space
for a minimum. Instead, we deﬁne an algorithm that iterates through two activities:
Section 6.2 Pooled Texture Representations by Discovering Textons 175
FIGURE 6.10: Pattern elements can also be identiﬁed by vector quantizing vectors obtained
by reshaping an image window centered on each pixel. Here we show the top 50 pattern
elements (or textons), obtained using this strategy from all 1,000 images of the collection
of material images described in Figure 6.2. Each subimage here illustrates a cluster center.
For some cluster centers, we show the closest 25 image patches. To measure distance, we
ﬁrst subtracted the average image intensity, and we weighted by a Gaussian to ensure that
pixels close to the center of the patch were weighted higher than those far from the center.
This ﬁgure shows elements of a database collected by C. Liu, L. Sharan, E. Adelson, and
R. Rosenholtz, and published at http: // people. csail. mit. edu/ lavanya/ research_
sharan. html . Figure by kind permission of the collectors.
• Assume the cluster centers are known and, allocate each point to the closest
cluster center.
• Assume the allocation is known, and choose a new set of cluster centers. Each
center is the mean of the points allocated to that cluster.
We then choose a start point by randomly choosing cluster centers, and then iterate
these stages alternately. This process eventually converges to a local minimum of
the objective function (the value either goes down or is ﬁxed at each step, and
it is bounded below). It is not guaranteed to converge to the global minimum of
the objective function, however. It is also not guaranteed to produce k clusters,
unless we modify the allocation phase to ensure that each cluster has some nonzero
number of points. This algorithm is usually referred to as k-means (summarized in
Section 6.3 Synthesizing Textures and Filling Holes in Images 176
Algorithm 6.3). It is possible to search for an appropriate number of clusters by
applying k-means for diﬀerent values of k and comparing the results; we defer a
discussion of this issue until Section 10.7.
Choose k data points to act as cluster centers
Until the cluster centers change very little
Allocate each data point to cluster whose center is nearest.
Now ensure that every cluster has at least
one data point; one way to do this is by
supplying empty clusters with a point chosen at random from
points far from their cluster center.
Replace the cluster centers with the mean of the elements
in their clusters.
end
Algorithm 6.3: Clustering by K-Means.
6.3 SYNTHESIZING TEXTURES AND FILLING HOLES IN IMAGES
Many diﬀerent kinds of user want to remove things from images or from video. Art
directors might like to remove unattractive telephone wires; restorers might want to
remove scratches or marks; there’s a long history of government oﬃcials removing
people with embarrassing politics from publicity pictures (see the fascinating pic-
tures in King (1997)); and home users might wish to remove a relative they dislike
from a family picture. All these users must then ﬁnd something to put in place of
the pixels that were removed. Ideally, a program would create regions of texture
that ﬁt in and look convincing, using either other parts of the original image, or
other images.
There are other important applications for such a program. One is to produce
large quantities of texture for digital artists to apply to object models. We know
that good textures make models look more realistic (it’s worth thinking about why
this should be true). Tiling small texture images tends to work poorly, because it
can be hard to obtain images that tile well. The borders have to line up properly,
and even when they do, the resulting periodic structure can be annoying.
6.3.1 Synthesis by Sampling Local Models
As Efros and Leung (1999) point out, an example texture can serve as a probability
model for texture synthesis (Figure 6.11). Assume for the moment that we know
every pixel in the synthesized image, except one. To obtain a probability model for
the value of that pixel, we could match a neighborhood of the pixel to the example
image. Every matching neighborhood in the example image has a possible value for
the pixel of interest. This collection of values is a conditional histogram for the pixel
of interest. By drawing a sample uniformly and at random from this collection, we
obtain the value that is consistent with the example image.
We must now take some form of neighborhood around the pixel of interest,
Section 6.3 Synthesizing Textures and Filling Holes in Images 177
compare it to neighborhoods in the example image, and select some of these to form
a set of example values. The size and shape of this neighborhood is signiﬁcant,
because it codes the range over which pixels can aﬀect one another’s values directly
(see Figure 6.12). Efros et al. use a square neighborhood, centered at the pixel of
interest.
Choose a small square of pixels at random from the example image
Insert this square of values into the image to be synthesized
Until each location in the image to be synthesized has a value
For each unsynthesized location on
the boundary of the block of synthesized values
Match the neighborhood of this location to the
example image, ignoring unsynthesized
locations in computing the matching score
Choose a value for this location uniformly and at random
from the set of values of the corresponding locations in the
matching neighborhoods
end
end
Algorithm 6.4: Non-parametric Texture Synthesis.
The neighborhoods we select will be similar to the image example in some
sense. A good measure of similarity between two image neighborhoods can be
measured by forming the sum of squared diﬀerences (or ssd) of corresponding pixel
values. We assume that the missing pixel is at the center of the patch to be
synthesized, which we write o. We assume the patch is square, and adjust the
indexes of the patch to run from −n to n in each direction. The sum of squared
diﬀerences between this patch and an image patch T of the same size is given by

(i,j)∈patch,(i,j)=(0,0)
(/
ij
−B
ij
)
2
.
The notation implies that because we don’t know the value of the pixel to be
synthesized (which is at (0, 0)), we don’t count it in the sum of squared diﬀerences.
This similarity value is small when the neighborhoods are similar, and large when
they are diﬀerent (it is essentially the length of the diﬀerence vector). However,
this measure places the same weight on pixels close to the unknown value as it does
on distant pixels. Better results are usually obtained by weighting up nearby pixels
and weighting down distant pixels. We can do so using Gaussian weights, yielding

(i,j)∈patch,(i,j)=(0,0)
(/
ij
−B
ij
)
2
exp
_
−(i
2
+j
2
)
2σ
2
_
.
Now we know how to obtain the value of a single missing pixel: choose uniformly
and at random amongst the values of pixels in the example image whose neigh-
borhoods match the neighborhood of our pixel. We cannot choose those matching
Section 6.3 Synthesizing Textures and Filling Holes in Images 178
FIGURE 6.11: Efros and Leung (1999) synthesize textures by matching neighborhoods of
the image being synthesized to the example image, and then choosing at random amongst
the possible values reported by matching neighborhoods (Algorithm 6.4). This means
that the algorithm can reproduce complex spatial structures, as these examples indicate.
The small block on the left is the example texture; the algorithm synthesizes the block
on the right. Note that the synthesized text looks like text: it appears to be constructed
of words of varying lengths that are spaced like text, and each word looks as though it
is composed of letters (though this illusion fails as one looks closely). This ﬁgure was
originally published as Figure 3 of “Texture Synthesis by Non-parametric Sampling,” A.
Efros and T.K. Leung, Proc. IEEE ICCV, 1999 c _ IEEE, 1999.
neighborhoods by just setting a threshold on the similarity function, because we
might not have any matches. A better strategy to ﬁnd matching neighborhoods
is to select all whose similarity value is less than (1 + )s
min
, where s
min
is the
similarity function of the closest neighborhood and is a parameter.
Generally, we need to synthesize more than just one pixel. Usually, the values
of some pixels in the neighborhood of the pixel to be synthesized are not known;
these pixels need to be synthesized too. One way to obtain a collection of examples
for the pixel of interest is to count only the known values in computing the sum
of squared diﬀerences, and scale the similarity to take into account the number of
known pixels. Write / for the set of pixels around a point whose values are known,
and / for the size of this set. We now have, for the similarity function,
1
/

(i,j)∈K
(/
ij
−B
ij
)
2
exp
_
−(i
2
+j
2
)
2σ
2
_
.
The synthesis process can be started by choosing a block of pixels at random from
the example image, yielding Algorithm 6.4.
Filling in Patches
Section 6.3 Synthesizing Textures and Filling Holes in Images 179
FIGURE 6.12: The size of the image neighborhood to be matched makes a signiﬁcant
diﬀerence in Algorithm 6.4. In the ﬁgure, the textures at the right are synthesized from
the small blocks on the left, using neighborhoods that are increasingly large as one moves
to the right. If very small neighborhoods are matched, then the algorithm cannot capture
large-scale eﬀects easily. For example, in the case of the spotty texture, if the neighborhood
is too small to capture the spot structure (and so sees only pieces of curve), the algorithm
synthesizes a texture consisting of curve segments. As the neighborhood gets larger, the
algorithm can capture the spot structure, but not the even spacing. With very large
neighborhoods, the spacing is captured as well. This ﬁgure was originally published as
Figure 2 of “Texture Synthesis by Non-parametric Sampling,” A. Efros and T.K. Leung,
Proc. IEEE ICCV, 1999 c _ IEEE, 1999.
Synthesizing a large texture in terms of individual pixels will be unnecessarily
slow. Because textures repeat, we expect that whole blocks of pixels also should
repeat. This suggests synthesizing a texture in terms of image patches, rather than
just pixels. Most of the mechanics of the procedure follow those for pixels: to
synthesize a texture patch at a location, we ﬁnd patches likely to ﬁt (because they
have pixels that match the boundary at that location), then choose uniformly and
at random from among them. However, when we place down the new patch, we
must deal with the fact that some (ideally, many) of its pixels overlap with pixels
that have already been synthesized. This problem is typically solved by image
segmentation methods, and we defer that discussion to Chapter 9.
6.3.2 Filling in Holes in Images
There are four approaches we can use to ﬁll a hole in an image. Matching meth-
ods ﬁnd another image patch that looks a lot like the boundary of the hole, place
that patch over the hole, and blend the patch and the image together. The patch
might well be found in the image (for example, Figure 6.13). If we have a very large
set of images, we could ﬁnd a patch by looking for another image that matches the
image with a hole in it. Hays and Efros (2007) show this strategy can be extremely
Section 6.3 Synthesizing Textures and Filling Holes in Images 180
FIGURE 6.13: If an image contains repeated structure, we have a good chance of ﬁnding
examples to ﬁll a hole by searching for patches that are compatible with its boundaries.
Top left: An image with a hole in it (black pixels in a rough pedestrian shape). The
pixels on the region outside the hole, but inside the boundary marked on the image,
match pixels near the other curve, which represents a potentially good source of hole-
ﬁlling pixels. Top right: The hole ﬁlled by placing the patch over the hole, then using
a segmentation method (Chapter 9) to choose the right boundary between patch and
image. This procedure can work for apparently unpromising images, such as the one on
the bottom left, an image of the facade of a house, seen at a signiﬁcant slant. This
slant means that distant parts of the facade are severely foreshortened. However, if we
rectify the facade using methods from Section 1.3, then there are matching patches. On
the bottom right, the hole has been ﬁlled in using a patch from the rectiﬁed image,
that is then slanted again. This ﬁgure was originally published as Figures 3 and 6 of
“Hole Filling through Photomontage,” by M. Wilczkowiak, G. Brostow, B. Tordoﬀ, and
R. Cipolla, Proc. BMVC, 2005 and is reproduced by kind permission of the authors.
successful. Blending is typically achieved using methods also used for image seg-
mentation (Section 9.4.3 describes one method that can be used for blending).
As you would expect, matching methods work very well when a good match
is available, and poorly otherwise. If the hole is in a region of relatively regular
texture, then a good match should be easy to ﬁnd. If the texture is less strongly
structured, it might be hard to ﬁnd a good match. In cases like this, it makes
sense to try and synthesize the texture over the region of the hole, using the rest
of the image as an example. Making such texture synthesis methods work well
requires considerable care, because the order in which pixels are synthesized has
a strong eﬀect on the results. Texture synthesis tends to work better for patches
when most of their neighbors are known, because the match is more constrained.
As a result, one wants to synthesize patches at the boundary of the hole. It is
also important to extend edges at the boundary of the hole into the interior (for
Section 6.3 Synthesizing Textures and Filling Holes in Images 181
Image
Hole
Onionskin order
Boundary edges
FIGURE 6.14: Texture synthesis methods can ﬁll in holes accurately, but the order in
which pixels are synthesized is important. In this ﬁgure, we wish to remove the sign,
while preserving the signpost. Generally, we want to ﬁll in pixels where most of the
neighbors are known ﬁrst. This yields better matching patches. One way to do so is to ﬁll
in from the boundary. However, if we simply work our way inwards (onionskin ﬁlling), long
scale image structures tend to disappear. It is better to ﬁll in patches close to edges ﬁrst.
This ﬁgure was originally published as Figure 11 of “Region Filling and Object Removal
by Exemplar-Based Image Inpainting,” by A. Criminisi, P. Perez, and K. Toyama, IEEE
Transactions on Image Processing, 2004 c _ IEEE, 2004.
example, see Figure 6.14); in practice, this means that it is important to synthesize
patches at edges on the boundary before one ﬁlls in other patches. It is possible
to capture both requirements in a priority function ((Criminisi et al. 2004)), which
speciﬁes where to synthesize next.
If we choose an image patch at (i, j) as an example to ﬁll in location (u, v)
in the hole, then image patches near (i, j) are likely to be good for ﬁlling in points
near (u, v). This observation is the core of coherence methods, which apply this
constraint to texture synthesis. Finally, some holes in images are not really texture
holes; for example, we might have a hole in a smoothly shaded region. Texture
synthesis and matching methods tend to work poorly on such holes, because the
intensity structure on the boundary is not that distinctive. As a result, we may
ﬁnd many matching patches, some of which have jarring interiors. Variational
methods apply in these cases. Typically, we try to extend the level curves of the
image into the hole in a smooth way. Modern hole-ﬁlling methods use a combi-
nation of these approaches, and can perform very well on quite demanding tasks
(Figure 6.15).
Section 6.4 Image Denoising 182
Initial Image
Initial Image
Object masked out
Object masked out
Object composited back
Initial Image Hole Extended by hole filling
FIGURE 6.15: Modern hole-ﬁlling methods get very good results using a combination of
texture synthesis, coherence, and smoothing. Notice the complex, long-scale structure
in the background texture for the example on the top row. The center row shows an
example where a subject was removed from the image and replaced in a diﬀerent place.
Finally, the bottom row shows the use of hole-ﬁlling to resize an image. The white block
in the center mask image is the “hole” (i.e., unknown pixels whose values are required to
resize the image). This block is ﬁlled with a plausible texture. This ﬁgure was originally
published as Figures 9 and 15 of “A Comprehensive Framework for Image Inpainting,”
by A. Bugeau, M. Bertalm´ıo, V. Caselles, and G. Sapiro, Proc. IEEE Transactions on
Image Processing, 2010 c _ IEEE, 2010.
6.4 IMAGE DENOISING
This section addresses the problem of reconstructing an image given the noisy ob-
servations gathered by a digital camera sensor. Today, with advances in sensor
design, the signal is relatively clean for digital SLRs at low sensitivities, but it
remains noisy for consumer-grade and mobile-phone cameras at high sensitivities
(low-light and/or high-speed conditions). Adding to the demands of consumer and
professional photography those of astronomy, biology, and medical imaging, it is
thus clear that image restoration is still of acute and in fact growing importance.
Working with noisy images recorded by digital cameras is diﬃcult because diﬀerent
devices produce diﬀerent kinds of noise, and introduce diﬀerent types of artifacts
and spatial correlations in the noise as a result of internal post-processing (demo-
saicking, white balance, etc.).
Section 6.4 Image Denoising 183
We have already seen that linear ﬁlters such as Gaussian kernels are eﬀective
at suppressing noise, but that the price to pay is a loss in image detail. We brieﬂy
discuss in this section three related approaches to image denoising that are much
more eﬀective. They rely on two properties of natural images: the prominence of
self-similarities—that is, many small regions in the same picture often look the
same—and the eﬀectiveness of sparse linear models—that is, small image patches
are typically well reconstructed as a linear combination of very few elements from
a potentially large basis set, or dictionary.
6.4.1 Non-local Means
Efros and Leung (1999) have shown that the self-similarities inherent to natural
images can be used eﬀectively in texture synthesis tasks. Following this insight,
Buades, Coll, and Morel (2005) have introduced the non-local means approach to
image denoising, where the prominence of self-similarities is used as a prior on
natural images. Concretely, let us consider a noisy image written as a column
vector y in R
n
, and denote by y[i] the i-th pixel value and by y
i
the patch of size
m centered on this pixel and considered as an element of R
m
. Similar patches y
i
and
y
j
should have similar values y[i] and y[j]. This suggests estimating the denoised
pixel x[i] as a weighted average (the so-called Nadaraya-Watson estimator) of all
the other pixels in the image:
x[i] =
n

j=1
G
h
(y
i
−y
j
)

n
l=1
G
h
(y
i
−y
l
)
y[j], (6.1)
where G
h
is a multi-dimensional Gaussian kernel of standard deviation h. The
weights depend on appearance similarity instead of spatial proximity in the case
of Gaussian smoothing, hence the name of non-local means. This simple approach
gives excellent results in practice, and, although naive implementations are slow (all
image pixels are used to denoise a single one), they can be sped up by using various
heuristics (by considering only patches y
j
in some ﬁxed spatial neighborhood of
y
i
, for example). The parameters h can be taken proportional to the standard
deviation σ of the noise in practice; for example, h = 12σ is used in the experiments
of Buades, Coll and Morel (2005).
6.4.2 Block Matching 3D (BM3D)
Classical shrinkage is a very diﬀerent method for denoising. It can be summarized
as follows: Consider a signal y in R
m
and some nonsingular m m matrix T .
We associate with y its code α = T y and the thresholded value α
ε
, obtained by
zeroing all coeﬃcients α
i
smaller than some ε > 0 in the hard thresholding case, or
by setting
α
i
ε
= sign(α
i
)([α
i
[ −ε)
+
,
in the soft thresholding one (here, x
+
is equal to x when x > 0, and to zero oth-
erwise). The denoised signal is x
ε
= T
−1
α
ε
, the idea being that noise shows up
mostly in small coeﬃcients in the transformed domain, which is of course true only
for appropriate transformations. A classical example is wavelet shrinkage (Donoho
and Johnstone 1995), where T is the orthogonal matrix representing the discrete
Section 6.4 Image Denoising 184
wavelet transform (Mallat 1999), and the denoised signal is x
ε
= T
T
α
ε
. In this
case, a method for selecting ε for a given noise level is available, along with theo-
retical guarantees about the reconstructed signal.
By construction, the vector x
ε
usually admits a sparse decomposition on
the basis of R
m
formed by the columns of T
T
= T
−1
—that is, only a few of the
coeﬃcients α
i
ε
are nonzero. As further discussed in Chapter 22, sparse linear models
are well suited to natural images, and Dabov et al. (2007) combine sparsity-inducing
shrinkage with the use of self-similarities. They stack similar image patches into
three-dimensional arrays (groups), then use shrinkage on the groups, coupled with
transformations such as the (three-dimensional) discrete cosine transform, or DCT.
Because the patches are similar, the decomposition of each group is expected to be
quite sparse, and denoised patches x
ε
are retrieved from the shrunken groups. The
ﬁnal value of a pixel is taken to be the average of the values x
ε
at this point for all
patches passing through it. In conjunction with a few simple heuristics, this simple
idea has proven to be very eﬀective, and it typically gives better results than regular
non-local means.
6.4.3 Learned Sparse Coding
An alternative is to assume that the clean signal can be approximated by a sparse
linear combination of elements from a (potentially large) set of vectors forming the
k columns of a so-called dictionary, which may be overcomplete (k > m). Under
this assumption, denoising a patch y in R
m
with a dictionary T in R
m×k
composed
of k elements amounts to solving the sparse decomposition problem
min
α∈R
k
[[α[[
1
s.t. [[y −Tα[[
2
2
≤ ε, (6.2)
where Tα is an estimate of the clean signal, and [[α[[
1
is the sparsity-inducing

1
norm formed by the sum of the absolute values of the coeﬃcients of α. As
shown in Chapter 22, the
1
regularization in Equation (6.2) leads to the convex
Lasso (Tibshirani 1996) and basis pursuit (Chen et al. 1999) problems, for which
eﬃcient algorithms are available. As shown in Elad and Aharon (2006), ε can be
chosen according to the standard deviation of the noise.
Various types of wavelets have been used as dictionaries for natural images.
Elad and Aharon (2006) have proposed instead to learn a dictionary T adapted
to the image at hand, and demonstrated that learned dictionaries lead to better
empirical performance than oﬀ-the-shelf ones. For an image of size n, a dictionary
in R
m×k
adapted to the n overlapping patches of size m (typically m = 8 8 ¸n)
associated with the image pixels, is learned by addressing the following optimization
problem:
min
D∈C,A
n

i=1
[[α
i
[[
1
s.t. [[y
i
−Tα
i
[[
2
2
≤ ε, (6.3)
where ( is the set of matrices in R
m×k
with unit
2
-norm columns, / = [α
1
, . . . , α
n
]
is a matrix in R
k×n
, y
i
is the i-th patch of the noisy image y, α
i
is the correspond-
ing code, and Tα
i
is the estimate of the denoised patch. Note that this procedure
implicitly assumes that the patches are independent from each other, which is ques-
tionable because they overlap. However, this approximation makes the correspond-
Section 6.4 Image Denoising 185
FIGURE 6.16: Sparsity vs. joint sparsity: Gray squares represents nonzero values in vectors
(left) or matrix (right). Reprinted from “Non-local Sparse Models for Image Restoration,”
by J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Proc. International
Conference on Computer Vision, (2009). c _ 2009, IEEE.
ing optimization tractable. Indeed, although dictionary learning is traditionally
considered very costly, online algorithms such as the procedure described in Chap-
ter 22 and Mairal, Bach, Ponce, and Sapiro (2010) make it possible to eﬃciently
process millions of patches, allowing the use of large photographs and/or large im-
age databases. In typical applications, the dictionary T is ﬁrst learned on such a
database, then reﬁned on the image of interest itself using the same process.
Once the dictionary T and codes α
i
have been learned, every pixel admits m
estimates (one per patch containing it), and its value can be computed by averaging
these values.
Let us close this section by showing that self-similarities also can be exploited
in this framework. Concretely, a joint sparsity pattern—that is, a common set
of nonzero coeﬃcients—can be imposed to a set of vectors α
1
, . . . , α
l
through a
grouped-sparsity regularizer on the matrix / = [α
1
, . . . , α
l
] in R
k×l
(Figure 6.16).
This amounts to limiting the number of nonzero rows of /, or replacing the
1
vector norm in Equation (6.3) by the
1,2
matrix norm
[[/[[
1,2
=
k

i=1
[[α
i
[[
2
, (6.4)
where α
i
denotes the i-th row of /.
Similar to the BM3D groups, we can deﬁne for each y
i
the set S
i
of patches
similar to it, using for example a threshold on the inter-patch Euclidean distance.
The dictionary learning problem can now be written as
min
(A
i
)
n
i=1
,D∈C
n

i=1
[[/
i
[[
1,2
[S
i
[
s.t. ∀i

j∈S
i
[[y
j
−Tα
ij
[[
2
2
≤ ε
i
, (6.5)
where /
i
= [α
ij
]
j∈§
i
∈ R
k×|S
i
|
, and an appropriate value of ε
i
can be chosen
as before. The normalization by [S
i
[ gives equal weights for all groups. For a
ﬁxed dictionary, simultaneous sparse coding is convex and can be solved eﬃciently
(Friedman 2001; Bach, Jenatton, Mairal, & Obozinski 2011). In turn, the dictionary
can be learned using a simple and eﬃcient modiﬁcation of the algorithm presented
in Chapter 22 and Mairal et al. (2010), and the ﬁnal image is estimated by averaging
the estimates of each pixel. In practice, this method gives better results than plain
dictionary learning.
Section 6.4 Image Denoising 186
FIGURE 6.17: Denoising images artiﬁcially corrupted with additive Gaussian noise. Left:
noisy images. Right: restored ones using LSSC. Note that the algorithm reproduces the
original brick texture in the house image (σ = 15) and the hair texture for the man image
(σ = 50), both hardly visible in the noisy images. Reprinted from “Non-local Sparse Models
for Image Restoration,” by J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman,
Proc. International Conference on Computer Vision, (2009). c _ 2009, IEEE.
6.4.4 Results
The three methods discussed in this section all give very good results, with a slight
edge to BM3D and learned simultaneous sparse coding (or LSSC), according to the
experiments of Dabov, Foi, Katkovnik, and Egiazarian (2007) and Mairal, Bach,
Ponce, Sapiro, and Zisserman (2009), which use standard images with added Gaus-
sian noise (Figure 6.17) for quantitative evaluation. Qualitative results on a photo-
graph corrupted by real noise are shown in Figure 6.18. The image was taken by a
Canon Powershot G9 digital camera at 1,600 ISO with a short time exposure. At
this setting, pictures are typically quite noisy. This time, we compare the original
JPEG output of the camera, and results from Adobe Camera Raw 5.0, the DxO
Optics Pro 5.3 package for professional photographers, and LSSC.
Section 6.5 Shape from Texture 187
FIGURE 6.18: From left to right and top to bottom: Camera JPEG output, Adobe
Camera Raw, DxO Optics Pro, and LSSC. Reprinted from “Non-local Sparse Models for
Image Restoration,” by J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Proc.
International Conference on Computer Vision, (2009). c _ 2009, IEEE.
6.5 SHAPE FROM TEXTURE
A patch of texture viewed frontally looks very diﬀerent from the same patch viewed
at a glancing angle because foreshortening causes the texture elements (and the gaps
between them!) to shrink more in some directions than in others. This suggests
that we can recover some shape information from texture, at the cost of supplying
a texture model. This is a task at which humans excel (Figure 6.19). Remarkably,
quite general texture models appear to supply enough information to infer shape.
6.5.1 Shape from Texture for Planes
If we know we are viewing a plane, shape from texture boils down to determining
the conﬁguration of the plane relative to the camera. Assume that we hypothesize
a conﬁguration; we can then project the image texture back onto that plane. If
we have some model of the “uniformity” of the texture, then we can test that
property for the backprojected texture. We now obtain the plane with the “best”
Section 6.5 Shape from Texture 188
0
10
20
30
40
50
0
10
20
30
40
50
0
0.2
0.4
0.6
0.8
1
FIGURE 6.19: Humans obtain information about the shape of surfaces in space from the
appearance of the texture on the surface. The ﬁgure on the left shows one common use
for this eﬀect; away from the contour regions, our only source of information about the
surface depicted is the distortion of the texture on the surface. On the right, the texture
gives a clear sense of the orientation of the ground plane, how the plants stand out from
the path, and how far away the building at the back is. Geoﬀ Brightling c _ Dorling
Kindersley, used with permission.
backprojected texture on it.
Assume that we are viewing a single textured plane in an orthographic camera.
Because the camera is orthographic, there is no way to measure the depth to the
plane. However, we can think about the orientation of the plane. Let us work in
terms of the camera coordinate system. First, we need to know the angle between
the normal of the textured plane and the viewing direction—sometimes called the
slant—and second, the angle the projected normal makes in the camera coordinate
system—sometimes called the tilt (Figure 6.20).
In an image of a plane, there is a tilt direction—the direction in the plane
parallel to the projected normal.
An isotropic texture is one where the probability of encountering a texture
element does not depend on the orientation of that element. This means that a
probability model for an isotropic texture need not depend on the orientation of
the coordinate system on the textured plane.
If we assume that the texture is isotropic, both slant and tilt can be read from
the image. We could synthesize an orthographic view of a textured plane by ﬁrst
rotating the coordinate system by the tilt and then contracting along one coordinate
direction by the cosine of the slant—call this process a viewing transformation.
The easiest way to see this is to assume that the texture consists of a set of circles,
scattered about the plane. In an orthographic view, these circles will project to
ellipses, whose minor axes will give the tilt and whose aspect ratios will give the
slant (see the exercises and Figure 6.20).
An orthographic view of an isotropic texture is not isotropic (unless the plane
is parallel to the image plane). This is because the contraction in the slant direction
interferes with the isotropy of the texture. Elements that point along the contracted
Section 6.5 Shape from Texture 189
Textured
plane
Image
plane
Tilt
Viewing
direction
Projected
normal
Plane
normal
FIGURE 6.20: The orientation of a plane with respect to the camera plane can be given
by the slant, which is the angle between the normal of the textured plane and the viewing
direction, and the tilt, which is the angle the projected normal makes with the camera
coordinate system. The ﬁgure illustrates the tilt, and shows a circle projecting to an
ellipse. The direction of the minor axis of this image ellipse is the tilt, and the slant is
revealed by the aspect ratio of the ellipse. However, the slant is ambiguous because the
foreshortening is given by cos σ, where σ is the slant angle. There will be two possible
values of σ for each foreshortening, so two diﬀerent slants yield the same ellipse (one is
slanted forwards, the other backwards).
direction get shorter. Furthermore, elements that have a component along the
contracted direction have that component shrunk. Now corresponding to a viewing
transformation is an inverse viewing transformation (which turns an image
plane texture into the object plane texture, given a slant and tilt). This yields
a strategy for determining the orientation of the plane: ﬁnd an inverse viewing
transformation that turns the image texture into an isotropic texture, and recover
the slant and tilt from that inverse viewing transformation.
There are various ways to ﬁnd this viewing transformation. One natural
strategy is to use the energy output of a set of oriented ﬁlters. This is the squared
response, summed over the image. For an isotropic texture, we would expect the
energy output to be the same for each orientation at any given scale, because the
probability of encountering a pattern does not depend on its orientation. Thus, a
measure of isotropy is the standard deviation of the energy output as a function of
orientation. We could sum this measure over scales, perhaps weighting the measure
by the total energy in the scale. The smaller the measure, the more isotropic the
texture. We now ﬁnd the inverse viewing transformation that makes the image
looks most isotropic by this measure, using standard methods from optimization.
The main diﬃculty with using an assumption of isotropy to recover the orientation
of a plane is that there are very few isotropic textures in the world.
Section 6.5 Shape from Texture 190
FIGURE 6.21: On the left, a textured surface, whose texture is a set of repeated elements,
in this case, spots. Center left, a reconstruction of the surface, made using texture
information alone. This reconstruction has been textured, which hides some of its imper-
fections. Center right, the same reconstruction, now rendered as a slightly glossy gray
surface. Because texture elements are repeated, we can assume that if diﬀerent elements
have a signiﬁcantly diﬀerent brightness, this is because they experience diﬀerent illumi-
nation. Right shows an estimate of the illumination on the surface obtained from this
observation. Notice how folds in the dress (arrows) tend to be darker; this is because, for a
surface element at the base of a fold, nearby cloth blocks a high percentage of the incident
light. This ﬁgure was originally published as Figure 4 of “Recovering Shape and Irradiance
Maps from Rich Dense Texton Fields,” by A. Lobay and D. Forsyth Proc. IEEE CVPR
2004 c _ IEEE, 2004.
6.5.2 Shape from Texture for Curved Surfaces
Shape from texture is more complicated for curved surfaces, because there are
more parameters to estimate. There are a variety of strategies, and there remains
no consensus on what is best. If we assume that a texture consists of repeated
small elements, then individual elements display no observable perspective eﬀects
(because they are small). Furthermore, curved surfaces often span fairly small
ranges of depth, because if they curve fast enough, they must eventually turn away
from the eye (if they don’t, we might be able to model them as planes). All this
suggests that we assume the surface is viewed in an orthographic camera.
Now consider a set of elements on a curved surface. Each element is an
instance of a model element; you should think of the surface elements as copies of
the model, which have been placed on the surface at diﬀerent locations. Each is
small, so we can model them as lying on the surface’s tangent plane. Each element
has diﬀerent slant and tilt directions. This means that each image instance of the
element is an instance of the model element, that has been rotated and translated
to place it in the image, then scaled along the image slant direction. It turns out
that, given suﬃcient image instances, we can infer both the model element and the
surface normal at the element (up to a two-fold ambiguity; Figure 6.20) from this
Section 6.6 Notes 191
information (the proof will take us far out of our way; it is in Lobay and Forsyth
(2006)). We must now ﬁt a surface to this information. Doing so is complicated,
because we need to resolve the ambiguity at each surface normal. This can be done
by assuming that the surface is smooth (so that elements that lie near one another
tend to share normal values), and by assuming we have some geometric constraints
on the surface.
Interestingly, modeling a texture as a set of repeated elements reveals illumi-
nation information. If we can ﬁnd multiple instances of an element on a surface,
then the reason for their diﬀerent image brightnesses is that they experience dif-
ferent illumination (typically, because they are at diﬀerent angles to the incoming
light). We can estimate surface irradiance directly from this information, even if
the illumination ﬁeld is complex (Figure 6.21).
6.6 NOTES
The idea that textures involve repetition of elements is fundamental, and appears
in a wide variety of forms. Under some circumstances, one can try to infer the
elements directly, as in Liu et al. (2004). Image compression can take advantage of
the repetitions created by texture. If we have large, plane ﬁgures in view (say, the
faces of buildings), then it can be advantageous to model viewing transformations
to compress the image (because then the same element repeats more often). This
means that, on occasion, image compression papers contain a shape from texture
component (for example, Wang et al. (2008)).
Filters, Pyramids and Eﬃciency
If we are to represent texture with the output of a large range of ﬁlters at many
scales and orientations, then we need to be eﬃcient at ﬁltering. This is a topic that
has attracted much attention; the usual approach is to try and construct a tensor
product basis that represents the available families of ﬁlters well. With an appropri-
ate construction, we need to convolve the image with a small number of separable
kernels, and can estimate the responses of many diﬀerent ﬁlters by combining the
results in diﬀerent ways (hence the requirement that the basis be a tensor product).
Signiﬁcant papers include Freeman and Adelson (1991), Greenspan et al. (1994),
Hel-Or and Teo (1996), Perona (1992), (1995), Simoncelli and Farid (1995), and
Simoncelli and Freeman (1995b).
Pooled Texture Representations
The literature does not seem to draw the distinction between local and pooled
texture representations explicitly. We think it is important, because quite diﬀer-
ent texture properties are being represented. There has been a fair amount of
discussion of just what should be vector quantized to form these representations.
Typically, one evaluates the goodness of a particular representation by its dis-
criminative performance in a texture classiﬁcation task; we discuss this topic in
Chapter 16. Signiﬁcant papers include Varma and Zisserman (2003), Varma and
Zisserman (2005), Varma and Zisserman (2009), Leung and Malik (2001), Leung
and Malik (1999), Leung and Malik (1996), Schmid (2001),
Section 6.6 Notes 192
Texture Synthesis
Texture synthesis exhausted us long before we could exhaust it. Patch based texture
synthesis is due to Efros and Freeman (2001); this paper hints at a form of con-
ditional texture synthesis. Hertzmann et al. (2001) demonstrate that conditional
texture synthesis can do the most amazing tricks. Vivek Kwatra and Li-Yi Wei
organized an excellent course on texture synthesis at SIGGRAPH 2007; the notes
are at http://www.cs.unc.edu/
~
kwatra/SIG07_TextureSynthesis/index.htm.
Denoising
Early work on image denoising relied on various smoothness assumptions—such as
Gaussian smoothing, anisotropic ﬁltering (Perona and Malik 1990c), total varia-
tion (Rudin et al. 2004), or image decompositions on ﬁxed bases such as wavelets
(Donoho & Johnstone 1995; Mallat 1999), for example. More recent approaches
include non-local means ﬁltering (Buades et al. 2005), which exploits image self-
similarities, learned sparse models (Elad & Aharon 2006; Mairal et al. 2009), Gaus-
sian scale mixtures (Portilla et al. 2003), ﬁelds of experts (Agarwal and Roth May
2002), and block matching with 3D ﬁltering (BM3D) (Dabov et al. 2007). The idea
of using self-similarities as a prior for natural images exploited by the non-local
means approach of Buades et al. (2005) has in fact appeared in the literature in
various guises and under diﬀerent equivalent interpretations, e.g., kernel density es-
timation (Efros and Leung 1999), Nadaraya-Watson estimators (Buades et al. 2005),
mean-shift iterations (Awate and Whitaker 2006), diﬀusion processes on graphs
(Szlam et al. 2007), and long-range random ﬁelds (Li and Huttenlocher 2008).
We have restricted our discussion of sparsity-inducing regularizers to the
1
norm
here, but the
0
pseudo-norm, which counts the number of nonzero coeﬃcients
in the code associated with a noisy signal can be used as well. Chapter 22 dis-
cusses
0
-regularized sparse coding and dictionary learning in some detail. Let
us just note here that simultaneous sparse coding is also relevant in that case,
the
1,2
norm being replaced by the
0,∞
pseudo-norm, which directly counts the
number of nonzero rows. See (Mairal et al. 2009) for details. An implemen-
tation of non-local means is available at: http://www.ipol.im/pub/algo/bcm_
non_local_means_denoising/, and BM3D is available at http://www.cs.tut.
fi/
~
foi/GCF-BM3D/. An implementation of LSSC is available at http://www.di.
ens.fr/
~
mairal/denoise_ICCV09.tar.gz.
Shape from Texture
We have assumed that textures are albedo marks on smooth surfaces. This really
isn’t true, as van Ginneken et al. (1999) point out; an immense number of textures
are caused by indentations on surfaces (the bark on a tree, for example, where the
main texture eﬀect seems to be dark shadows in the grooves of the bark), or by
elements suspended in space (the leaves of a tree, say). Such textures still give
us a sense of shape—for example, in Figure 6.1, one has a sense of the free space
in the picture where one could move. The resulting changes in appearance as the
illumination and view directions change are complicated (Dana et al. 1999, Lu et
al. 1999, Lu et al. 1998, Pont and Koenderink 2002). We don’t discuss this case
Section 6.6 Notes 193
because not much is known about how to proceed.
There is a long tradition of using marked point processes as texture models
(explicitly in, for example (Ahuja and Schachter 1983a, Ahuja and Schachter 1983b,
Blake and Marinos 1990, Schachter 1980, Schachter and Ahuja 1979) and implicitly
in pretty much all existing literature). A Poisson model has the property that
the expected number of elements in a domain is proportional to the area of the
domain. The constant of proportionality is known as the model’s intensity. A
texture is isotropic if the choice of element rotation is uniform and random, and is
homogeneous if the density from which texture elements are drawn is independent
of position on the surface.
There are surprisingly few shape from texture methods. Global methods
attempt to recover an entire surface model, using assumptions about the distri-
bution of texture elements. Appropriate assumptions are isotropy (Witkin 1981)
(the disadvantage of this method is that there are relatively few natural isotropic
textures) or homogeneity (Aloimonos 1986, Blake and Marinos 1990).
Texture deformation can be exploited in global methods, with some assump-
tions about the element—see the methods in (Lee and Kuo 1998, Sakai and Finkel
1994, Stone and Isard 1995)). Alternatively, one observes that the per-element
imaging transformations are going to aﬀect the spatial frequency components on
the surface; this means that if the texture has constrained spatial frequency prop-
erties, one may observe the orientation from the texture gradient (Bajcsy and
Lieberman 1976, Krumm and Shafer 1990, Krumm and Shafer 1992, Sakai and
Finkel 1994, Super and Bovik 1995).
Local methods recover some diﬀerential geometric parameters at a point on
a surface (typically, normal and curvatures). This class of methods, which is due
to Garding (1992), has been successfully demonstrated for a variety of surfaces by
Malik and Rosenholtz (1997) and Rosenholtz and Malik (1997); a reformulation in
terms of wavelets is due to Clerc and Mallat (1999). The methods have a crucial
ﬂaw; it is necessary either to know that texture element coordinate frames form
a frame ﬁeld that is locally parallel around the point in question, or to know the
diﬀerential rotation of the frame ﬁeld (see Garding (1995) for this point, which is
emphasized by the choice of textures displayed in Rosenholtz and Malik (1997); the
assumption is known as texture stationarity). For example, if one were to use
these methods to recover the curvature of a doughnut dipped in chocolate sprinkles,
it would be necessary to ensure that the sprinkles were all parallel on the surface
(or that the ﬁeld of angles from sprinkle to sprinkle was known).
One might construct a generative model, where object texture is modelled
with a parametric random model, then choose a geometry and parameters that
minimizes the diﬀerence between either a predicted image and the observed im-
age (Choe and Kashyap 1991) or a predicted image density and the observed image
density (Lee and Kuo 1998).
More recent local methods emphasize repetition. Forsyth (2001) infers shape
from slant estimates only, establishing an analogy with shape from shading. Forsyth
(2002) shows that element repetition is suﬃcient to get normal estimates up to an
ambiguity, with a cleaner version in (Lobay and Forsyth 2006); Loh and Hartley
(2005) give a method to reconstruct a surface in this case; and Lobay and Forsyth
(2004) demonstrate that repetition of textons gives cues to illumination.
Section 6.6 Notes 194
Applications for shape from texture have been largely absent, explaining its
status as a minority interest. However, we believe that image-based rendering of
clothing is an application with substantial promise. Cloth is diﬃcult to model
for a variety of reasons. It is much more resistant to stretch than to bend: this
means that dynamical models result in stiﬀ diﬀerential equations (for example,
see (Terzopolous et al. 1987)) and that it buckles in ﬁne scale, complex folds (for
example, see (Bridson et al. 2002)). However, rendering cloth is an important
technical problem, because people are interesting to look at and most people wear
clothing. A natural strategy for rendering objects that are intrinsically diﬃcult to
model satisfactorily is to rearrange existing pictures of the objects to yield a render-
ing. In particular, one would wish to be able to retexture and reshade such images.
Earlier work on motion capturing cloth used stereopsis, but faced diﬃculties with
motion blur and calibration (Pritchard 2003, Pritchard and Heidrich 2003). More
recent work prints a ﬁne pattern on the cloth (White et al. 2007), or uses volume
intersections (Bradley et al. 2008b). We believe that, in future, shape from texture
methods might make it possible to avoid some of these problems.
PROBLEMS
6.1. Show that a circle appears as an ellipse in an orthographic view, that the minor
axis of this ellipse is the tilt direction, and that the aspect ratio is the cosine
of the slant angle.
6.2. We will study measuring the orientation of a plane in a perspective view, given
that the texture consists of points laid down by a homogeneous Poisson point
process. Recall that one way to generate points according to such a process is
to sample the x and y coordinate of the point uniformly and at random. We
assume that the points from our process lie within a unit square.
(a) Show that the probability that a point will land in a particular set is
proportional to the area of that set.
(b) Assume we partition the area into disjoint sets. Show that the number of
points in each set has a multinomial probability distribution.
We will now use these observations to recover the orientation of the plane. We
partition the image texture into a collection of disjoint sets.
(c) Show that the area of each set, backprojected onto the textured plane, is a
function of the orientation of the plane.
(d) Use this function to suggest a method for obtaining the plane’s orientation.
PROGRAMMING EXERCISES
6.3. Texture synthesis: Implement the non-parametric texture synthesis algo-
rithm of Algorithm 6.4. Use your implementation to study:
(a) the eﬀect of window size on the synthesized texture;
(b) the eﬀect of window shape on the synthesized texture; and
(c) the eﬀect of the matching criterion on the synthesized texture (i.e., using
a weighted sum of squares instead of a sum of squares, etc.).
P A R T T H R E E
EARLY VISION: MULTIPLE
IMAGES
This page intentionally left blank
C H A P T E R 7
Stereopsis
Fusing the pictures recorded by our two eyes and exploiting the diﬀerence (or dis-
parity) between them allows us to gain a strong sense of depth. This chapter is
concerned with the design and implementation of algorithms that mimic our ability
to perform this task, known as stereopsis. Reliable computer programs for stereo-
scopic perception are of course invaluable in visual robot navigation (Figure 7.1),
cartography, aerial reconnaissance, and close-range photogrammetry. They are also
of great interest in tasks such as image segmentation for object recognition or the
construction of three-dimensional scene models for computer graphics applications.
FIGURE 7.1: Left: The Stanford cart sports a single camera moving in discrete increments
along a straight line and providing multiple snapshots of outdoor scenes. Center: The
INRIA mobile robot uses three cameras to map its environment. Right: The NYU
mobile robot uses two stereo cameras, each capable of delivering an image pair. As shown
by these examples, although two eyes are suﬃcient for stereo fusion, mobile robots are
sometimes equipped with three (or more) cameras. The bulk of this chapter is concerned
with binocular perception but stereo algorithms using multiple cameras are discussed in
Section 7.6. Photos courtesy of Hans Moravec, Olivier Faugeras, and Yann LeCun.
Stereo vision involves two processes: The fusion of features observed by two
(or more) eyes and the reconstruction of their three-dimensional preimage. The
latter is relatively simple: The preimage of matching points can (in principle) be
found at the intersection of the rays passing through these points and the associ-
ated pupil centers (or pinholes; see Figure 7.2, left). Thus, when a single image
feature is observed at any given time, stereo vision is easy. However, each picture
typically consists of millions of pixels, with tens of thousands of image features
such as edge elements, and some method must be devised to establish the correct
correspondences and avoid erroneous depth measurements (Figure 7.2, right).
We start this chapter by examining in Section 7.1 the geometric epipolar con-
straint associated with a pair of cameras, which is a key to controlling the cost
of the binocular fusion process. Next, we stay on the geometric side of things in
Section 7.2 as we present a number of methods for binocular reconstruction. After
197
Section 7.1 Binocular Camera Geometry and the Epipolar Constraint 198
d
c
b
a
a’
d’
c’
D
C B
A
p’ p
O’ O
O’ O
P
b’
FIGURE 7.2: The binocular fusion problem: In the simple case of the diagram shown on
the left, there is no ambiguity, and stereo reconstruction is a simple matter. In the more
usual case shown on the right, any of the four points in the left picture may, a priori,
match any of the four points in the right one. Only four of these correspondences are
correct; the other ones yield the incorrect reconstructions shown as small gray discs.
a brief incursion into human stereopsis (Section 7.3), we switch with Section 7.4 to
the presentation of several algorithms for binocular fusion that rely on the compar-
ison of local brightness or edge patterns to establish correspondences. Section 7.5
shows that ordering and smoothness constraints among nearby pixels can be in-
corporated in the matching process. In this setting, stereo fusion is naturally cast
as a combinatorial optimization problem, which can be solved by several eﬃcient
algorithms (Chapter 22). We conclude in Section 7.6 with a discussion of multi-
camera stereo fusion (see also Chapter 19 for applications of multi-view stereopsis
to image-based modeling and rendering).
Note: We assume throughout that all cameras have been carefully calibrated so
their intrinsic and extrinsic parameters are precisely known relative to some ﬁxed
world coordinate system. The case of uncalibrated cameras is examined in the
context of structure from motion in Chapter 8.
7.1 BINOCULAR CAMERA GEOMETRY AND THE EPIPOLAR CONSTRAINT
As noted in the introduction, it appears a priori that, given a stereo image pair,
any pixel in the ﬁrst (or left) image may match any pixel in the second (or right)
one. As shown in this section, matching pairs of pixels are in fact restricted to
lie on corresponding epipolar lines in the two pictures. This constraint plays a
fundamental role in the stereo fusion process because it reduces the quest for image
correspondences to a set of one-dimensional searches.
7.1.1 Epipolar Geometry
Consider the images p and p

of a point P observed by two cameras with optical
centers O and O

. These ﬁve points all belong to the epipolar plane deﬁned by
the two intersecting rays OP and O

P (Figure 7.3). In particular, the point p

lies
on the line l

where this plane and the retina Π

of the second camera intersect.
Section 7.1 Binocular Camera Geometry and the Epipolar Constraint 199
The line l

is the epipolar line associated with the point p, and it passes through
the point e

where the baseline joining the optical centers O and O

intersects Π

.
Likewise, the point p lies on the epipolar line l associated with the point p

, and
this line passes through the intersection e of the baseline with the plane Π.
’
P
e’ e
p’ p
O’ O
l’ l
FIGURE 7.3: Epipolar geometry: The point P, the optical centers O and O

of the two
cameras, and the two images p and p

of P all lie in the same plane. Here, as in the other
ﬁgures of this chapter, cameras are represented by their pinholes and a virtual image
plane located in front of the pinhole. This is to simplify the drawings; the geometric and
algebraic arguments presented in the rest of this chapter hold just as well for physical
image planes located behind the corresponding pinholes.
The points e and e

are called the epipoles of the two cameras. The epipole e

is the projection of the optical center O of the ﬁrst camera in the image observed
by the second camera, and vice versa. As noted before, if p and p

are images of the
same point, then p

must lie on the epipolar line associated with p. This epipolar
constraint plays a fundamental role in stereo vision and motion analysis.
In the setting studied in the rest of this chapter, where the cameras are inter-
nally and externally calibrated, the most diﬃcult part of constructing an artiﬁcal
stereo vision system is to ﬁnd eﬀective methods for establishing correspondences be-
tween the two images—that is, deciding which points in the second picture match
the points in the ﬁrst one. The epipolar constraint greatly limits the search for
these correspondences. Indeed, since we assume that the rig is calibrated, the coor-
dinates of the point p completely determine the ray joining O and p, and thus the
associated epipolar plane OO

p and epipolar line l

. The search for matches can be
restricted to this line instead of the whole image (Figure 7.4). In the motion analy-
sis setting studied in Chapter 8, each camera may be internally calibrated, but the
rigid transformation separating the two camera coordinate systems is unknown. In
this case, the epipolar geometry constrains the set of possible motions.
As shown next, it proves convenient to characterize the epipolar constraint
in terms of the bilinear forms associated with two 3 3 essential and fundamental
matrices.
Section 7.1 Binocular Camera Geometry and the Epipolar Constraint 200
2
1
2
1
p’
P
P
p’
l’ l
O
’
P
e’ e
p’
p
O’
FIGURE 7.4: Epipolar constraint: Given a calibrated stereo rig, the set of possible matches
for the point p is constrained to lie on the associated epipolar line l

.
7.1.2 The Essential Matrix
We assume in this section that the intrinsic parameters of each camera are known,
and work in normalized image coordinates—that is, take p = ˆ p. According to
the epipolar constraint, the three vectors
−→
Op,
−−→
O

p

, and
−−→
OO

must be coplanar.
Equivalently, one of them must lie in the plane spanned by the other two, or
−→
Op [
−−→
OO

−−→
O

p

] = 0.
We can rewrite this coordinate-independent equation in the coordinate frame asso-
ciated to the ﬁrst camera as
p [t (!p

)] = 0, (7.1)
where p and p

denote the homogeneous normalized image coordinate vectors of
p and p

, t is the coordinate vector of the translation
−−→
OO

separating the two
coordinate systems, and ! is the rotation matrix such that a free vector with
coordinates w

in the second coordinate system has coordinates !w

in the ﬁrst
one. In this case, the two projection matrices are given in the coordinate system
attached to the ﬁrst camera by [Id 0] and [!
T
−!
T
t].
Equation (7.1) can ﬁnally be rewritten as
p
T
cp

= 0, (7.2)
where c = [t
×
]!, and [a
×
] denotes the skew-symmetric matrix such that [a
×
]x =
a x is the cross-product of the vectors a and x. The matrix c is called the
essential matrix, and it was ﬁrst introduced by Longuet–Higgins (1981). Its nine
coeﬃcients are only deﬁned up to scale, and they can be parameterized by the
three degrees of freedom of the rotation matrix ! and the two degrees of freedom
deﬁning the direction of the translation vector t.
Note that l = cp

can be interpreted as the coordinate vector of the epipolar
line l associated with the point p

in the ﬁrst image. Indeed, Equation (7.2) can be
written as p l = 0, expressing the fact that the point p lies on l. By symmetry, it
Section 7.2 Binocular Reconstruction 201
is also clear that l

= c
T
p is the coordinate vector representing the epipolar line l

associated with p in the second image. Essential matrices are singular because t is
parallel to the coordinate vector e of the ﬁrst epipole, so that c
T
e = −!
T
[t
×
]e = 0.
Likewise, it is easy to show that e

is in the nullspace of c. As shown by Huang
and Faugeras (1989), essential matrices are in fact characterized by the fact that
they are singular with two equal nonzero singular values (see the problems).
7.1.3 The Fundamental Matrix
The Longuet–Higgins relation holds in normalized image coordinates. In native
image coordinates, we can write p = /ˆ p and p

= /

ˆ p

, where / and /

are the
3 3 calibration matrices associated with the two cameras. The Longuet–Higgins
relation holds for these vectors, and we obtain
p
T
Tp

= 0, (7.3)
where the matrix T = /
−T
c/

−1
, called the fundamental matrix, is not, in general,
an essential matrix. It has again rank two, and the eigenvector of T (resp. T
T
) cor-
responding to its zero eigenvalue is as before the position e

(resp. e) of the epipole.
Likewise, l

= Tp

(resp. l = T
T
p) represents the epipolar line corresponding to
the point p

(resp. p) in the ﬁrst (resp. second) image.
The matrices c and T can readily be computed from the intrinsic and extrinsic
parameters. Let us close this section by noting that Equations (7.2) and (7.3) also
provide constraints on the entries of these matrices, irrespective of the 3D position
of the observed points. In particular, this suggests that c and T can be computed
from a suﬃcient number of image correspondences without the use of a calibration
chart. We will come back to this issue in Chapter 8. For the time being, we will
assume that the cameras are calibrated and that the epipolar geometry is known.
7.2 BINOCULAR RECONSTRUCTION
Given a calibrated stereo rig and two matching image points p and p

, it is in prin-
ciple straightforward to reconstruct the corresponding scene point by intersecting
the two rays R = Op and R

= O

p

(Figure 7.2). However, the rays R and R

never
actually intersect in practice, due to calibration and feature localization errors. In
this context, various reasonable approaches to the reconstruction problem can be
adopted. For example, consider the line segment perpendicular to R and R

that
intersects both rays (Figure 7.5): its mid-point P is the closest point to the two
rays and can be taken as the preimage of p and p

.
Alternatively, one can reconstruct a scene point using a purely algebraic ap-
proach: given the projection matrices / and /

and the matching points p and
p

, we can rewrite the constraints Zp = /P and Z

p

= /P as
_
p /P = 0
p

/

P = 0
⇐⇒
_
[p
×
]/
[p

×
]/

_
P = 0.
This is an overconstrained system of four independent linear equations in the ho-
mogeneous coordinates of P that is easily solved using the linear least-squares tech-
niques introduced in Chapter 22. Unlike the previous approach, this reconstruction
Section 7.2 Binocular Reconstruction 202
’
O’
q
q’
R
R’
p
P
p’
O
Q
FIGURE 7.5: Triangulation in the presence of measurement errors. See text for details.
method does not have an obvious geometric interpretation, but generalizes readily
to the case of three or more cameras, with each new picture simply adding two
additional constraints.
Finally, one can reconstruct the scene point associated with p and p

as the
point Q with images q and q

that minimizes d
2
(p, q) +d
2
(p

, q

) (Figure 7.5). Un-
like the two other methods presented in this section, this approach does not allow
the closed-form computation of the reconstructed point, which must be estimated
via nonlinear least-squares techniques such as those introduced in Chapter 22. The
reconstruction obtained by either of the other two methods can be used as a rea-
sonable guess to initialize the optimization process. This nonlinear approach also
readily generalizes to the case of multiple images.
7.2.1 Image Rectiﬁcation
The calculations associated with stereo algorithms are often considerably simpliﬁed
when the images of interest have been rectiﬁed—that is, replaced by two equivalent
pictures with a common image plane parallel to the baseline joining the two optical
centers (Figure 7.6). The rectiﬁcation process can be implemented by projecting
the original pictures onto the new image plane. With an appropriate choice of
coordinate system, the rectiﬁed epipolar lines are scanlines of the new images, and
they are also parallel to the baseline. There are two degrees of freedom involved
in the choice of the rectiﬁed image plane: (a) the distance between this plane and
the baseline, which is essentially irrelevant because modifying it only changes the
scale of the rectiﬁed pictures—an eﬀect easily balanced by an inverse scaling of the
image coordinate axes; and (b) the direction of the rectiﬁed plane normal in the
plane perpendicular to the baseline. Natural choices include picking a plane parallel
to the line where the two original retinas intersect and minimizing the distortion
associated with the reprojection process.
In the case of rectiﬁed images, the informal notion of disparity introduced at
Section 7.3 Human Stereopsis 203
l
p’ p
’
l’
’
p p’
l’ l
O O’
e e’
P
FIGURE 7.6: A rectiﬁed stereo pair: The two image planes Π and Π

are reprojected onto a
common plane
¯
Π =
¯
Π

parallel to the baseline. The epipolar lines l and l

associated with
the points p and p

in the two pictures map onto a common scanline
¯
l =
¯
l

also parallel to
the baseline and passing through the reprojected points ¯ p and ¯ p

. With modern computer
graphics hardware and software, the rectiﬁed images are easily constructed by considering
each input image as a polyhedral mesh and using texture mapping to render the projection
of this mesh onto the plane
¯
Π =
¯
Π

.
the beginning of this chapter takes a concrete meaning: given two points p and p

located on the same scanline of the left and right images, with coordinates (x, y) and
(x

, y), the disparity is deﬁned as the diﬀerence d = x

− x. We assume in the rest
of this section that image coordinates are normalized—that is, as before, p = ˆ p. As
shown in the problems, if B denotes the distance between the optical centers, also
called the baseline in this context, the depth of P in the (normalized) coordinate
system attached to the ﬁrst camera is Z = −B/d. In particular, the coordinate
vector of the point P in the frame attached to the ﬁrst camera is P = −(B/d)p,
where p = (x, y, 1)
T
is the vector of normalized image coordinates of p. This
provides yet another reconstruction method for rectiﬁed stereo pairs.
7.3 HUMAN STEREOPSIS
Before moving on to algorithms for establishing binocular correspondences, let us
pause for a moment to discuss the mechanisms underlying human stereopsis. First,
it should be noted that, unlike the cameras rigidly attached to a passive stereo rig,
the two eyes of a person can rotate in their sockets. At each instant, they ﬁxate on
a particular point in space (i.e., they rotate so that the corresponding images form
in the centers of their foveas). Figure 7.7 illustrates a simpliﬁed, two-dimensional
situation: if l and r denote the (counterclockwise) angles between the vertical planes
of symmetry of two eyes and two rays passing through the same scene point, we
deﬁne the corresponding disparity as d = r − l. It is an elementary exercise in
trigonometry to show that d = D − F, where D denotes the angle between these
Section 7.3 Human Stereopsis 204
"
F
D
Disparate dot
Fixated dot
l
r
Vieth-Muller Circle
FIGURE 7.7: In this diagram, the close-by dot is ﬁxated by the eyes, and it projects onto
the center of their foveas with no disparity. The two images of the far dot deviate from
this central position by diﬀerent amounts, indicating a diﬀerent depth.
rays, and F is the angle between the two rays passing through the ﬁxated point.
Points with zero disparity lie on the Vieth–M¨ uller circle that passes through the
ﬁxated point and the optical centers of the eyes. Points lying inside this circle
have a positive disparity, points lying outside it have, as in Figure 7.7, a negative
disparity, and the locus of all points having a given disparity d forms, as d varies,
the family of all circles passing through the two eyes’ optical centers. This property
is clearly suﬃcient to rank order dots that are near the ﬁxation point according to
their depth. However, it is also clear that the vergence angles between the vertical
median plane of symmetry of the head and the two ﬁxation rays must be known to
reconstruct the absolute position of scene points.
The three-dimensional case is naturally more complicated, with the locus of
zero-disparity points becoming a surface, the horopter, but the general conclu-
sion is the same, and absolute positioning requires the vergence angles. There is
some evidence that these angles cannot be measured accurately by our nervous sys-
tem (Helmholtz 1909). However, relative depth, or rank ordering of points along the
line of sight, can be judged quite accurately. For example, it is possible to decide
which one of two targets near the horopter is closer to an observer for disparities of a
few seconds of arc (stereoacuity threshold), which matches the minimum separation
that can be measured with one eye (monocular hyperacuity threshold).
Concerning the construction of correspondences between the left and right
images, Julesz (1960) asked the following question: Is the basic mechanism for
binocular fusion a monocular process (where local brightness patterns [micropat-
terns] or higher organizations of points into objects [macropatterns] are identiﬁed
before being fused), a binocular one (where the two images are combined into a
single ﬁeld where all further processing takes place), or a combination of both? To
settle this matter, he introduced a new device, the random dot stereogram: a pair of
synthetic images obtained by randomly spraying black dots on white objects, typi-
cally one (or several) small square plate(s) ﬂoating over a larger one (Figure 7.8).
The results were striking. To quote Julesz: “When viewed monocularly, the im-
ages appear completely random. But when viewed stereoscopically, the image pair
Section 7.4 Local Methods for Binocular Fusion 205
FIGURE 7.8: From left to right: the two pictures forming a random dot stereogram that
depicts four planes at varying depth (a “wedding cake”), and the disparity map obtained
by the Marr-Poggio (1976) algorithm. The layered structure of the scene is correctly
recovered. Reprinted from Vision: A Computational Investigation into the Human Repre-
sentation and Processing of Visual Information, by David Marr, c _ 1982 by David Marr.
Reprinted by permission of Henry Holt and Company, LLC.
gives the impression of a square markedly in front of (or behind) the surround.”
The conclusion is clear: Human binocular fusion cannot be explained by peripheral
processes directly associated with the physical retinas. Instead, it must involve the
central nervous system and an imaginary cyclopean retina that combines the left
and right image stimuli as a single unit.
Several cooperative models of human stereopsis—where near-by matches in-
ﬂuence each other to avoid ambiguities and promote a global scene analysis—have
been proposed, including Julesz’s own dipole model (1960) and that of Marr and
Poggio (1976). Although the latter has been implemented, allowing the reliable
fusion of random dot stereograms (Figure 7.8), it fails on most natural images. In
contrast, the algorithms proposed in the following sections do not attempt to model
the human visual system, but they usually give good results on natural imagery.
7.4 LOCAL METHODS FOR BINOCULAR FUSION
We start here by introducing simple methods for stereo fusion that exploit purely
local information, such as the similarity of brightness patterns near candidate
matches, to establish correspondences.
7.4.1 Correlation
Correlation methods ﬁnd pixel-wise image correspondences by comparing intensity
proﬁles in the neighborhood of potential matches, and they are among the ﬁrst
techniques ever proposed to solve the binocular fusion problem (Kelly, McConnell
& Mildenberger 1977; Gennery 1980). Concretely, let us consider a rectiﬁed stereo
pair and a point (x, y) in the ﬁrst image (Figure 7.9). We associate with the window
of size p = (2m+1) (2n +1) centered in (x, y) the vector w(x, y) ∈ R
p
obtained
by scanning the window values one row at a time (the order is in fact irrelevant as
long as it is ﬁxed). Now, given a potential match (x+d, y) in the second image, we
can construct a second vector w

(x+d, y) and deﬁne the corresponding normalized
Section 7.4 Local Methods for Binocular Fusion 206
u
v
p
w
1
w w
2
w’-w’
w-w
v
u+d
w’ w
FIGURE 7.9: Correlation of two 3 5 windows along corresponding epipolar lines. The
second window position is separated from the ﬁrst one by an oﬀset d. The two windows
are encoded by vectors w and w

in R
15
, and the correlation function measures the cosine
of the angle θ between the vectors w− ¯ w and w

)],
where the x, y, and d indexes have been omitted for the sake of conciseness and ¯ a
denotes the vector whose coordinates are all equal to the mean of the coordinates
of a.
The normalized correlation function C clearly ranges from −1 to +1. It
reaches its maximum value when the image brightnesses of the two windows are
related by an aﬃne transformation I

= λI + μ for some constants λ and μ with
λ > 0 (see the problems). In other words, maxima of this function correspond
to image patches separated by a constant oﬀset and a positive scale factor, and
stereo matches can be found by seeking the maximum of the C function over some
predetermined range of disparities.
1
At this point, let us make a few remarks about matching methods based
on correlation. First, it is easily shown (see the problems) that maximizing the
1
The invariance of C to aﬃne transformations of the brightness function aﬀords correlation-
based matching techniques some degree of robustness in situations where the observed surface is
not quite Lambertian or the two cameras have diﬀerent gains or lenses with diﬀerent f-numbers.
Section 7.4 Local Methods for Binocular Fusion 207
O’ O
l’ l
L
FIGURE 7.10: The foreshortening of an oblique plane is not the same for the left and right
cameras: l/L ,= l

)[
2
,
or equivalently the sum of the squared diﬀerences between the pixel values of the
two windows after they have been submitted to the corresponding normalization
process. Second, although the calculation of the normalized correlation function at
every pixel of an image for some range of disparities is computationally expensive,
it can be implemented eﬃciently using recursive techniques (see problems). Third,
other functions, such as the sum of absolute diﬀerence

p
i=1
[w
i
−w

i
[, can be used
to measure the discrepancy between two brightness patterns, and they may give
better results in certain situations (Scharstein and Szeliski 2002). Finally, a major
problem with correlation-based techniques for establishing stereo correspondences
is that they implicitly assume that the observed surface is (locally) parallel to the
two image planes, since the foreshortening of (oblique) surfaces depends on the
position of the cameras observing them (Figure 7.10).
This suggests a two-pass algorithm where initial estimates of the disparity
are used to warp the correlation windows to compensate for unequal amounts of
foreshortening in the two pictures. For example, Devernay and Faugeras (1994)
propose to deﬁne a warped window in the right image for each rectangle in the
left one, using the disparity in the center of the rectangle and its derivatives. An
optimization process is used to ﬁnd the values of the disparity and its derivatives
that maximize the correlation between the left rectangle and the right window,
using interpolation to retrieve appropriate values in the right image. Figure 7.11
illustrates this approach with an example.
7.4.2 Multi-Scale Edge Matching
Slanted surfaces pose problems to correlation-based matchers. Other arguments
against correlation can be found in Julesz (1960) and Marr (1982), suggesting that
correspondences should be found at a variety of scales, with matches between (hope-
fully) physically signiﬁcant image features such as edges preferred to matches be-
Section 7.4 Local Methods for Binocular Fusion 208
(a) (b) (c)
FIGURE 7.11: Correlation-based stereo matching: (a) a pair of stereo pictures; (b) a
texture-mapped view of the reconstructed surface; (c) comparison of the regular (left)
and reﬁned (right) correlation methods in the nose region. The latter clearly gives better
results. Reprinted from “Computing Diﬀerential Properties of 3D Shapes from Stereopsis
Without 3D Models,” by F. Devernay and O.D. Faugeras, Proc. IEEE Conference on
Computer Vision and Pattern Recognition, (1994). c _ 1994 IEEE.
tween raw pixel intensities. These principles are implemented in Algorithm 7.1,
which is due to Marr and Poggio (1979).
1. Convolve the two (rectiﬁed) images with ∇
2
G
σ
ﬁlters of increasing standard
deviations σ
1
< σ
2
< σ
3
< σ
4
.
2. Find zero crossings of the Laplacian along horizontal scanlines of the ﬁltered
images.
3. For each ﬁlter scale σ, match zero crossings with the same parity and roughly
equal orientations in a [−w
σ
, +w
σ
] disparity range, with w
σ
= 2
√
2σ.
4. Use the disparities found at larger scales to oﬀset the images in the neighbor-
hood of matches and cause unmatched regions at smaller scales to come into
correspondence.
Algorithm 7.1: The Marr–Poggio (1979) Multi-Scale Binocular Fusion Algorithm.
Section 7.4 Local Methods for Binocular Fusion 209
Matching zero crossings at multiple scales
Matching zero crossings at a single scale
Match
Match
Rematch
<
’
Offset
Width
Scale
Width
Scale
FIGURE 7.12: Top: Single-Scale matching. Middle: Multi-Scale matching. Bottom:
Results. Bottom left: The input data (including one of the input pictures, the out-
put of four ∇
2
G
σ
ﬁlters, and the corresponding zero crossings). Bottom right: Two
views of the disparity map constructed by the matching process and two views of the
surface obtained by interpolating the reconstructed points. Reprinted from Vision: A
Computational Investigation into the Human Representation and Processing of Visual In-
formation, by David Marr, c _ 1982 by David Marr. Reprinted by permission of Henry
Holt and Company, LLC.
Section 7.5 Global Methods for Binocular Fusion 210
Matches are sought at each scale in the [−w
σ
, w
σ
] disparity range, where w
σ
=
2
√
2σ is the width of the central negative portion of the ∇
2
G
σ
ﬁlter. This choice is
motivated by psychophysical and statistical considerations. In particular, assuming
that the convolved images are white Gaussian processes, Grimson (1981a) showed
that the probability of a false match occurring in the [−w
σ
, +w
σ
] disparity range
of a given zero crossing is only 0.2 when the orientations of the matched features
are within 30
◦
of each other. A simple mechanism can be used to disambiguate
the multiple potential matches that might still occur within the matching range.
See Grimson (1981a) for details. Of course, limiting the search for matches to
the [−w
σ
, +w
σ
] range prevents the algorithm from matching correct pairs of zero
crossings whose disparity falls outside this interval. Since w
σ
is proportional to
the scale σ at which matches are sought, eye movements (or equivalently image
oﬀsets) controlled by the disparities found at large scales must be used to bring
large-disparity pairs of zero crossings within matchable range at a ﬁne scale. This
process occurs in Step 4 of Algorithm 7.1 and is illustrated by Figure 7.12 (top).
Once matches have been found, the corresponding disparities can be stored in
a buﬀer called the 2
1
2
-dimensional sketch by Marr and Nishihara (1978). This
algorithm has been implemented by Grimson (1981a), and extensively tested on
random dot stereograms and natural images. An example appears in Figure 7.12
(bottom).
7.5 GLOBAL METHODS FOR BINOCULAR FUSION
The stereo fusion techniques presented in the previous section are purely local, in
the sense that they match brightness or edge patterns around individual pixels,
but ignore the constraints that may link nearby points. In contrast, we present in
this section two global approaches to stereo fusion, that formulate this problem as
the minimization of a single energy function incorporating ordering or smoothness
constraints among adjacent pixels.
7.5.1 Ordering Constraints and Dynamic Programming
It is reasonable to assume that the order of matching image features along a pair
of epipolar lines is the inverse of the order of the corresponding surface attributes
along the curve where the epipolar plane intersects the observed object’s boundary
(Figure 7.13, left). This is the so-called ordering constraint introduced in the early
1980s (Baker & Binford 1981; Ohta & Kanade 1985). Interestingly enough, it might
not be satisﬁed by real scenes, in particular when small solids occlude parts of larger
ones (Figure 7.13, right) or more rarely, at least in robot vision, when transparent
objects are involved. Despite these reservations, the ordering constraint remains a
reasonable one, and it can be used to devise eﬃcient algorithms relying on dynamic
programming (Forney 1973; Aho, Hopcroft, & Ullman 1974) to establish stereo
correspondences (see Figure 7.14 and Algorithm 7.2).
Speciﬁcally, let us assume that a number of feature points (say, edgels) have
been found on corresponding epipolar lines. Our objective here is to match the
intervals separating those points along the two intensity proﬁles (Figure 7.14, left).
According to the ordering constraint, the order of the feature points must be the
same, although the occasional interval in either image may be reduced to a single
Section 7.5 Global Methods for Binocular Fusion 211
d
a
d’
c’
O’ O’ O
A
B
C
a
b
c a’
c’
A
B
C
D
O
b’
b b’
FIGURE 7.13: Ordering constraints. In the (usual) case shown in the left part of the
diagram, the order of feature points along the two (oriented) epipolar lines is the same.
In the case shown in the right part of the ﬁgure, a small object lies in front of a larger
one. Some of the surface points are not visible in one of the images (e.g., A is not visible
in the right image), and the order of the image points is not the same in the two pictures:
b is on the right of d in the left image, but b

is on the left of d

in the right image.
point corresponding to missing correspondences associated with occlusion and/or
noise.
This setting allows us to recast the matching problem as the optimization
of a path’s cost over a graph whose nodes correspond to pairs of left and right
image features; and arcs represent matches between left and right intensity pro-
ﬁle intervals bounded by the features of the corresponding nodes (Figure 7.14,
right). The cost of an arc measures the discrepancy between the corresponding in-
tervals (e.g., the squared diﬀerence of the mean intensity values). This optimization
problem can be solved, exactly and eﬃciently, using dynamic programming (Algo-
rithm 7.2). As given, this algorithm has a computational complexity of O(mn),
where m and n denote the number of edge points on the matched left and right
scanlines, respectively.
2
Variants of this approach have been implemented by Baker
and Binford (1981), who combine a coarse-to-ﬁne intra-scanline search procedure
with a cooperative process for enforcing inter-scanline consistency, and Ohta and
Kanade (1985), who use dynamic programming for both intra- and inter-scanline
optimization, the latter procedure being conducted in a three-dimensional search
space. Figure 7.15 shows a sample result taken from Ohta and Kanade (1985).
7.5.2 Smoothness Constraints and Combinatorial Optimization over Graphs
Dynamic programming is a combinatorial optimization algorithm aimed at mini-
mizing an error function (a path cost) over some discrete variables (correspondences
between pairs of features). It was used in the previous section to incorporate or-
dering constraints in the matching process. We now present a diﬀerent approach to
stereo fusion that relies instead on smoothness constraints, and a diﬀerent combina-
torial optimization technique aimed at minimizing certain energy functions deﬁned
over graphs.
2
Our version of the algorithm assumes that all edges are matched. To account for noise and
edge-detection errors, it is reasonable to allow the matching algorithm to skip a bounded number
of edges, but this does not change its asymptotic complexity (Ohta and Kanade 1985).
Section 7.5 Global Methods for Binocular Fusion 212
u
I
6
5
4
3
2
1
6 5 4 3 2 1
5
4
3
2 1
6
5
4
3
2
1
6
FIGURE 7.14: Dynamic programming and stereopsis: The left part of the ﬁgure shows
two intensity proﬁles along matching epipolar lines. The polygons joining the two proﬁles
indicate matches between successive intervals (some of the matched intervals may have
zero length). The right part of the diagram represents the same information in graphical
form: an arc (thick line segment) joins two nodes (i, i

(q +dp)]
2
, where A(p) is some neighborhood
of p. The second one is used to regularize the optimization process, making sure
that the disparity function is smooth enough. For example, a sensible choice may
be B
pq
(d
p
, d
q
) = γ
pq
[d
p
−d
q
[ for some γ
pq
> 0.
Under this model, binocular fusion can be formulated as the minimization of
E(d) with respect to d in T
n
. As discussed in Chapter 22 (Section 22.4), this is
a particular instance of a general combinatorial optimization problem, related to
maximum a posteriori (MAP) inference in ﬁrst-order Markov random ﬁelds (Geman
and Geman 1984), which is in general NP-hard but admits eﬀective approximate
and even exact algorithmic solutions under certain so-called submodularity assump-
tions. In particular, it can be shown (Ishikawa 2003; Schlesinger & Flach 2006; Dar-
bon 2009) that when B
pq
(d
p
, d
q
) = γ
pq
[d
p
− d
q
[ for some γ
pq
> 0 (total-variation
prior) or, more generally, when B
pq
= g(d
p
− d
q
) for some convex real function
g : Z → R, minimizing E(d) reduces to a submodular quadratic pseudo-Boolean
problem that involves only binary variables and can be solved exactly in polynomial
3
Here we abuse the notation and, if the images coordinates of pixel p are (u
p
, v
p
), denote by
p + d
p
the pixel with coordinates (u
p
+ d
p
, v
p
).
Section 7.6 Using More Cameras 214
FIGURE 7.16: An application of alpha expansion to stereo fusion. The data used in
this experiment is part of the benchmark described in Scharstein and Szeliski (2002), for
which ground truth disparities are available. From left to right: Input image, ground
truth disparities, and disparities recovered using alpha expansion. Reprinted from “A
Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,” by
D. Scharstein and R. Szeliski, International Journal of Computer Vision, 47(1/2/3):7–42,
(2002). c _ 2002 Springer.
time by an eﬃcient min-cut/max-ﬂow algorithm (Ford & Fulkerson 1956; Goldberg
& Tarjan 1988; Boykov & Kolmogorov 2004).
In practice, however, it may prove important to use binary terms that do
not lead to submodular problems and thus cannot be solved in an exact manner.
The Potts model, where B
pq
(d
p
, d
q
) = γ
pq
χ(d
p
,= d
q
), the characteristic function
χ is one if its argument is true and zero otherwise, and γ
pq
> 0, is a typical
example. Using it instead of, say, a total-variation prior to encourage the disparity
function to be smooth, does not overpenalize the disparity discontinuities naturally
associated with occlusion boundaries. In this setting, an approximate solution to
the minimization of E(d) over T
n
can be found using alpha expansion (Boykov et al.
2001), an iterative procedure that also solves a min-cut/max-ﬂow problem at each
step, but makes weaker assumptions on the energy function it minimizes. Figure
7.16 shows the result of an experiment using this approach, taken from Scharstein
and Szeliski (2002).
7.6 USING MORE CAMERAS
Adding a third camera eliminates (in large part) the ambiguity inherent in two-
view point matching. In essence, the third image can be used to check hypothetical
matches between the ﬁrst two pictures (Figure 7.17): The three-dimensional point
associated with such a match is ﬁrst reconstructed and then reprojected into the
third image. If no compatible point lies nearby, then the match must be wrong.
In most trinocular stereo algorithms, potential correspondences are hypothe-
Section 7.7 Application: Robot Navigation 215
2
6
5 4
2
2
2
2
2
3 3 3 3
a
c
d
a b c d
1
1
1
1
d
3
O
O
1
3 1
c
b
a
D
C B
A
O
b
FIGURE 7.17: The small gray discs indicate the incorrect reconstructions associated with
the left and right images of four points. The addition of a central camera removes the
matching ambiguity: none of the corresponding rays intersects any of the six discs. Alter-
natively, matches between points in the ﬁrst two images can be checked by reprojecting
the corresponding three-dimensional point in the third image. For example, the match
between b
1
and a
2
is obviously wrong because there is no feature point in the third image
near the reprojection of the hypothetical reconstruction numbered 1 in the diagram.
sized using two of the images, then conﬁrmed or rejected using the third one. In
contrast, Okutami and Kanade (1993) have proposed to ﬁnd matches simultane-
ously in three or more pictures. The basic idea is simple, but elegant: assuming
that all the images have been rectiﬁed, the search for the correct disparities is
replaced by a search for the correct depth, or rather its inverse. Of course, the in-
verse depth is proportional to the disparity for each camera, but the disparity varies
from camera to camera, and the inverse depth can be used as a common search
index. Picking the ﬁrst image as a reference, Okutami and Kanade add the sums
of squared diﬀerences associated with all other cameras into a global evaluation
function E (as shown earlier, this is of course equivalent to adding the correlation
functions associated with the images).
Figure 7.18 plots the value of E as a function of inverse depth for various
subsets of 10 cameras observing a scene that contains a repetitive pattern (Fig-
ure 7.19). In that case, using only two or three cameras does not yield a single,
well-deﬁned minimum. However, adding more cameras provides a clear minimum
corresponding to the correct match. Figure 7.19 shows a sequence of 10 rectiﬁed
images and a plot of the surface reconstructed by the algorithm.
7.7 APPLICATION: ROBOT NAVIGATION
Applications of wide-baseline multi-view stereopsis to the construction of three-
dimensional object and scene models are discussed in Chapter 19. Let us brieﬂy
discuss here an application of binocular stereo vision to navigation for the robot
shown in Figure 7.1 (right). The system described in Hadsell et al. (2009) and Ser-
manet et al. (2009) uses two Point Grey Bumblebee stereo cameras, each capable
of delivering a pair of 1024 768 color images at 15 frames per second, and runs a
Section 7.8 Notes 216
0
5000
10000
15000
20000
25000
30000
35000
40000
e
v
a
l
u
a
t
i
o
n
f
u
n
c
t
i
o
n
0 5 10 15 20
inverse distance
B=8b
B=4b,8b
B=2b,4b,6b,8b
B=b,2b,...,8b
FIGURE 7.18: Combining multiple views: The sum of squared diﬀerences is plotted here
as a function of the inverse depth for various numbers of input pictures. The data are
taken from a scanline near the top of the images shown in Figure 7.19, whose intensity
is nearly periodic. The diagram clearly shows that the minimum of the function becomes
less and less ambiguous as more images are added. Reprinted from “A Multiple-Baseline
Stereo System,” by M. Okutami and T. Kanade, IEEE Transactions on Pattern Analysis
and Machine Intelligence, 15(4):353–363, (1993). c _ 1993 IEEE.
separate binocular stereo process for each pair (Figure 7.20). The fusion algorithm
itself is local, and uses the sum of absolute diﬀerences as a matching criterion, with
additional heuristics to ﬁlter out outliers. The ground plane is then found by a
voting procedure before obstacles are detected, based on the recovered point cloud
distribution. The overall process runs at 5–10 160 120 frames per second, but its
useful range is limited to 5 meters. A slower program (one 512 384 frame per sec-
ond), combining stereo vision with convolutional nets used for classiﬁcation, yields
useful depth measurements for distances up to 12 meters, and detects obstacles up
to 50 meters away. The overall system has been successfully used to drive the robot
in ﬁeld experiments with many outdoor settings, including parks and backyards,
open ﬁelds, urban and suburban environments, military bases, sandy areas near
beaches, forests with and without paths, etc. See (Hadsell et al. 2009; Sermanet et
al. 2009) for details.
7.8 NOTES
The essential matrix as an algebraic form of the epipolar constraint was introduced
in the computer vision community by Longuet-Higgins (1981), and its properties
have been elucidated by Huang and Faugeras (1989). The fundamental matrix was
introduced by Luong and Faugeras (1992, 1996). Just as a bilinear constraint holds
for the image coordinates of two point matches, trilinear constraints hold among
matching triples of points (Hartley 1997) and lines (Spetsakis & Aloimonos 1990;
Weng, Huang & Ahuja 1992; Shashua 1995), and quadrilinear constraints also hold
among matching quadruples of points (Faugeras and Mourrain 1995; Triggs 1995;
Faugeras & Papadopoulo 1997). See the problems for some examples. Similar con-
straints have also been studied for decades in the photogrammetry domain (Slama
et al. 1980).
The fact that disparity gives rise to stereopsis in human beings was ﬁrst
Section 7.8 Notes 217
FIGURE 7.19: A series of 10 images and the corresponding reconstruction. The gridboard
near the top of the images is the source for the nearly periodic brightness signal giving
rise to ambiguities in Figure 7.18. Reprinted from “A Multiple-Baseline Stereo System,”
by M. Okutami and T. Kanade, IEEE Transactions on Pattern Analysis and Machine
Intelligence, 15(4):353–363, (1993). c _ 1993 IEEE.
demonstrated by Wheatstone’s (1838) invention of the stereoscope. That disparity
is suﬃcient for stereopsis without eye movements was demonstrated shortly after-
ward by Dove (1841) with illumination provided by an electric spark too brief for eye
vergence to take place. Human stereopsis is further discussed in the classical book of
Helmholtz (1909), an amazing read for anyone interested in the history of the ﬁeld,
as well as the books by Julesz (1960, 1971), Frisby (1980), and Marr (1982). The-
ories of human binocular perception not presented in this chapter for lack of space
include Koenderink and Van Doorn (1976a), Pollard, Mayhew, and Frisby (1970),
McKee, Levi, and Brown (1990), and Anderson and Nayakama (1994).
Excellent treatments of machine stereopsis can be found in the books of Grim-
son (1981b), Marr (1982), Horn (1986), and Faugeras (1993). Marr focuses on the
computational aspects of human stereo vision, whereas Horn’s account emphasizes
the role of photogrammetry in artiﬁcial stereo systems. Grimson and Faugeras
emphasize the geometric and algorithmic aspects of stereopsis. The constraints
associated with stereo matching are discussed by Binford (1984). Early techniques
for line matching in binocular stereo include Medioni and Nevatia (1984) and Ay-
ache and Faugeras (1987). Algorithms for trinocular fusion include Milenkovic
and Kanade (1985), Yachida, Kitamura, and Kimachi (1986), Ayache and Lust-
man (1987), and Robert and Faugeras (1991). Global approaches to dense stereo
fusion based on combinatorial optimization and the underlying min-cut/max-ﬂow
algorithms include Ishikawa and Geiger (1998), Roy and Cox (1998), Boykov, Vek-
sler, and Zabih (2001), and Kolgomorov and Zabih (2001). Variational approaches
have also been used in this context, see Faugeras and Keriven (1998) for example.
All of the algorithms presented in this chapter (implicitly) assume that the
images being fused are quite similar. This is equivalent to considering a narrow base-
line. The wide-baseline case is treated in Chapter 19 in the context of image-based
modeling and rendering. We have also limited our attention here to stereo rigs with
Section 7.8 Notes 218
FIGURE 7.20: Robot navigation using the approach proposed in Hadsell et al. (2009)
and Sermanet et al. (2009). The detected ground plane (lighter shade) and obstacles
(darker one) are overlaid on one of the input images as well as a top view of the stereo
reconstruction. Image courtesy of Yann LeCun.
ﬁxed intrinsic and extrinsic parameters. Active vision is concerned with the con-
struction of vision systems capable of dynamically modifying these parameters, e.g.,
changing camera zoom and vergence angles, and taking advantage of these capa-
bilities in perceptual and robotic tasks (Aloimonos, Weiss & Bandyopadhyay 1987;
Bajcsy 1988; Ahuja & Abbott 1993; Brunnstr¨om, Ekhlund, & Uhlin 1996).
Finally, let us mention the very useful resource assembled by D. Scharstein
and R. Szeliski at http://vision.middlebury.edu/stereo/. One can ﬁnd there
benchmark data, an evaluation of various algorithms on this data, and code for
many classical approaches to stereo fusion. See the web-site and Scharstein and
Szeliski (2002) for details.
PROBLEMS
7.1. Show that one of the singular values of an essential matrix is 0 and the other
two are equal. (Huang and Faugeras [1989] have shown that the converse is
also true; that is, any 3 3 matrix with one singular value equal to 0 and the
other two equal to each other is an essential matrix.)
Hint: The singular values of c are the eigenvalues of cc
T
(Chapter 22).
7.2. Inﬁnitesimal epipolar geometry. Here we consider the case of inﬁnitesimal cam-
era displacements, and derive the instantaneous form of the Longuet–Higgins
relation, Equation (7.2), which captures the epipolar geometry in the discrete
case.
(a) We consider a moving camera with translational velocity v and rotational
velocity ω. The matrix associated with the rotation whose axis is the unit
vector a and whose angle is θ can be shown to be equal to
1 = e
θ[a
×
]
def
=
+∞

= p +δt ˙ p.
(7.5)
(b) Use this result to show that Equation (7.2) reduces to
p
T
([v
×
][ω
×
])p −(p ˙ p) v = 0. (7.6)
for inﬁnitesimal motions.
7.3. The focus of expansion. Consider an inﬁnitesimal translational motion (ω =
0). We deﬁne the focus of expansion (or inﬁnitesimal epipole) as the point
where the line passing through the optical center and parallel to the velocity
vector v pierces the image plane. Use Equation (7.6) to show that the motion
ﬁeld points toward the focus expansion in this pure translational case.
7.4. Show that, in the case of a rectiﬁed pair of images, the depth of a point P in
the normalized coordinate system attached to the ﬁrst camera is Z = −B/d,
where B is the baseline and d is the disparity.
7.5. Use the deﬁnition of disparity to characterize the accuracy of stereo recon-
struction as a function of baseline and depth.
7.6. Give reconstruction formulas for verging eyes in the plane.
7.7. Give an algorithm for generating an ambiguous random dot stereogram that
can depict two diﬀerent planes hovering over a third one.
7.8. Show that the correlation function reaches its maximum value of 1 when the
image brightnesses of the two windows are related by the aﬃne transform
I

= λI +μ for some constants λ and μ with λ > 0.
7.9. Prove the equivalence of correlation and sum of squared diﬀerences for images
with zero mean and unit Frobenius norm.
7.10. Recursive computation of the correlation function.
(a) Show that (w− ¯ w) (w

− ¯ w

) = w w

−(2m+ 1)(2n + 1)
¯
I
¯
I

.
(b) Show that the average intensity
¯
I can be computed recursively, and esti-
mate the cost of the incremental computation.
(c) Generalize the prior calculations to all elements involved in the construc-
tion of the correlation function, and estimate the overall cost of correlation
over a pair of images.
7.11. Show how a ﬁrst-order expansion of the disparity function for rectiﬁed images
can be used to warp the window of the right image corresponding to a rectan-
gular region of the left one. Show how to compute correlation in this case using
interpolation to estimate right-image values at the locations corresponding to
the centers of the left window’s pixels.
7.12. Trifocal and quadrifocal matching constraints. We show in this exercise the
existence of trilinear and quadrilinear constraints that must be satisﬁed by
matching points in three or four images, and generalize the epipolar constraint
to that case.
(a) Suppose that we have four views of a point, with known intrinsic param-
eters and projection matrices /
i
(i = 1, 2, 3, 4). Write an 8 4 homoge-
neous system of linear equations in the coordinate vector P in R
4
of this
point that must be satisﬁed by its projections into the four images.
Hint: Rewrite each projection equation as two linear equations in P,
parameterized by the corresponding projection matrix and image coordi-
nates.
Section 7.8 Notes 220
(b) Use the fact that this homogeneous system of linear equations has P as a
nontrivial solution to characterize matching constraints using two, three
or four images.
Hint: Use determinants.
(c) Show that the conditions involving two images (say, the ﬁrst and second
one) reduces to the epipolar constraints of Equation (7.2) when we take
/
1
= (Id 0) and /
2
= (1
T
−1
T
t).
(d) Show that the conditions involving three images are trilinear in the image
coordinates and derive an explicit form for these conditions when /
1
=
(Id 0), /
2
= (1
T
2
−1
T
2
t
2
), and /
3
= (1
T
3
−1
T
3
t
3
).
(e) Show that the conditions involving four images are quadrilinear in the
image coordinates.
(f) Can you imagine a method for deriving matching constraints involving
more than four images?
7.13. Generalize the constructions of the previous problem to the uncalibrated case.
PROGRAMMING EXERCISES
7.14. Implement the rectiﬁcation process.
7.15. Implement a correlation-based approach to stereopsis.
7.16. Implement a multi-scale approach to stereopsis.
7.17. Implement a dynamic-programming approach to stereopsis.
7.18. Implement a trinocular approach to stereopsis.
C H A P T E R 8
Structure from Motion
This chapter revisits the problem of estimating the three-dimensional shape of a
scene from multiple pictures. In the context of stereopsis, the cameras used to
acquire the input images are normally calibrated so their intrinsic parameters are
known, and their extrinsic ones have been determined relative to some ﬁxed world
coordinate system. This greatly simpliﬁes the reconstruction process and explains
the emphasis put on binocular (or, more generally, multi-view) fusion in Chap-
ter 7. We consider here a diﬀerent setting where the cameras’ positions and pos-
sibly their intrinsic parameters are a priori unknown and might change over time.
This is typical of the image-based modeling and rendering applications discussed in
Chapter 19, where images recorded by a handheld camcorder or multiple cameras
scattered through a scene are used to capture its shape and render it under new
viewing conditions. This is also relevant for active vision systems whose calibration
parameters vary dynamically, and planetary robot probes for which these parame-
ters may change due to the large accelerations at takeoﬀ and landing. Recovering
the cameras’ positions is of course just as important as estimating the scene shape
in the context of mobile robot navigation.
We assume in the rest of this chapter that the projections of n points have been
matched across m pictures,
1
and focus instead on the purely geometric structure-
from-motion (or SFM for short) problem of using this information to estimate both
the three-dimensional positions of the points in some ﬁxed coordinate system (the
scene structure) and the projection matrices associated with the cameras observing
them (or, equivalently, the apparant motion of the cameras relative to the points).
Figure 8.1 shows a small dataset, consisting of 38 points matched in a six-image
sequence of a toy house, courtesy of Fran¸ coise Veillon and Roger Mohr. Ground-
truth data available for the 3D position of these points is used in the quantitative
evaluation of most of the algorithms presented in the rest of this chapter.
We address in turn three instances of the structure-from-motion problem. We
start in Section 8.1 with the case where the cameras are internally calibrated—that
is, their intrinsic parameters are known, so it is possible to work in normalized image
coordinates. We turn our attention to completely uncalibrated weak-perspective
and perspective cameras in Sections 8.2 and 8.3.
8.1 INTERNALLY CALIBRATED PERSPECTIVE CAMERAS
Let us ﬁrst consider m pinhole perspective cameras with known intrinsic parameters
but unknown spatial conﬁgurations, observing a scene that consists of n ﬁxed points
P
j
(j = 1, . . . , n). We work in normalized image coordinates, and assume that cor-
respondences have been established between the m images, so the mn homogeneous
coordinate vectors p
ij
= ˆ p
ij
= (x
ij
, y
ij
, 1)
T
(i = 1, . . . , m) of the projections of the
1
Methods for establishing such correspondences across both continuous image sequences and
scattered views of a scene are discussed in Chapters 11 and 12.
221
Section 8.1 Internally Calibrated Perspective Cameras 222
FIGURE 8.1: The house dataset. Top left: One frame in the sequence, with the matched
points overlaid as small circles. Top right: A “wireframe” display of the correspond-
ing ground-truth 3D points, observed from some arbitrary viewpoint, with line segments
drawn between some of the points. Bottom: Wireframe views of the 38 data points
matched in six images. The line segments shown in these pictures do not correspond to
physical edges, and are never used in any computation. However, wireframe views are
useful to visually compare various reconstructions of the scene’s structure and epipolar
geometry. Data and image courtesy of Fran¸coise Veillon and Roger Mohr.
points P
j
are known. Because the cameras are internally calibrated, we can write
the corresponding perspective projection equations as
p
ij
=
1
Z
ij
_
!
i
t
i
_
_
P
j
1
_
, (8.1)
where !
i
and t
i
are respectively the rotation matrix and the translation vector
representing the position and orientation of camera number i in some ﬁxed coordi-
nate system, P
j
is the nonhomogeneous coordinate vector of the point P
j
in that
coordinate system, and Z
ij
is the depth of that point relative to camera number i.
We deﬁne Euclidean structure from motion as the problem of estimating the
n vectors P
j
, together with the m rotation matrices !
i
and translation vectors t
i
,
Section 8.1 Internally Calibrated Perspective Cameras 223
from the mn image correspondences p
ij
.
2
8.1.1 Natural Ambiguity of the Problem
Before trying to solve this problem, let us ﬁrst observe that its solution is, at
best, deﬁned up to a rigid transformation ambiguity. Indeed, given some arbitrary
rotation matrix ! and translation vector t, we can rewrite Equation (8.1) as
p
ij
=
1
Z
ij
_
_
!
i
t
i
_
_
! t
0
T
1
__ __
!
T
−!
T
t
0
T
1
__
P
j
1
__
=
1
Z
ij
_
!

i
t

i
_
_
P

j
1
_
,
where !

i
= !
i
!, t

i
= !
i
t +t
i
, and P

j
= !
T
(P
j
−t). (Note that because !
i
and
! are rotations, so are !

i
and !
T
: as mentioned in Chapter 1, rotation matrices
form a multiplicative group.)
This ambiguity simply stems from the fact that the structure and motion pa-
rameters consistent with image data can be expressed in diﬀerent Euclidean frames,
separated from each other by rigid transformations. Perhaps more surprisingly, it
is in fact also impossible to recover the absolute scale of the observed scene, since
we can rewrite Equation (8.1) as
p
ij
=
1
λZ
ij
_
!
i
λt
i
_
_
λP
j
1
_
=
1
Z

ij
_
!
i
t

i
_
_
P

j
1
_
,
where λ is an arbitrary positive nonzero scalar (because the sign of the depth of a
point lying in front of a camera must always be negative), t

i
= λt
i
, P

j
= λP
j
, and
Z

ij
= λZ
ij
. Intuitively, this corresponds to a well-known property of perspective
projection, already noted in Chapter 1: the apparent size of objects depends on
their distance from the cameras observing them, and an object twice as large as
another one will not appear any larger if it is twice as far.
The solution of the Euclidean SFM problem is thus deﬁned only up to an ar-
bitrary similarity—that is, a rigid transformation followed by an isotropic positive
scaling. Like rotations, rigid transformations form a group under composition (and
so do their 4 4 matrix representations under multiplication, of course). They
map points onto points and lines onto lines, and preserve incidence relations—that
is, the point where two lines (or, say, a line and a plane) intersect maps onto the
intersection of their images, as well as angles, distances, and parallelism. Similari-
ties form a group and include rigid transformations as a subgroup, sharing most of
their properties, but not preserving distances. Instead, they preserve the ratio of
distances measured along arbitrary directions. Because similarities form a group, it
makes sense to talk about the Euclidean shape of a set of points as the equivalence
class formed by all copies of these points related by these transformations (some
authors use the term metric shape). See the problems at the end of this chapter.
In particular, Euclidean structure from motion can be thought of as the re-
covery of the Euclidean shape of the observed scene, along with the corresponding
perspective projection matrices. Since Equation (8.1) provides 2mn constraints on
2
As already noted in Chapter 1, the depths of the observed points relative to the cameras are
not independent unknowns, because Z
ij
= r
i3
· P
j
+ t
i3
, where r
T
i3
is the third row vector of the
matrix R
i
, and t
i3
is the third coordinate of the vector t
i
.
Section 8.1 Internally Calibrated Perspective Cameras 224
the 6m extrinsic parameters of the matrices /
i
and the 3n parameters of the vec-
tors P
j
, taking into account the ambiguity of this problem suggests that it admits
a ﬁnite number of solutions as soon as 2mn ≥ 6m+ 3n −7. For m = 2, ﬁve point
correspondences should thus be suﬃcient to determine (up to a similarity) a ﬁnite
number of projection matrix pairs and the positions of all scene points.
In practice, 2mn is in general (much) greater than 6m + 3n − 7, and Equa-
tion (8.1) does not admit an exact solution. Instead, an approximate solution can
be found by minimizing the mean-squared error
E =
1
mn

i,j
¸
¸
¸
¸
p
ij
−
1
Z
ij
_
!
i
t
i
_
_
P
j
1
_
¸
¸
¸
¸
2
(8.2)
with respect to the 6m+3n−7 structure and motion parameters using the nonlinear
least squares optimization techniques described in Chapter 22. The main problem
with this approach is that these techniques require a reasonable initial guess to
converge to something close to the global minimum of the error function they
attempt to minimize. Reliable methods for ﬁnding such guesses are thus required.
8.1.2 Euclidean Structure and Motion from Two Images
We present in this section simple methods for computing the projection matri-
ces associated with two cameras, which in turn allows the reconstruction of the
associated scene points using triangulation. These techniques take as input the
corresponding essential or fundamental matrices, so we ﬁrst address the problem of
estimating the epipolar geometry from point correspondences, a problem known as
weak calibration.
Weak Calibration. An essential matrix can be written as c = [t
×
]!, and can
thus be parameterized by two translation parameters (t is deﬁned only up to scale)
and three rotation angles. Each correspondence between points p and p

observed
in two images provide one constraint p
T
cp

= 0 on these parameters, and thus one
can expect that c can be estimated (perhaps up to some discrete ambiguity) from a
minimum of ﬁve correspondences. Such ﬁve-point solutions to the weak calibration
problem do exist (Nist´er 2004), but they are too involved to be described here. We
focus here on the simpler case where a redundant set of n ≥ 8 point correspondences
are available. When the internal camera parameters are a priori unknown, the
output of weak calibration is an estimate of the fundamental matrix. On the other
hand, when they are known and normalized image coordinates are used in the
estimation process, additional constraints come into play, and an estimate of the
essential matrix is obtained.
Let us start with the uncalibrated case. The epipolar constraint can be written
as
p
T
Tp

i
)
2
(8.4)
with respect to the unknown entries of f (or equivalently of T) under the constraint
[[f[[
2
= 1. As shown in Chapter 22, the solution is the eigenvector f associated
with the smallest eigenvalue of |
T
|.
3
With exactly eight points, this method fails when the n 9 matrix | has a
rank strictly smaller than eight. As shown in Faugeras (1993) and the problems,
this happens only when the eight points and two optical centers lie on a quadric
surface. Fortunately, this is quite unlikely because a quadric surface is completely
determined by nine points, which means that there is generally no quadric that
passes through these 10 points.
The least-squares error E deﬁned by Equation (8.4) does not have an obvi-
ous geometric interpretation. Thus it may be preferable to minimize instead the
mean-squared geometric distance between the image points and the corresponding
epipolar lines—that is,
1
n
n

i=1
[d
2
(p
i
, Tp

i
) + d
2
(p

i
, T
T
p
i
)],
where d(p, l) denotes the (signed) Euclidean distance between the point p and the
line l, and Tp

and T
T
p are the epipolar lines associated with p

and p. This is a
nonlinear problem, but the minimization can be initialized with the result of the lin-
ear algorithm. This method, ﬁrst proposed in Luong, Deriche, and Faugeras (Luong
et al. 1993), gives much better results than the linear one.
Hartley (1995) has proposed instead to normalize the linear algorithm, ob-
serving that its poor performance is due, for the most part, to poor numerical
conditioning.
4
He suggests translating and scaling the data so they are centered at
3
We have ignored here the fact that F should be singular. When taking the nonlinear constraint
Det(F) = 0 into account, it is in fact possible to compute F from seven correspondences (see
problems).
4
The columns of U have widely diﬀerent scales for typical pictures where mean image coordi-
nates might be, say, about 500 pixels. See the problems for an alternative to Hartley’s method
that also handles this issue.
Section 8.1 Internally Calibrated Perspective Cameras 226
FIGURE 8.2: Weak-calibration experiment using two images of the house sequence and a
linear least-squares implementation of weak calibration, together with Hartley’s normal-
ization. The mean distances between the points and the corresponding epipolar lines are
0.96 and 0.90 pixels for these two images. Without the normalization, they become 10.00
and 9.12 pixels.
the origin and the average distance to the origin is
√
2. In practice, this normaliza-
tion dramatically improves the conditioning of the linear least-squares estimation
process. Concretely, the algorithm is divided into four steps: First, transform the
image coordinates using appropriate translation and scaling operators T : p
i
→ ˜ p
i
and T

as the ﬁnal estimate of the fundamental
matrix. Figure 8.2 shows the results of a weak-calibration experiment using this
method with 38 point correspondences between two images of the toy house. The
data points are shown in the ﬁgure as small discs, and the recovered epipolar lines
are shown as short line segments.
As shown in Chapter 7, given the (internal) calibration matrices / and /

of
two cameras and the corresponding essential matrix c, the fundamental matrix can
be written as T = /
−T
c/

−1
. Conversely, given T, /, and /

, we can compute
an estimate of the essential matrix as c = /
T
T/

. By construction, the matrix
c has rank 2, but, due to numerical errors, its two nonzero singular values are, in
general, not equal. The SVD once again proves useful in this setting: because it is
impossible to recover the absolute scale of the vector t from image correspondences
Section 8.1 Internally Calibrated Perspective Cameras 227
alone, we can take, without loss of generality, c = | diag(1, 1, 0)1
T
, where |J1
T
is this time the SVD of /
T
T/

.
From Essential Matrix to Camera Motion. Let us assume from now on that
the essential matrix c is known. As shown in Chapter 7, given two internally
calibrated cameras with projection matrices
_
Id 0
_
and
_
!
T
−!
T
t
_
, the corre-
sponding essential matrix is c = [t
×
]!. Specifying ! and t—that is, the camera
motion between the two views—obviously determines c. We address in this section
the inverse problem of recovering ! and t from c.
Because c
T
= 1 diag(1, 1, 0)|
T
, the nullspace of this matrix—that is, the set
of all vectors v such that c
T
v = 0—is Ru
3
, where u
3
is the third column of |
and a unit vector. In turn, because c
T
t = 0, there are two possible solutions for t,
deﬁned up to positive scale factors, namely t

= u
3
and t

= −u
3
.
Let us now show that there are also two solutions for the rotational part of
the essential matrix, namely
!

= |J1
T
and !

= |J
T
1
T
, where J =
⎛
⎝
0 −1 0
1 0 0
0 0 1
⎞
⎠
.
First, let us observe that we can always assume that the orthogonal matrices
| and 1 are rotation matrices: indeed, since the third singular value of c is zero,
we can always replace the third column of either matrix by its opposite to make
the corresponding determinant positive. The resulting decomposition of c is still a
valid SVD. Since the matrices |, 1, and J (and their transposes) are rotations,
so are !

= c. Since c is deﬁned only up to a
(possibly negative) scale factor, both solutions are valid essential matrices. The
same reasoning holds when t

is used instead of t

. We therefore have four possible
solutions for the camera motion. It is easy to show that only one of them places
the reconstructed points in front of the two cameras (see problems). It is found
by reconstructing one point and picking the solution that gives it negative depths
relative to both cameras.
The Eight-Point Algorithm. Putting weak calibration and motion estimation
together, we obtain the “eight-point” algorithm for binocular motion estimation
ﬁrst proposed by Longuet-Higgins (1981) in the case where exactly eight point
correspondences are available. The version shown in Algorithm 8.1 uses n ≥ 8
correspondences and incorporates Hartley’s normalization idea.
Figure 8.3 shows the result of an experiment involving two pictures in the
house sequence from Figure 8.1. The left part of the diagram shows the reconstruc-
tion obtained by the algorithm and viewed from an arbitrary viewpoint. Its right
part shows this reconstruction (solid lines) after it has been registered with the
Section 8.1 Internally Calibrated Perspective Cameras 228
1. Estimate T.
(a) Compute Hartley’s normalization transformation T and T

= −u
3
, where u
3
is the third
column of the matrix |.
(b) Output the combination of the rotation matrices !

, !

, and the trans-
lation vectors t

, t

such that the reconstructed points lie in front of
both cameras.
Algorithm 8.1: The Longuet-Higgins Eight-Point Algorithm for Euclidean Structure
and Motion from Two Views.
ground-truth data (dashed lines) through a similarity transformation. Once reg-
istered, the mean Euclidean distance between the reconstructed and ground-truth
3D points is 0.87cm (the house is about 20cm high), or a mean relative error of
3.1% compared to the radius of a sphere bounding the points. The results of all
algorithms presented in this section will be illustrated in the same format.
8.1.3 Euclidean Structure and Motion from Multiple Images
The binocular approach to Euclidean structure from motion described in the pre-
vious section does not readily generalize to multiple pictures. It is, however, a
rather simple matter to stitch together the structure and motion estimates asso-
ciated with diﬀerent pairs of images: Consider the graph whose nodes correspond
to image pairs and whose edges link two images that share at least three points.
Let k and l denote the indices of two adjacent pictures in this graph, and let J
kl
denote the set of indices of the points P
j
observed in both images, with homoge-
neous coordinate vectors
k
P
j
and
l
P
j
in the corresponding camera frames. The
3 4 similarity transformation o
kl
separating the coordinate systems associated
Section 8.1 Internally Calibrated Perspective Cameras 229
FIGURE 8.3: Euclidean reconstruction of the house from two views. The mean absolute
and relative errors are respectively 0.87cm and 3.1%. See the text for details.
with these cameras can be estimated by minimizing
1
n
kl

j∈J
kl
[[
k
P
j
−o
kl
l
P
j
[[
2
with respect to the unknown rotation and translation parameters. Although this
appears to be a nonlinear optimization problem, we will show in Chapter 14 that
using quaternions to represent rotations reduces it to a simple eigenvalue problem
when [J
kl
[ ≥ 3.
Picking some arbitrary base node in the graph and applying this registration
procedure to its neighbors, the neighbors’ neighbors, etc., provides a simple method
for estimating the projection matrices associated with all the nodes from the same
connected component of the graph in the coordinate system of the base node. Once
this is done, the position of every point observed by at least two cameras is easily
triangulated. The camera projection matrices and the point positions can then
be used as initial guesses for the nonlinear minimization of the error deﬁned by
Equation (8.2) (see Chapter 22 for how to solve this type of optimization problem).
Note that this technique does not require all points to be visible in all images; a
full reconstruction is possible as soon as the image graph is connected, and each
point in the scene is visible in at least two images.
The reconstruction task becomes even easier when all points are visible in all
images: Let us consider the m−1 image pairs (1, k) with k = 2, . . . , m. Applying the
eight-point algorithm to any of these pairs yields a diﬀerent reconstruction of the
scene in the coordinate system associated with the ﬁrst camera, with point positions
P
jk
(j = 1, . . . , n), and projection matrices
_
Id 0
_
and
_
!
T
k
−!
T
k
t
k
_
. In the
absence of measurement and numerical errors, the m−1 reconstructions are scaled
versions of each other (remember that the absolute scale cannot be recovered). In
practice, it is a simple matter to (roughly) estimate the corresponding scale factors:
deﬁning λ
k
= [[P
12
[[/[[P
1k
[[, we can use P
j2
(j = 1, . . . , n) and
_
!
T
k
−λ
k
!
T
k
t
k
_
(k = 2, . . . , m) as reasonable initial guesses for the scene structure and camera
Section 8.2 Uncalibrated Weak-Perspective Cameras 230
motion in the minimization of Equation (8.2). Note that this method is easily
adapted to the case where at least eight points correspondences can be established
between one (base) image and all other ones (these eight correspondences do not
have to be the same from one picture to the next), and any two images have at
least one point in common.
Figure 8.4 shows the results of an experiment using this method to recover
the Euclidean structure of the toy house. The top-left part of the ﬁgure shows the
reconstructions associated with the ﬁve corresponding image pairs, rescaled using
the ﬁrst triangulated point, as well as the track formed by the optical centers of
the recovered cameras. The top-right part of the ﬁgure shows the recovered scene
structure and camera positions after the nonlinear minimization of Equation (8.2).
The bottom part of the ﬁgure shows the ﬁnal reconstruction before and after align-
ment with the ground-truth 3D structure via a similarity transformation. Adding
images clearly helps the quality of the reconstruction, with the reconstruction error
dropping to 1.4% compared to 3.1% in the binocular case of Figure 8.3.
8.2 UNCALIBRATED WEAK-PERSPECTIVE CAMERAS
Let us now assume that the intrinsic parameters of the cameras are unknown. The
cost to pay for this is an increased ambiguity in the reconstruction (from the class
of similarity transformations to the larger classes of so-called aﬃne and projective
transformations; these will be deﬁned shortly). However, using uncalibrated cam-
eras has two distinct advantages: (1) This does not require a preliminary calibration
stage for these parameters. Instead, the structure and motion estimation process
is decomposed in two stages, where the “essential” (aﬃne or projective) structure
and motion parameters are ﬁrst recovered using simple and robust algorithms, be-
fore additional constraints associated with known camera parameters are used to
“upgrade” the reconstruction to a Euclidean one uniquely deﬁned up to a similar-
ity. (2) By “linearizing” the algebraic constraints associated with structure from
motion, this approach aﬀords simple and eﬀective methods for handling multiple
images in a uniform way.
We start in this section with the case of scenes whose relief is small compared
with their overall depth relative to the cameras observing them, so perspective pro-
jection can be approximated by the simpler weak-perspective model of the imaging
process. Concretely, according to Theorem 2 in Chapter 1, given n ﬁxed points
P
j
(j = 1, . . . , n) observed by m aﬃne cameras with unknown intrinsic and ex-
trinsic parameters, and the corresponding mn nonhomogeneous coordinate vectors
p
ij
of their images, we can rewrite the corresponding weak-perspective projection
equations as
p
ij
= /
i
_
P
j
1
_
= /
i
P
j
+b
i
for i = 1, . . . , m and j = 1, . . . , n, (8.5)
where /
i
=
_
/
i
b
i
_
is a general rank-2 2 4 matrix, and the vector P
j
in R
3
is the position of the point P
j
in some ﬁxed coordinate system. We deﬁne aﬃne
structure from motion as the problem of estimating the m matrices /
i
and the n
vectors P
j
from the mn image correspondences p
ij
.
Section 8.2 Uncalibrated Weak-Perspective Cameras 231
FIGURE 8.4: Euclidean structure and motion from multiple images. Top: Scene recon-
struction and camera trajectory before (left) and after (right) nonlinear optimization.
Bottom: The reconstructed house before (left) and after (right) alignment with the
ground truth. The mean absolute and relative errors are respectively 0.38cm and 1.4%.
See the text for details.
8.2.1 Natural Ambiguity of the Problem
In the Euclidean case, we have shown earlier that the 4 4 matrix associated
with a rigid transformation (or a similarity) and its inverse can be inserted in the
projection equations. Likewise, if /
i
and P
j
are solutions of Equation (8.5), so
are /

i
and P

j
, where
/

i
= /
i
Q,
_
P

j
1
_
= Q
−1
_
P
j
1
_
, (8.6)
and Q is an arbitrary aﬃne transformation matrix; that is, it can be written as
Q =
_
( d
0
T
1
_
with Q
−1
=
_
(
−1
−(
−1
d
0
T
1
_
, (8.7)
Section 8.2 Uncalibrated Weak-Perspective Cameras 232
where ( is a nonsingular 3 3 matrix and d is a vector in R
3
. It is easy to show
that aﬃne transformations are the most general class of 4 4 nonsingular matrices
that preserve the relationship between coordinates expressed in Equation (8.6) for
any point P
j
(see the problems).
In particular, it follows that solutions to Equation (8.5) can be deﬁned only
up to an aﬃne transformation ambiguity. Aﬃne transformations form a group and
include similarities as a subgroup. Like similarities, they map lines onto lines and
planes onto planes, and preserve parallelism and incidence relationships. Unlike
them, they do not preserve angles. They do, however, preserve the ratio of signed
lengths along parallel lines, and they can be constructed by composing a rigid
transformation, an anisotropic scaling with diﬀerent scale factors along the three
coordinate axes, and a shear. Clearly, aﬃne transformations do not preserve shape
in the Euclidean sense. Since they form a group, it is possible to talk about the
aﬃne shape of a set of points as the equivalence class formed by all the copies
of these points separated from each other by some aﬃne transformation. Aﬃne
SFM can thus be thought of as the problem of recovering the scene’s aﬃne shape,
together with the corresponding aﬃne projection matrices. Taking into account
the 12 parameters deﬁning a general aﬃne transformation, we thus expect a ﬁnite
number of solutions as soon as 2mn ≥ 8m + 3n − 12. For m = 2, this suggests
that four point correspondences should be suﬃcient to determine (up to an aﬃne
transformation and possibly some discrete ambiguity) the two projection matrices
and the three-dimensional positions of the scene points. This is conﬁrmed formally
in Sections 8.2.2 and 8.2.3.
When the intrinsic parameters of the cameras are known so the correspond-
ing calibration matrices can be taken equal to the identity, the parameters of the
projection matrices /
i
=
_
/
i
b
i
_
must obey additional constraints. For exam-
ple, according to Equation (1.22) in Chapter 1, the matrix /
i
associated with a
(calibrated) weak-perspective camera is formed by the ﬁrst two rows of a rotation
matrix, scaled by the inverse of the depth of the corresponding reference point.
As shown in Section 8.2.4, constraints such as these can be used to eliminate the
aﬃne ambiguity (or more precisely, to reduce it to a similarity ambiguity) when a
suﬃcient number of images is available. This suggests decomposing the solution of
the aﬃne structure-from-motion problem into two steps: (a) ﬁrst, use at least two
views of the scene to reconstruct its three-dimensional aﬃne shape and the corre-
sponding projection matrices; then, (b) use additional views and the constraints
associated with known camera calibration parameters to uniquely determine the
rigid Euclidean structure of the scene. The ﬁrst stage of this approach yields the
essential part of the solution: the aﬃne shape is a full-ﬂedged three-dimensional
representation of the scene, which can be used in its own right to synthesize new
views of the scene, for example. The second step amounts to ﬁnding a Euclidean
upgrade of the reconstruction—that is, to computing a single aﬃne transformation
that accounts for its rigidity and maps its aﬃne shape onto a Euclidean one.
Using three or more images overconstrains the structure-from-motion problem
and leads to more robust least-squares solutions. Accordingly, a signiﬁcant portion
of this section is devoted to the problem of recovering the aﬃne shape of a scene
from several (possibly many) pictures.
Section 8.2 Uncalibrated Weak-Perspective Cameras 233
8.2.2 Aﬃne Structure and Motion from Two Images
Let us start with the case where two aﬃne images of the same scene are available.
Introducing the aﬃne equivalent of the epipolar constraint and exploiting the nat-
ural ambiguity of aﬃne structure from motion will provide us with a very simple
method for solving this problem.
Aﬃne Epipolar Geometry. We consider two aﬃne images and rewrite the
corresponding projection equations
_
p = /P +b
p

. This is the
aﬃne epipolar constraint. Indeed, given a point p in the ﬁrst image, the position of
the matching point p

is constrained by Equation (8.8) to lie on the line l

deﬁned
by α

x

+β

y

+γ

= 0, where γ

= αx+βy +δ (Figure 8.5). Note that the epipolar
lines associated with each image are parallel to each other. For example, moving p
changes γ

or, equivalently, the distance from the origin to the epipolar line l

, but
does not modify the direction of l

.
The aﬃne epipolar constraint can be rewritten in the familiar form
(x, y, 1)T
⎛
⎝
x

y

1
⎞
⎠
= 0, where T =
⎛
⎝
0 0 α
0 0 β
α

β

δ
⎞
⎠
(8.9)
is the aﬃne fundamental matrix. This suggests that the aﬃne epipolar geometry
can be seen as the limit of the perspective one. Indeed, it can be shown that an
aﬃne picture is the limit of a sequence of images taken by a perspective camera
that zooms in on the scene as it backs away from it. In turn, this leads to another
derivation of Equation (8.9). See the problems for details.
Aﬃne Weak Calibration. Given n ≥ 4 point correspondences p
j
↔ p

for p is
constrained to belong to this line. The same property holds for all other aﬃne projection
models.
As before, solving this equation in the least-squares sense amounts to computing
the eigenvector f associated with the smallest eigenvalue of |
T
|.
Figure 8.6 shows the results of a weak calibration experiment on two images
in the house sequence. Weak perspective is a rather coarse approximation of the
image formation process, and, as expected, errors are worse than in the perspective
case (compare to Figure 8.2). Note that, as in that case, Hartley’s normalization is
essential for obtaining reasonable results.
From the Aﬃne Fundamental Matrix to Camera Motion. Let us now
show that the projection matrices can be estimated from the epipolar constraint.
The natural ambiguity of aﬃne structure from motion allows us to simplify the
calculations: according to Equations (8.6) and (8.7), if / = (/ b) and /

=
(/

b

) are solutions of our problem, so are
˜
/= /Q and
˜
/

= /

Q, where
Q =
_
( d
0
T
1
_
is an arbitrary aﬃne transformation. The new projection matrices can be written as
˜
/= (/( /d+b) and
˜
/

= (/

( /

d+b

). Note that, according to Equation
(8.7), applying this transformation to the projection matrices amounts to applying
the inverse transformation to every scene point P, whose position P is replaced by
˜
P = (
−1
(P −d).
As shown in the problems at the end of this chapter, it is possible to choose
Section 8.2 Uncalibrated Weak-Perspective Cameras 235
FIGURE 8.6: Aﬃne weak-calibration experiment using two images of the house sequence
and linear least squares, together with Hartley’s normalization. The mean distances be-
tween the points and the corresponding epipolar lines are 3.24 and 3.15 pixels for these
two images.
( and d so that the two projection matrices take the canonical forms:
˜
/=
_
1 0 0 0
0 1 0 0
_
and
˜
/

−d = 0,
where the coeﬃcients a, b, c, and d are related to the parameters α, β, α

, β

, and
δ by a : α = b : β = c : α

= −1 : β

= d : δ.
Once the coeﬃcients a, b, c, and d have been estimated via linear least squares,
the two projection matrices are known, and the position of any point can be esti-
mated from its image coordinates by using once again linear least squares to solve
the corresponding system of four equations,
⎛
⎜
⎜
⎝
1 0 0 x
0 1 0 y
0 0 1 x

)
T
without estimating the coeﬃcients
Section 8.2 Uncalibrated Weak-Perspective Cameras 236
FIGURE 8.7: The aﬃne reconstruction of the house from two views. Left: Aﬃne recon-
struction obtained by reducing the second row of /

to (0, 0, 1). In this particular case,
the computation is numerically much better behaved than when reducing the ﬁrst row to
the same form. Right: The reconstruction after aﬃne registration with the ground-truth
data. The mean Euclidean distance between the reconstructed and ground-truth points
is 0.92cm, or a mean relative error of 3.2%.
a, b, c, and d, and without requiring a minimum number of matches. This is not as
surprising as one might think. In the case of two calibrated orthographic cameras
with perpendicular projection directions and parallel y axes, taking X = x, Y = y,
and Z = x

does yield the correct Euclidean reconstruction (have another look at
Figure 8.5, assuming orthographic projection and imagining that the epipolar lines
are parallel to the x and x

axes). In practice, of course, using all four equations
might yield more accurate results. The proposed method reduces the ﬁrst row of
/

to (0, 0, 1) via the aﬃne transformation Q. When the ﬁrst row of the matrix /

(almost) lies in the plane spanned by the rows of /, the matrix inversion involved in
this process is numerically ill behaved, and it is preferable to apply instead the same
reduction to the second row of /

. When both matrices constructed in this fashion
are singular, the two image planes are parallel and the scene structure cannot be
recovered.
Figure 8.7 shows the 3D aﬃne shape of the house recovered from two images.
In this case, taking X = x, Y = y, and Z = x

does not give very good results,
due to numerical conditioning problems. However, it is striking that taking X = x,
Y = y, and Z = y

yields quite a reasonable reconstruction of the house, even if
the mean relative error (3.2%) is slightly worse than that obtained by the eight-
point algorithm (3.1%). (To be fair, note that the recovered shape is registered
to the ground-truth one using an aﬃne transformation in this case. With more
degrees of freedom (12) than a similarity (7), it ﬁts the data better, which biases
the comparison a bit.)
Section 8.2 Uncalibrated Weak-Perspective Cameras 237
8.2.3 Aﬃne Structure and Motion from Multiple Images
The method presented in the previous section is aimed at recovering the aﬃne scene
structure and the corresponding projection matrices from a minimum number of
images. We now address the problem of estimating the same information from a
potentially large number of pictures.
As usual, it is convenient to simplify as much as possible all equations involved
in our problem. Under aﬃne projection, the image of the center of mass of a set of
points is the center of mass of their images (see problems). When we let P
0
denote
the center of mass of the n points P
1
, . . . , P
n
, and let p
i0
denote its projection into
image number i, we have
p
i0
= /
i
P
0
+b
i
, and thus p
ij
−p
i0
= /
i
(P
j
−P
0
).
Now, of course, we are free to pick P
0
as the origin of the world coordinate system,
so P
0
= 0. Because p
i0
is “observable” as the center of mass of the points p
ij
, we
are also free to choose it as the origin of the coordinate system attached with image
number i, so p
i0
= 0. This allows us to rewrite Equation (8.5) as
p
ij
= /
i
P
j
for i = 1, . . . , m and j = 1, . . . , n, (8.12)
and reduce the aﬃne ambiguity to a linear one.
The mn instances of Equation (8.12) can now be rewritten in matrix form as
T = /T, where T =
⎛
⎝
p
11
. . . p
1n
. . . . . . . . .
p
m1
. . . p
mn
⎞
⎠
, / =
⎛
⎜
⎝
/
1
.
.
.
/
m
⎞
⎟
⎠
, andT =
_
P
1
. . . P
n
_
.
As the product of a 2m 3 matrix and a 3 n matrix, the 2m 3n matrix
T has, in general, rank 3. As shown by Tomasi and Kanade (1992), singular value
decomposition provides a practical method for recovering both / and T from the
(observed) data matrix T. Indeed, if |J1
T
is the SVD of the rank-3 matrix T,
only three of the singular values are nonzero, thus T = |
3
J
3
1
T
3
, where |
3
and 1
3
denote the 2m3 and 3 n matrices formed by the three leftmost columns of the
matrices | and 1, and J
3
is the 33 diagonal matrix formed by the corresponding
nonzero singular values.
In the noiseless case where T is truly a rank-3 matrix, it is easy to exploit
the inherent ambiguity of aﬃne structure from motion to show that /
0
= |
3
√
J
3
and T
0
=
√
J
3
1
T
3
are representatives of the true (aﬃne) camera motion and scene
shape (see problems). In practice, due to image noise, errors in localization of
feature points, and the mere fact that actual cameras are not aﬃne, the equation
T = /T does not hold exactly, and the matrix T has (in general) full rank. In this
case, the best we can hope for is to minimize
E =

i,j
[[p
ij
−/
i
P
j
[[
2
=

j
[[q
j
−/P
j
[[
2
= [[T −/T[[
2
F
,
with respect to the matrices /
i
(i = 1, . . . , m) and vectors P
j
(j = 1, . . . , m) or,
equivalently, with respect to the matrices / and T. (Here [[/[[
F
denotes, as in
Section 8.2 Uncalibrated Weak-Perspective Cameras 238
Chapter 22, the Frobenius norm of the matrix /—that is, the square root of the
sum of the squared entries of that matrix.)
According to Theorem 6 in Chapter 22, the matrix /
0
T
0
is the closest rank-3
approximation to T. Because the rank of /T is 3 for any rank-3 2m3 matrix /
and rank-3 3 n matrix T, the minimum value of E is thus reached for / = /
0
and T = T
0
, which conﬁrms that /
0
and T
0
are the optimal estimates of the
true camera motion and scene structure. This does not contradict the inherent
ambiguity of aﬃne structure from motion: all aﬃnely equivalent solutions yield the
same value for E, and we just as well could have taken / = /
0
o and T = o
−1
T
0
for any nonsingular 3 3 matrix o. In particular, singular value decomposition
provides a solution to the aﬃne SFM problem, as illustrated by Algorithm 8.2.
1. Compute the singular value decomposition T = |J1
T
.
2. Construct the matrices |
3
, 1
3
, and J
3
formed by the three leftmost columns
of the matrices | and 1, and the corresponding 3 3 submatrix of J.
3. Deﬁne
/
0
= |
3
_
J
3
and T
0
=
_
J
3
1
T
3
;
the 2m 3 matrix /
0
is an estimate of the camera motion, and the 3 n
matrix T
0
is an estimate of the scene structure.
Algorithm 8.2: The Tomasi–Kanade Factorization Algorithm for Aﬃne Shape from
Motion.
Figure 8.8 shows the 3D (aﬃne) shape of the house recovered from six images
of 38 points of the house. The mean relative error of 2.8% is, as expected, smaller
than the 3.2% obtained from two views only.
8.2.4 From Aﬃne to Euclidean Shape
Let us now assume a weak-perspective model of the imaging process and (internally)
calibrated cameras. Recall from Chapter 1 that a weak-perspective projection ma-
trix can be written as
/=
1
Z
r
_
k s
0 1
_
_
!
2
t
2
_
,
where Z
r
is the depth of the reference point, k and s are aspect-ratio and skew
parameters, !
2
is the 23 matrix formed by the ﬁrst two rows of a rotation matrix,
and t
2
is a vector in R
2
. When the camera is calibrated, we can use normalized
image coordinates and take k = s = 1. The projection matrix becomes
ˆ
/=
_
ˆ
/
ˆ
b
_
=
1
Z
r
_
!
2
t
2
_
. (8.13)
It follows from Equation (8.13) that the matrix
ˆ
/ is part of a (scaled) rotation
matrix, with row vectors ˆ a
T
1
and ˆ a
T
2
that are orthogonal to each other and have
the same norm. In other words, a (calibrated) weak-perspective camera is an aﬃne
Section 8.2 Uncalibrated Weak-Perspective Cameras 239
FIGURE 8.8: The aﬃne reconstruction of the house from multiple views. Left: The orig-
inal reconstruction. Right: An overlay of the reconstruction after it has been registered
(via an aﬃne transformation) with the ground truth data. The mean Euclidean distance
between the reconstructed and ground-truth points is 0.77cm, or a mean relative error of
2.8%.
camera with the additional constraints
ˆ a
1
ˆ a
2
= 0 and [[ˆ a
1
[[
2
= [[ˆ a
2
[[
2
. (8.14)
Let us suppose that we have recovered the aﬃne shape of a scene and the pro-
jection matrix / associated with each view. We already know that all solutions
of the structure-from-motion problem are the same up to an aﬃne ambiguity. In
particular, the Euclidean coordinate vectors
ˆ
P of scene points and the correspond-
ing projection matrices
ˆ
/ must be related to their aﬃne counterparts P and /
by some aﬃne transformation
Q =
_
( d
0
T
1
_
such that
ˆ
/ = /Q and
ˆ
P = (
−1
(
˜
P − d). Such a transformation is called a
Euclidean upgrade because it maps the aﬃne shape of a scene onto its Euclidean
one.
Let us now show how to compute such an upgrade when m ≥ 3 weak-
perspective images are available. Let /
i
= (/
i
b
i
) denote the corresponding
projection matrices, estimated using the factorization method of Section 8.2.3, for
example. If
ˆ
/
i
= /
i
Q, we can rewrite the weak-perspective constraints of Equa-
tion (8.14) as
_
ˆ a
i1
ˆ a
i2
= 0,
[[ˆ a
i1
[[
2
= [[ˆ a
i2
[[
2
,
⇐⇒
_
a
T
i1
((
T
a
i2
= 0,
a
T
i1
((
T
a
i1
= a
T
i2
((
T
a
i2
,
for i = 1, . . . , m,
(8.15)
where a
T
i1
and a
T
i2
denote the rows of the matrix /
i
. This overconstrained system
of 3m quadratic equations in the coeﬃcients of ( can be solved via nonlinear least
Section 8.3 Uncalibrated Perspective Cameras 240
FIGURE 8.9: The Euclidean reconstruction of the house using a weak-perspective model.
Left: The original reconstruction. Right: An overlay of the reconstruction after it has
been registered (via a similarity transformation) with the ground-truth data. The mean
Euclidean distance between the reconstructed and ground-truth points is 0.83cm, or a
mean relative error of 3.0%. Note that the registration error is a bit larger than for the
plain aﬃne reconstruction shown in Figure 8.8. As before, this is not surprising, since an
aﬃne transformation has more “degrees of freedom” (12) than a similarity (7).
squares, but this requires some reasonable initial guess for these coeﬃcients. An
alternative is to consider Equation (8.15) as a set of linear constraints on the matrix
T = ((
T
. The coeﬃcients of T can be found in this case via linear least squares,
and ( can then be computed as
√
T using Cholesky decomposition. It should be
noted that this requires that the recovered matrix T (or its opposite) be positive
deﬁnite, which is not guaranteed in the presence of noise. Note also that the solution
of Equation (8.15) is deﬁned only up to an arbitrary rotation. To determine Q
uniquely and simplify the calculations, it is possible to map /
1
(and possibly /
2
)
to canonical forms as before.
Figure 8.9 shows the weak-perspective upgrade associated with the Tomasi-
Kanade aﬃne reconstruction of the house shown in Figure 8.8. The mean relative
error is 3.0% in this case.
8.3 UNCALIBRATED PERSPECTIVE CAMERAS
Let us come back to perspective projection, and assume again that the intrinsic
parameters of the cameras are unknown. Given n ﬁxed points P
j
(j = 1, . . . , n)
observed by m cameras and the corresponding mn homogeneous coordinate vec-
tors p
ij
= (x
ij
, y
ij
, 1)
T
of their images, we write the corresponding perspective
projection equations as
⎧
⎪
⎪
⎨
⎪
⎪
⎩
x
ij
=
m
i1
P
j
m
i3
P
j
y
ij
=
m
i2
P
j
m
i3
P
j
for i = 1, . . . , m and j = 1, . . . , n, (8.16)
Section 8.3 Uncalibrated Perspective Cameras 241
where m
T
i1
, m
T
i2
, and m
T
i3
denote the rows of the 3 4 projection matrix /
i
associated with camera number i in some ﬁxed coordinate system, and P
j
denotes
the homogeneous coordinate vector of the point P
j
in that coordinate system.
According to Theorem 1 (Chapter 1), any 3 4 matrix / =
_
/ b
_
, where
/ is a nonsingular 33 matrix and b is an arbitrary vector in R
3
can be interpreted
as a perspective projection matrix; that is, it can be written as /= ρ/
_
! t
_
for
some nonzero real ρ, calibration matrix /, 33 rotation matrix !, and translation
vector t in R
3
. We relax this condition a bit in this chapter, and deﬁne a projective
projection matrix as an arbitrary rank-3 3 4 matrix. Clearly, perspective pro-
jection matrices are projective ones, but not all projective projection matrices are
perspective ones. We will come back to the implications of this relaxation shortly.
In the mean time, let us deﬁne projective structure from motion as the problem of
estimating the m rank-3 matrices /
i
and the n vectors P
j
from the mn image
correspondences p
ij
.
8.3.1 Natural Ambiguity of the Problem
When /
i
and P
j
are solutions of Equation (8.16), so are λ
i
/
i
and μ
j
P
j
for
any nonzero values of λ
i
and μ
j
. In particular, as already noted in Chapter 1,
the matrices /
i
satisfying Equation (8.16) are deﬁned only up to scale, with 11
independent parameters, and so are the homogeneous coordinate vectors P
j
in R
4
(with only three independent parameters; when necessary, these can be reduced to
the canonical form (X
j
, Y
j
, Z
j
, 1)
T
as long as their fourth coordinate is not zero,
which is the case in general). Like its aﬃne cousin, projective SFM is subject to
a deeper ambiguity that justiﬁes its name: Let Q denote an arbitrary projective
transformation matrix (or homography; the two terms are strictly equivalent)—
that is, an arbitrary nonsingular 4 4 matrix. Postmultiplying /
i
by Q does not
change its rank, and it follows that, if /
i
and P
j
are solutions of the projective
structure-from-motion problem, so are /

i
= /
i
Q and P

j
= Q
−1
P
j
.
Projective transformations form a group and include aﬃne transformations
as a subgroup. Like aﬃne transformations, they map lines onto lines and planes
onto planes, and preserve incidence relationships. Unlike these, however, they do
not preserve parallelism, or the ratio of lengths along parallel lines, and thus do not
preserve aﬃne shape. They preserve instead the cross-ratio of four points along the
same line (see problems). Because homographies form a group, it is again possible
to talk about the projective shape of a set of points, and projective structure from
motion can be thought of as the recovery of the observed scene’s projective shape,
along with the corresponding projection matrix parameters.
The matrix Q is deﬁned only up to scale, with 15 free parameters, because
multiplying it by a nonzero scalar simply amounts to applying inverse scalings to /
i
and P
j
. Because Equation (8.16) provides 2mn constraints on the 11m parameters
of the matrices /
i
and the 3n parameters of the vectors P
j
, taking into account
the natural ambiguity of structure from motion suggests that this problem admits
a ﬁnite number of solutions as soon as 2mn ≥ 11m + 3n − 15. For m = 2, seven
point correspondences should thus be suﬃcient to determine (up to a projective
transformation and possibly a ﬁnite ambiguity) the two projection matrices and
the position of any other point. This is conﬁrmed in Sections 8.3.2 and 8.3.3.
Section 8.3 Uncalibrated Perspective Cameras 242
Before proceeding, let us come back to the diﬀerence between (uncalibrated)
perspective and projective structure from motion. A formal argument is beyond the
scope of this book, but let us just note that a perspective projection matrix / =
_
/ b
_
with det(/) ,= 0 is just the analytical representation in some Euclidean
coordinate system of the geometric perspective projection operator which, given
some pinhole O and retinal plane Π, associates with any point P ,= O in E
3
the
point where the line joining P and O intersects Π. A projective projection matrix is
just another representation for the same operator, expressed this time in a projective
coordinate system, whose formal deﬁnition is once again beyond the scope of this
book, but can be thought of intuitively as a warped frame in which the projective
shape has its Euclidean coordinates.
Similar to the aﬃne case, we will decompose in the rest of this section the so-
lution to structure from motion into two steps: ﬁrst, (a) use at least two views of the
scene to reconstruct its three-dimensional projective shape and the corresponding
projective projection matrices; then, (b) use additional views and the constraints
associated with known camera calibration parameters to uniquely determine the
Euclidean structure of the scene. The second step amounts to ﬁnding a Euclidean
upgrade of the scene—that is, to computing a single projective transformation that
maps its projective shape onto a Euclidean one.
8.3.2 Projective Structure and Motion from Two Images
Let us now assume that the fundamental matrix T associated with two pictures
has been estimated from binocular correspondences. As in the aﬃne case, the
projection matrices can in fact be estimated from a parameterization of T that
exploits the inherent ambiguity of projective SFM. In the projective setting, the
scene structure and camera motion are deﬁned only up to an arbitrary homography,
and we can reduce the two matrices to canonical forms
˜
/= /Q and
˜
/

= /

Q
by postmultiplying them by an appropriate 4 4 matrix Q. (We must of course
simultaneously premultiply the coordinate vector P of any point P by its inverse,
yielding
˜
P = Q
−1
P.) This time, we take
˜
/

to be proportional to (Id 0) and
leave
˜
/ in the general form (/ b) (this determines 11 of Q’s entries). Let us now
derive a new expression for the fundamental matrix using the canonical form of
˜
/

.
If Z and Z

denote the depths of the point P relative to the two cameras, we can
write the projection equations associated with the two cameras as Zp =
_
/ b
_
˜
P
and Z

p

=
_
Id 0
_
˜
P or, equivalently,
Zp = /(Id 0)
˜
P +b = Z

/p

+b.
It follows that Zb p = Z

b /p

, and forming the dot product of this expression
with p ﬁnally yields
p
T
Tp

= 0 where T = [b
×
]/. (8.17)
Note the similarity with the expression for the essential matrix derived in Chapter
7. In particular, we have T
T
b = 0, so (as could have been expected) b is the homo-
geneous coordinate vector of the ﬁrst epipole in the corresponding image coordinate
system. This new parameterization of the matrix T provides a simple method for
computing the projection matrix
˜
/. First, note that because the overall scale of
˜
/
Section 8.3 Uncalibrated Perspective Cameras 243
FIGURE 8.10: The projective reconstruction of the house from two views. Left: The
original reconstruction. Right: An overlay of the reconstruction after it has been regis-
tered (via a projective transformation) with the ground-truth data. The mean Euclidean
distance between the reconstructed and ground-truth points is 0.34cm, or a mean relative
error of 1.2%.
is irrelevant, we can always take [[b[[ = 1. This allows us to ﬁrst compute b as the
linear least-squares solution of T
T
b = 0 with unit norm, and we pick /
0
= −[b
×
]T
as the value of /. It is easy to show that, for any vector a, [a
×
]
2
= aa
T
−[[a[[
2
Id,
thus:
[b
×
]/
0
= −[b
×
]
2
T = −bb
T
T +[[b[[
2
T = T,
since T
T
b = 0 and [[b[[
2
= 1. This shows that
˜
/ =
_
/
0
b
_
is a solution of
Equation (8.17).
5
As shown in the problems, there is in fact a four-parameter
family of solutions whose general form is
˜
/=
_
/ b
_
with / = λ/
0
+ ( μb νb τb ). (8.18)
The four parameters correspond, as could have been expected, to the remaining
degrees of freedom of the projective transformation Q. Once the matrix
˜
/ is
known, we can compute the position of any point P by solving in the least-squares
sense the nonhomogeneous linear system of equations in Z and Z

i,j
[[Z
ij
p
j
−/
i
P
j
[[
2
= [[T −/T[[
2
F
(8.20)
with respect to the depths Z
ij
and the entries of the matrices / and T. When the
depths Z
ij
are known, we can compute / and T by using singular value decom-
position to compute a rank-4 (as opposed to rank-3 in the aﬃne case) factorization
of T. On the other hand, when / and T are known, we can read out the values of
the depths Z
ij
from Equation (8.19). This suggests an iterative scheme, alternating
steps where one group of variables is estimated while the other one is kept constant.
Note, however, that a trivial minimum of E corresponds to taking all variables Z
ij
,
/
i
, and P
j
equal to zero. To avoid this, Sturm and Triggs (1996) propose renor-
malizing after each iteration the rows of the matrix T, then its columns, so they
have unit norm. Unfortunately, with this normalization, there is no guarantee that
the error will decrease at each step or that the method will converge to some local
minimum.
Bilinear projective SFM. An alternative to this approach can be obtained by
noting again that the variables Z
ij
are not independent of /
i
and P
j
, and try-
ing to eliminate them to construct a nonredundant parameterization of projective
SFM (Mahamud et al. 2001). Writing that the derivative of E with respect to Z
ij
should be zero at an extremum of this function, a simple calculation shows that at
such a point the value of E is given by:
E =

ij
[[p
ij
(/
i
P
j
)[[
2
, (8.21)
where, without loss of generality, the vectors p
ij
are supposed to have unit norm.
Note that the depths Z
ij
have been eliminated in the process.
It is thus possible to minimize E by alternating steps where the vectors P
j
are kept constant (resp. estimated) while the matrices /
i
are estimated (resp.
Section 8.3 Uncalibrated Perspective Cameras 245
kept constant). Because the error term p
ij
(/
i
P
j
) is bilinear in /
i
and P
j
,
the global optimum in /
i
or P
j
at each step of this algorithm can be obtained
using linear least squares under the constraints [[/
i
[[
2
F
= 1 and [[P
j
[[
2
= 1 for
i = 1, . . . , m and j = 1, . . . , n. Note that this choice of constraints avoids the
degenerate (global) minimum corresponding to /
i
= 0 and P
j
= 0. However,
it does not necessarily avoid other degeneracies, corresponding, for example, to
picking /
i
= /
0
(i = 1, . . . , m) and P
j
= P
0
(j = 1, . . . , n), where /
0
is an
arbitrary rank-3 3 4 matrix with unit Frobenius form and P
0
is a unit vector
in its nullspace (there are other trivial zeros corresponding to lower-rank values of
/
0
and families of vectors in their nullspaces).
It is possible to show that the error decreases at each iteration, and that the
parameters converge to those of one of its critical points (Mahamud et al. 2001). As
demonstrated experimentally by Hartley, however, the minimization process might
be unstable, and after ﬁnding an acceptable solution in relatively few steps (say, 50
to 100), it might eventually switch to a degenerate zero minimum (typically after
tens of thousands of iterations). It should therefore be stopped before that—for
example, by monitoring the rate at which the error decreases, and stopping when
it becomes small enough.
Bundle adjustment. As discussed in Mahamud et al. (2001), degeneracy prob-
lems may be unavoidable for any method attempting to minimize the error func-
tion deﬁned by Equation (8.20). For example, the normalization mechanism used
in Sturm and Triggs (1996) does not guarantee that all depth parameters Z
ij
are
nonzero. An alternative is to revert to the original formulation of projective SFM,
and use nonlinear least squares to minimize directly
E =
1
mn

i,j
⎡
⎣
_
x
ij
−
m
i1
P
j
m
i3
P
j
_
2
+
_
y
ij
−
m
i2
P
j
m
i3
P
j
_
2
⎤
⎦
with respect to the matrices /
i
(i = 1, . . . , m) and vectors P
j
(j = 1, . . . , n).
This is the method of bundle adjustment, whose name originates in the ﬁeld
of photogrammetry. It is not susceptible to degeneracies, and it combines all mea-
surements to minimize a physically signiﬁcant error measure—namely, the mean-
squared error between the actual image point positions and those predicted using
the estimated scene structure and camera motion. It also can take advantage of
the variants of Newton’s algorithm available for nonlinear least-squares problems
(Chapter 1), which typically converge in fewer (although possibly more expensive)
iterations than the alternation methods discussed so far. Figure 8.11 shows the
projective reconstruction of the toy house obtained using this method after 15 iter-
ations of the Levenberg-Marquardt implementation of nonlinear least squares, using
the coordinates of the points reconstructed by aﬃne factorization as initial guesses
for the vectors P
j
and the projections computed from these points as initial values
for the matrices /
i
. The mean relative error is 0.2%.
Section 8.3 Uncalibrated Perspective Cameras 246
FIGURE 8.11: The projective reconstruction of the house from multiple views obtained
using bundle adjustment. Left: The original reconstruction. Right: An overlay of
the reconstruction after it has been registered (via a projective transformation) with the
ground-truth data. The mean Euclidean distance between the reconstructed and ground-
truth points is 0.07cm, or a mean relative error of 0.2%.
8.3.4 From Projective to Euclidean Shape
Although projective structure is useful by itself, in most cases it is the Euclidean
structure of the scene that is the true object of interest. We saw in Section 8.1 that
the best we can hope for is to estimate the Euclidean shape of the scene, deﬁned
up to an arbitrary similarity transformation.
Let us assume from now on that one of the techniques presented in Sec-
tion 8.3.3 has been used to estimate the projection matrices /
i
(i = 1, . . . , m) and
the point positions P
j
(j = 1, . . . , n) from m images of these points. We know
that any other reconstruction, and in particular a Euclidean one, is separated from
this one by a projective transformation. In other words, if
ˆ
/
i
and
ˆ
P
j
denote the
Euclidean shape and the corresponding motion parameters, there must exist a 44
Euclidean upgrade matrix Q such that
ˆ
/
i
= /
i
Q and
ˆ
P
j
= Q
−1
P
j
. The rest of
this section presents a method for computing Q and thus recovering the Euclidean
shape and motion from the projective ones when (some of) the intrinsic parameters
of the camera are known.
Let us ﬁrst note that, since the individual matrices /
i
are deﬁned only up to
scale, so are the matrices
ˆ
/
i
that can be written (in the most general case, where
some of the intrinsic parameters are unknown) as
ˆ
/
i
= ρ
i
/
i
(!
i
t
i
),
where ρ
i
accounts for the unknown scale of /
i
, and /
i
is a calibration matrix as
deﬁned by Equation (1.14). In particular, if we write the Euclidean upgrade matrix
as Q = (Q
3
q
4
), where Q
3
is a 4 3 matrix and q
4
is a vector in R
4
, we obtain
immediately
/
i
Q
3
= ρ
i
/
i
!
i
. (8.22)
Section 8.3 Uncalibrated Perspective Cameras 247
When the intrinsic parameters of all cameras are known, so the matrices /
i
can be taken equal to the identity, the 3 3 matrices /
i
Q
3
are scaled rotation
matrices. Writing that their rows m
T
ij
(j = 1, 2, 3) are perpendicular to each other
and have the same norm yields
⎧
⎪
⎪
⎪
⎪
⎨
⎪
⎪
⎪
⎪
⎩
m
T
i1
/m
i2
= 0,
m
T
i2
/m
i3
= 0,
m
T
i3
/m
i1
= 0,
m
T
i1
/m
i1
−m
T
i2
/m
i2
= 0,
m
T
i2
/m
i2
−m
T
i3
/m
i3
= 0,
(8.23)
where / = Q
3
Q
T
3
. The upgrade matrix Q is of course deﬁned only up to an arbi-
trary similarity. To determine it uniquely, we can assume that the world coordinate
system and the ﬁrst camera’s frame coincide. Given m images, we obtain 12 linear
equations and 5(m − 1) quadratic ones in the coeﬃcients of Q. These equations
can be solved using nonlinear least squares, which requires as usual a reasonable
initial guess.
Alternatively, the constraints in Equation (8.23) are linear in the 10 coeﬃ-
cients of the symmetric matrix /, allowing its estimation from at least two images
via linear least squares. Note that / has rank 3—a constraint not enforced by our
construction. To recover Q
3
, let us also note that, since / is symmetric, it can
be diagonalized in an orthonormal basis as / = |T|
T
, where T is the diagonal
matrix formed by the eigenvalues of /, and | is the orthogonal matrix formed
by its eigenvectors. In the absence of noise, / is positive semideﬁnite with three
positive and one zero eigenvalues, and Q
3
can be computed as |
3
√
T
3
, where |
3
is the matrix formed by the columns of | associated with the positive eigenvalues
of /, and T
3
is the corresponding submatrix of T. Because of noise, however, /
usually has maximal rank, and its smallest eigenvalue might even be negative. As
shown in Ponce (2000), if we take this time |
3
and T
3
to be the submatrices of |
and T associated with the three largest (positive) eigenvalues of /, then |
3
T
3
|
T
3
provides the best positive semideﬁnite rank-3 approximation of / in the sense of
the Frobenius norm,
6
and we can take as before Q
3
= |
3
√
T
3
. At this point, the
last column vector q
4
of Q can be determined by (arbitrarily) picking the origin of
the frame attached to the ﬁrst camera as the origin of the world coordinate system.
Figure 8.12 shows the perspective upgrade associated with the projective re-
construction of the house obtained by bundle adjustment and shown in Figure 8.11,
with a mean absolute error of 0.33cm and a relative error of 1.2%.
This method can be adapted easily to the case where only some of the intrinsic
camera parameters are known. Using the fact that !
i
is an orthogonal matrix allows
us to write
/
i
//
T
i
= ρ
2
i
/
i
/
T
i
. (8.24)
Thus, every image provides a set of constraints between the entries of /
i
and /.
Assuming, for example, that the center of the image is known for each camera, we
6
Note the similarity between this result and Theorem 6.
Section 8.4 Notes 248
FIGURE 8.12: The Euclidean reconstruction of the house obtained by a Euclidean upgrade
of the projective reconstruction obtained with bundle adjustment. Left: The original
reconstruction. Right: An overlay of the reconstruction after it has been registered
(via a similarity) with the ground-truth data. The mean Euclidean distance between
the reconstructed and ground-truth points is 0.33cm, or a mean relative error of 1.2%.
As before, the error has increased compared to the projective reconstruction because the
similarity used for registration has fewer parameters than a homography.
can take x
0
= y
0
= 0 and write the square of the matrix /
i
as
/
i
/
T
i
=
⎛
⎜
⎜
⎜
⎜
⎝
α
2
i
1
sin
2
θ
i
−α
i
β
i
cos θ
i
sin
2
θ
i
0
−α
i
β
i
cos θ
i
sin
2
θ
i
β
2
i
1
sin
2
θ
i
0
0 0 1
⎞
⎟
⎟
⎟
⎟
⎠
.
In particular, the part of Equation (8.24) corresponding to the zero entries of
/
i
/
T
i
provides two independent linear equations in the 10 coeﬃcients of the 4 4
symmetric matrix /:
_
m
T
i1
/m
i3
= 0,
m
T
i2
/m
i3
= 0.
As might have been expected, these equations form a subsert of those in Equa-
tion (8.23). With m ≥ 5 images, the parameters can be estimated via linear least
squares. Once / is known, Q can be estimated as before. Continuing to assume
that u
0
= v
0
= 0, it is easy to add zero-skew and unit aspect-ratio constraints.
For example, assuming zero skew (θ = π/2) provides the additional constraint
m
T
i1
/m
i2
= 0.
8.4 NOTES
The interested reader is invited to consult the two excellent textbooks dedicated
to structure from motion (Hartley & Zisserman 2000b; Faugeras, Luong, & Pa-
padopoulo 2001) for details beyond the scope of this book. See also Ma, Soatto,
Section 8.4 Notes 249
and Sastry (2003a) and the ﬁrst edition of this book. A deeper understanding
of aﬃne and projective SFM requires working knowledge of elementary aﬃne and
projective geometries. This is also beyond the scope of this book, but excellent text-
books are once again available for the interested reader—for example, (Snapper and
Troyer 1989) for aﬃne geometry, and Todd (1946), Coxeter (1974), Berger (1987),
and Samuel (1988) for projective geometry.
The SFM problem was ﬁrst studied in the calibrated orthographic setting by
Ullman (1979). Longuet-Higgins (1981) then gave the ﬁrst solution to the calibrated
perspective case with the eight-point algorithm discussed in this chapter. A solution
to the minimal ﬁve-point formulation of this problem can be found in Nist´er (2004).
The idea of stratifying SFM into a two-step problem where the aﬃne or projective
shape of a scene is recovered before additional constraints are brought into play
to construct a Euclidean model is due to Koenderink and Van Doorn (1990) for
the aﬃne case, and to Faugeras (1995) for the projective one. In the aﬃne case,
the ﬁrst solutions to this problem are due to Koenderink and Van Doorn (1990)
and Tomasi and Kanade (1992). The initial, aﬃne stage is valuable by itself: for
example, it is the basis for the motion-based segmentation methods introduced by
Gear (1998) and Costeira and Kanade (1998). The nonlinear least-squares method
for computing the Euclidean upgrade matrix Qis due to Tomasi and Kanade (1992).
The Cholesky approach to the same problem is due to Poelman and Kanade (1997);
see Weinshall and Tomasi (1995) for another variant. Various extensions have
been proposed recently, including the incremental recovery of structure and motion
(Weinshall & Tomasi 1995; Morita & Kanade 1997).
The ﬁrst solutions to projective SFM are due to Faugeras (1992) and Hart-
ley et al. (1992). Other notable work in this area includes, for example, Mohr et
al. (1992) and Shashua (1993). The two-view algorithm presented in this chapter
is due to Hartley (1994b). The extension of factorization approaches to structure
and motion recovery was ﬁrst proposed by Sturm and Triggs (1996). The bilinear
approach to projective SFM presented in Section 8.3.3 is an instance of a class
of techniques called resection-intersection methods in photogrammetry (Triggs et
al. 2000), which interleave steps where the camera parameters are estimated while
the oberved point positions are kept ﬁxed (resection) with steps where the point
positions are estimated while the camera parameters are kept constant (intersec-
tion). This bilinear algorithm is due to Mahamud et al. (2001), and it is provably
convergent to a critical point of its objective function. It should not, however, be
run for too many iterations, because it usually falls in the basin of attraction of
a degenerate solution after tens of thousands of steps. Algorithms for stitching
together pairs, triples or quadruples of successive views can be found in Beardsley
et al. (1997) and Pollefeys et al. (1999), for example.
Weak calibration is in fact an old problem: as mentioned by Faugeras (1993),
the problem of calculating the epipoles and the epipolar transformations compatible
with seven point correspondences was ﬁrst posed by Chasles (1855) and solved by
Hesse (1863). The problem of estimating the epipolar geometry from ﬁve point
correspondences for internally calibrated cameras was solved by Kruppa (1913).
An excellent modern account of Hesse’s and Kruppa’s techniques can be found in
Faugeras and Maybank (1990), where the absolute conic, an imaginary conic section
invariant through similarities, is used to derive two tangency constraints that make
Section 8.4 Notes 250
up for the missing point correspondences. These methods are of course mostly of
theoretical interest because their reliance on a minimal number of correspondences
limits their ability to deal with noise. The weak-calibration methods of Luong et
al. (1993, 1996) and Hartley (1995) described in this chapter provide reliable and
accurate alternatives.
The problem of computing Euclidean upgrades of projective reconstructions
when some of the intrinsic parameters are known has been addressed by a number
of authors (Heyden and
˚
Astr¨om 1996; Triggs 1997; Pollefeys 1999). The matrix
/ = Q
3
Q
T
3
introduced in Section 8.3.4 can be interpreted geometrically as the pro-
jective representation of the dual of the absolute conic, the absolute dual quadric
(Triggs 1997). Like the absolute conic, this quadric surface is invariant through sim-
ilarities, and the (dual) conic section associated with /
i
/
T
i
is simply the projection
of this quadric surface into the corresponding image. Self-calibration is the process
of computing the intrinsic parameters of a camera from point correspondences with
unknown Euclidean positions. Work in this area was pioneered by Faugeras and
Maybank (1992) for cameras with ﬁxed intrinsic parameters. A number of reliable
self-calibration methods are now available (Hartley 1994a; Fitzgibbon and Zisser-
man 1998; Pollefeys et al. 1999), and they also can be used to upgrade projective
reconstructions to Euclidean ones. The problem of computing Euclidean upgrades
of projective reconstructions under minimal camera constraints such as zero skew
is addressed in Heyden and
˚
Astr¨om (1998, 1999), Pollefeys et al. (1999), Ponce et
al. (2000, 2005), and Vald´es et al. (2006).
PROBLEMS
8.1. We derive in this exercise a method for computing a minimal parameteriza-
tion of the fundamental matrix and estimating the corresponding projection
matrices.
(a) Show that two projection matrices / and /

=
⎛
⎝
a
T
1
b
1
a
T
2
b
2
0
T
1
⎞
⎠
.
Note: For simplicity, you can assume that all the matrices involved in
your solution are nonsingular.
(b) Note that applying this transformation to the projection matrices amounts
to applying the inverse transformation to every scene point P. Let us
denote by
˜
P = (x, y, z)
T
the position of the transformed point
˜
P in the
world coordinate system and by p = (u, v, 1)
T
and p

= (u

, v

, 1)
T
the
homogeneous coordinate vectors of its images. Show that
(u

−b
1
)(a
2
p) = (v

−b
2
)(a
1
p).
(c) Derive from this equation an eight-parameter parameterization of the fun-
damental matrix, and use the fact that T is deﬁned only up to a scale
factor to construct a minimal seven-parameter parameterization.
(d) Use this parameterization to derive an algorithm for estimating T from at
least seven point correspondences and for estimating the projective shape
of the scene.
Section 8.4 Notes 251
8.2. Show that all copies of a set of points in E
3
that are related to each other by
geometric transformations forming a group form an equivalence class.
8.3. Show that the parameter F
33
can be easily estimated from the other entries of
T during the minimization of E in Equation (8.4), resulting in an eigenvalue
problem involving only eight parameters.
Hint: Write that ∂E/∂F
33
= 0 at a minimum of E.
8.4. Generalize this normalization to the estimation of the aﬃne fundamental ma-
trix.
8.5. Show that the eight-point algorithm fails to compute the fundamental matrix
when the eight points and two optical centers lie on a quadric surface.
8.6. Show that only one of the four possible solutions for camera motion found by
the binocular Euclidean SFM approach of Section 8.1 places the reconstructed
points in front of the two cameras.
8.7. Show that aﬃne transformations are the most general class of 4 4 nonsin-
gular matrices that preserve the relationship between coordinates expressed in
Equation (8.6) for any point P
j
.
8.8. Show that the projection matrices associated with two aﬃne cameras can al-
ways be reduced to the canonical forms of Equation (8.10) by an appropriate
aﬃne transformation.
8.9. Show that aﬃne cameras (and the corresponding epipolar geometry) can be
viewed as the limit of perspective images with increasing focal length reced-
ing away from the scene. Use this result to given an alternate derivation of
Equation (8.9).
8.10. Show that, under aﬃne projection, the image of the center of mass of a set of
points is the center of mass of their images.
8.11. Deﬁne the ratio of three collinear points A, B, C as
R(A, B, C) =
AB
BC
for some orientation of the line supporting the three points.
(a) Show that the area of any triangle PQR is
A(P, Q, R) =
1
2
PQRH =
1
2
PQPRsin θ,
where PQ denotes the distance between the two points P and Q, H is the
projection of R onto the line passing through P and Q, and θ is the angle
between the lines joining the point P to the points Q and R.
(b) Show that R(A, B, C) = A(A, B, O)/A(B, C, O), where O is some point
not lying on this line.
8.12. The cross-ratio of four collinear points A, B, C, and D is deﬁned as
|A, B; C, D¦ =
CA
CB
DB
DA
.
(a) Use the result of the previous problem to show that
|A, B; C, D¦ =
sin(a +b) sin(b +c)
sin(a +b +c) sin b
,
where the angles a, b, and c are deﬁned as below.
Section 8.4 Notes 252
B
D
4
D
3
D
2
D
1
A
C
D
D’
B’
O
A’
C’
L
L’
a
b
c
(b) Use this result to deﬁne the cross-ratio of four coplanar lines passing
through the same point.
Hint: Consider the lines L and L

in the diagram.
8.13. Show that there exists a four-parameter family of solutions to the binocular
projective SFM problem of Section 8.3.2, and that this family is given by
Equation (8.18).
8.14. Show that the value of the error E deﬁned by Equation (8.20) is given by Equa-
tion (8.21) at one of its extrema with respect to the variables Z
ij
, assuming
that the data vectors p
ij
have unit norm.
PROGRAMMING EXERCISES
8.15. Implement the eight-point algorithm.
8.16. Implement the estimation of aﬃne epipolar geometry from image correspon-
dences and the estimation of scene structure from the corresponding projection
matrices.
8.17. Implement the Tomasi–Kanade approach to aﬃne shape from motion.
8.18. Implement the binocular projective SFM algorithm of Section 8.3.2.
8.19. Implement the bundle adjustment algorithm of Section 8.3.3.
8.20. Implement the Euclidean algorithm of Section 8.3.4 for cameras with known
image center and zero skew.
P A R T F O U R
MID-LEVEL VISION
This page intentionally left blank
C H A P T E R 9
Segmentation by Clustering
A crucial problem in mid-level vision involves coming up with image repre-
sentations that are simultaneously compact and expressive. These representations
must summarize information available from the ﬁrst stages of visual processing,
and pass them on. Summaries are necessary because early vision produces vast
quantities of information. The richness of the available representation tends to
overwhelm what is signiﬁcant. Useful summaries could be computed from pixels
or from groups of pixels—for example, by constructing groups of pixels that all
have the same color or texture. They could also be computed from local pattern
elements—for example, by collecting together edge points that seem to lie on a
line or on a circle, or close to some complex geometric structure. The core idea
is collecting together pixels or pattern elements into summary representations that
emphasize important, interesting, or distinctive properties.
Obtaining such representation is known variously as segmentation, grouping,
perceptual organization, or ﬁtting. We use the term segmentation for a wide range
of activities because, although techniques may diﬀer, the motivation for all these
activities is the same: obtain a compact representation of what is helpful in the
image. It’s hard to see that there could be a comprehensive theory of segmentation,
not least because what is interesting and what is not depends on the application.
There is certainly no comprehensive theory of segmentation at time of writing, and
the term is used in diﬀerent ways in diﬀerent quarters.
FIGURE 9.1: As these images suggest, an important component of vision involves organiz-
ing image information into meaningful assemblies. The human vision system seems to do
so rather well. In each of these three images, blobs are organized together to form textured
surfaces that appear to bulge out of the page (you may feel that they are hemispheres).
The blobs appear to be assembled “because they form surfaces,” hardly a satisfactory
explanation and one that begs diﬃcult computational questions. Notice that saying that
they are assembled because together they form the same texture also begs questions (how
do we know?). In the case of the surface on the left, it might be quite diﬃcult to write
programs that can recognize a single coherent texture. This process of organization can
be applied to many diﬀerent kinds of input.
255
Section 9.1 Human Vision: Grouping and Gestalt 256
The details of what the summary representation should be depend on the task,
but there are a number of quite general desirable features. First, there should be
relatively few (that is, not more than later algorithms can cope with) components
in the representation computed for typical pictures. Second, these components
should be suggestive. It should be pretty obvious from these components whether
the objects we are looking for are present, again for typical pictures.
FIGURE 9.2: The famous M¨ uller-Lyer illusion; the horizontal lines are in fact the same
length, although that belonging to the lower ﬁgure looks longer. Clearly, this eﬀect arises
from some property of the relationships that form the whole (the gestaltqualit¨ at), rather
than from properties of each separate segment.
There are two important threads in segmentation, which aren’t wholly diﬀer-
ent. In the ﬁrst, our summary is assembled purely locally, by clustering methods
that focus on local relations between items. Here we are trying to assemble items
that look like one another. This approach allows us, for example, to assemble to-
gether clumps of pixels that look similar; such clumps are commonly called regions.
Generally, this approach uses clustering methods, and is the focus of this chapter.
In the second approach, we assemble together items based on global relations—for
example, all items that lie on a straight line. Figure 9.1 shows a collection of small
groups of pixels. When one looks at this ﬁgure, these groups of pixels appear to
belong together, most likely because taken together they suggest the presence of a
surface. In this approach, we are interested in methods that can collect together
tokens or pixels of groups of pixels that, when taken together, suggest the presence
of a structure of some form. This approach emphasizes methods that can identify
parametric models in pools of data; we describe such methods in Chapter 10.
9.1 HUMAN VISION: GROUPING AND GESTALT
A key feature of the human vision system is that context aﬀects how things are
perceived (e.g., see the illusion of Figure 9.2). This observation led the Gestalt
school of psychologists to reject the study of responses to stimuli and to emphasize
grouping as the key to understanding visual perception. To them, grouping meant
the tendency of the visual system to assemble some components of a picture together
and to perceive them together (this supplies a rather rough meaning to the word
context used above). Grouping, for example, is what causes the M¨ uller-Lyer illusion
of Figure 9.2: the vision system assembles the components of the two arrows, and
the horizontal lines look diﬀerent from one another because they are peceived as
components of a whole, rather than as lines. Furthermore, many grouping eﬀects
can’t be disrupted by cognitive input; for example, you can’t make the lines in
Figure 9.2 look equal in length by deciding not to group the arrows.
A common experience of segmentation is the way that an image can resolve
itself into a ﬁgure— typically, the signiﬁcant, important object—and a ground—
Section 9.1 Human Vision: Grouping and Gestalt 257
FIGURE 9.3: One view of segmentation is that it determines which component of the
image forms the ﬁgure and which the ground. The ﬁgure illustrates one form of ambiguity
that results from this view. The white circle can be seen as ﬁgure on the black rectangular
ground, or as ground where the ﬁgure is a black rectangle with a circular hole in it and
the ground is then a white square.
the background on which the ﬁgure lies. However, as Figure 9.3 illustrates, what
is ﬁgure and what is ground can be profoundly ambiguous, meaning that a richer
theory is required.
The Gestalt school used the notion of a gestalt—a whole or a group—and
of its gestaltqualit¨at—the set of internal relationships that makes it a whole (e.g.,
Figure 9.2) as central components in their ideas. Their work was characterized
by attempts to write down a series of rules by which image elements would be
associated together and interpreted as a group. There were also attempts to con-
struct algorithms, which are of purely historical interest (see Gordon (1997) for an
introductory account that places their work in a broad context).
The Gestalt psychologists identiﬁed a series of factors, which they felt predis-
posed a set of elements to be grouped. These factors are important because it is
quite clear that the human vision system uses them in some way. Furthermore, it
is reasonable to expect that they represent a set of preferences about when tokens
belong together that lead to a useful intermediate representation.
There are a variety of factors, some of which postdate the main Gestalt move-
ment:
• Proximity: Tokens that are nearby tend to be grouped.
• Similarity: Similar tokens tend to be grouped together.
• Common fate: Tokens that have coherent motion tend to be grouped to-
Section 9.1 Human Vision: Grouping and Gestalt 258
gether.
• Common region: Tokens that lie inside the same closed region tend to be
grouped together.
• Parallelism: Parallel curves or tokens tend to be grouped together.
• Closure: Tokens or curves that tend to lead to closed curves tend to be
grouped together.
• Symmetry: Curves that lead to symmetric groups are grouped together.
• Continuity: Tokens that lead to continuous—as in joining up nicely, rather
than in the formal sense—curves tend to be grouped.
• Familiar conﬁguration: Tokens that, when grouped, lead to a familiar
object tend to be grouped together.
These laws are illustrated in Figures 9.4, 9.5, 9.7, and 9.1.
Not grouped
Proximity
Similarity
Similarity
Common Fate
Common Region
FIGURE 9.4: Examples of Gestalt factors that lead to grouping (which are described in
greater detail in the text).
These rules can function fairly well as explanations, but they are insuﬃciently
crisp to be regarded as forming an algorithm. The Gestalt psychologists had serious
Section 9.1 Human Vision: Grouping and Gestalt 259
Parallelism
Symmetry
Continuity
Closure
FIGURE 9.5: Examples of Gestalt factors that lead to grouping (which are described in
greater detail in the text).
diﬃculty with the details, such as when one rule applied and when another. It
is diﬃcult to supply a satisfactory algorithm for using these rules; the Gestalt
movement attempted to use an extremality principle.
Familiar conﬁguration is a particular problem. The key issue is to understand
just what familiar conﬁguration applies in a problem and how it is selected. For
example, look at Figure 9.1. One might argue that the blobs are grouped because
Section 9.1 Human Vision: Grouping and Gestalt 260
they yield a sphere. The diﬃculty with this view is explaining how this occurred—
where did the hypothesis that a sphere is present come from? A search through all
views of all objects is one explanation, but one must then explain how this search
is organized. Do we check every view of every sphere with every pattern of spots?
How can this be done eﬃciently?
FIGURE 9.6: Occlusion appears to be an important cue in grouping. It may be possible to
see the pattern on the left as a collection of digits; the pattern next to it is quite clearly
some occluded digits. The black regions in each ﬁgure are the same. The important
diﬀerence between the two ﬁgures seems to be that the superimposed gray regions supply
evidence that the black regions are components of larger objects that are separated for
a reason, rather than just scattered black regions. On the right, two ﬁgures consisting
of tokens that suggest the presence of occluding objects whose boundaries don’t contrast
with much of the image. Notice that one has a clear impression of the position of the
entire contour of the occluding ﬁgures. These contours are known as illusory contours.
The Gestalt rules do oﬀer some insight because they explain what happens
in various examples. These explanations seem to be sensible because they suggest
that the rules help solve problems posed by visual eﬀects that arise commonly in
the real world—that is, they are ecologically valid. For example, continuity may
represent a solution to problems posed by occlusion; sections of the contour of an
occluded object could be joined up by continuity (see Figure 9.6).
This tendency to prefer interpretations that are explained by occlusion leads
to interesting eﬀects. One is the illusory contour, illustrated in Figure 9.6. Here
a set of tokens suggests the presence of an object, most of whose contour has no
contrast. The tokens appear to be grouped together because they provide a cue to
the presence of an occluding object, which is so strongly suggested by these tokens
that one could ﬁll in the no-contrast regions of contour.
This ecological argument has some force because it is possible to interpret
most grouping factors using it. Common fate can be seen as a consequence of the
fact that components of objects tend to move together. Equally, symmetry is a
useful grouping cue because there are a lot of real objects that have symmetric or
close to symmetric contours. Essentially, the ecological argument says that tokens
are grouped because doing so produces representations that are helpful for the visual
world that people encounter. The ecological argument has an appealing, although
vague, statistical ﬂavor. From our perspective, Gestalt factors provide interesting
Section 9.2 Important Applications 261
1 2 3
4 5 6 7
1 2 3
4 5 6 7
FIGURE 9.7: An example of grouping phenomena in real life. The buttons on an elevator in
the computer science building at U.C. Berkeley used to be laid out as in the top ﬁgure. It
was common to arrive at the wrong ﬂoor and discover that this was because you’d pressed
the wrong button; the buttons are diﬃcult to group unambiguously with the correct label,
and it is easy to get the wrong grouping at a quick glance. A public-spirited individual
ﬁlled in the gap between the numbers and the buttons, as in the bottom ﬁgure, and the
confusion stopped because the proximity cue had been disambiguated.
hints, but should be seen as the consequences of a larger grouping process, rather
than the process itself.
9.2 IMPORTANT APPLICATIONS
Simple segmentation algorithms are often useful in signiﬁcant applications. Gener-
ally, simple algorithms work best when it is easy to tell what a useful decomposition
is. Two important cases are background subtraction—where anything that doesn’t
look like a known background is interesting—and shot boundary detection—where
substantial changes in a video are interesting.
More complex algorithms are required for two other very important applica-
tions. In interactive segmentation, a user guides a segmentation system to cut out
an object from a picture. Finally, a major goal is to form image regions.
9.2.1 Background Subtraction
In many applications, objects appear on a largely stable background. The standard
example is detecting parts on a conveyor belt. Another example is counting motor
cars in an overhead view of a road; the road is pretty stable in appearance. Another,
less obvious, example is in human-computer interaction. Quite commonly, a camera
is ﬁxed (say, on top of a monitor) and views a room. Pretty much anything in the
view that doesn’t look like the room is interesting.
In these kinds of applications, a useful segmentation can often be obtained by
Section 9.2 Important Applications 262
FIGURE 9.8: The ﬁgure shows every ﬁfth frame from a sequence of 120 frames of a child
playing on a patterned sofa. The frames are used at an 80 x 60 resolution, for reasons
we discuss in Figure 9.10. Notice that the child moves from one side of the frame to the
other during the sequence.
a
b c
FIGURE 9.9: Background subtraction results for the sequence of Figure 9.8 using 80 x
60 frames. We compare two methods of computing the background: (a) The average of
all 120 frames. Notice that the child spent more time on one side of the sofa than the
other, leading to the faint blur in the average there. (b) Pixels whose diﬀerence from the
average exceeds a small threshold. (c) Those whose diﬀerence from the average exceeds
a somewhat larger threshold. Notice that, in each case, there are some excess pixels and
some missing pixels.
subtracting an estimate of the appearance of the background from the image and
looking for large absolute values in the result. The main issue is obtaining a good
estimate of the background. One method is simply to take a picture. This approach
works rather poorly because the background typically changes slowly over time. For
example, the road may get more shiny as it rains and less when the weather dries
up; people may move books and furniture around in the room; and so on.
An alternative that usually works quite well is to estimate the value of back-
ground pixels using a moving average. In this approach, we estimate the value of
Section 9.2 Important Applications 263
a
b c
FIGURE 9.10: Registration can be a signiﬁcant nuisance in background subtraction, par-
ticularly for textures. These ﬁgures show results for the sequence of Figure 9.8, using
160 x 120 frames. We compare two methods of computing the background: (a) The av-
erage of all 120 frames. Notice that the child spent more time on one side of the sofa
than the other, leading to a faint blur in the average there. (b) Pixels whose diﬀerence
from the average exceeds a small threshold. (c) Those whose diﬀerence from the average
exceeds a somewhat larger threshold. Notice that the number of problem pixels—where
the pattern on the sofa has been mistaken for the child—has markedly increased. This is
because small movements can cause the high spatial frequency pattern on the sofa to be
misaligned, leading to large diﬀerences.
Form a background estimate B
(0)
. At each frame T
Update the background estimate, typically by
forming B
(n+1)
=
w
a
F+

i
w
i
B
(n−i)
w
c
for a choice of weights w
a
, w
i
and w
c
.
Subtract the background estimate from the
frame, and report the value of each pixel where
the magnitude of the diﬀerence is greater than some
threshold.
end
Algorithm 9.1: Background Subtraction.
a particular background pixel as a weighted average of the previous values. Typi-
cally, pixels in the distant past should be weighted at zero, and the weights increase
smoothly. Ideally, the moving average should track the changes in the background,
meaning that if the weather changes quickly (or the book mover is frenetic), rela-
tively few pixels should have nonzero weights, and if changes are slow, the number
of past pixels with nonzero weights should increase. This yields Algorithm 9.1. For
those who have read the ﬁlters chapter, this is a ﬁlter that smooths a function of
time, and we would like it to suppress frequencies that are larger than the typical
frequency of change in the background and pass those that are at or below that
frequency. The approach can be quite successful, but needs to be used on quite
coarse scale images as Figures 9.9 and 9.10 illustrate.
Section 9.2 Important Applications 264
For each frame in an image sequence
Compute a distance between this frame and the
previous frame
If the distance is larger than some threshold,
classify the frame as a shot boundary.
end
Algorithm 9.2: Shot Boundary Detection Using Interframe Diﬀerences.
9.2.2 Shot Boundary Detection
Long sequences of video are composed of shots: much shorter subsequences that
show largely the same objects. These shots are typically the product of the editing
process. There is seldom any record of where the boundaries between shots fall.
It is helpful to represent a video as a collection of shots; each shot can then be
represented with a key frame. This representation can be used to search for videos
or to encapsulate their content for a user to browse a video or a set of videos.
Finding the boundaries of these shots automatically—shot boundary detec-
tion—is an important practical application of simple segmentation algorithms. A
shot boundary detection algorithm must ﬁnd frames in the video that are signiﬁ-
cantly diﬀerent from the previous frame. Our test of signiﬁcance must take account
of the fact that, within a given shot, both objects and the background can move
around in the ﬁeld of view. Typically, this test takes the form of a distance; if the
distance is larger than a threshold, a shot boundary is declared (Algorithm 9.2).
There are a variety of standard techniques for computing a distance:
• Frame diﬀerencing algorithms take pixel-by-pixel diﬀerences between each
two frames in a sequence and sum the squares of the diﬀerences. These algo-
rithms are unpopular, because they are slow—there are many diﬀerences—
and because they tend to ﬁnd many shots when the camera is shaking.
• Histogram-based algorithms compute color histograms for each frame and
compute a distance between the histograms. A diﬀerence in color histograms
is a sensible measure to use because it is insensitive to the spatial arrangement
of colors in the frame (e.g., small camera jitters will not aﬀect the histogram).
• Block comparison algorithms compare frames by cutting them into a grid of
boxes and comparing the boxes. This is to avoid the diﬃculty with color his-
tograms, where a red object disappearing oﬀ-screen in the bottom-left corner
is equivalent to a red object appearing on screen from the top edge. Typically,
these block comparison algorithms compute an interframe distance that is a
composite—taking the maximum is one natural strategy—of interblock dis-
tances, each computed using methods like those used for interframe distances.
• Edge diﬀerencing algorithms compute edge maps for each frame, and then
compare these edge maps. Typically, the comparison is obtained by counting
the number of potentially corresponding edges (nearby, similar orientation,
Section 9.2 Important Applications 265
FIGURE 9.11: A user who wants to cut an object out of an image (left) could mark some
foreground pixels and some background pixels (center), then use an interactive segmen-
tation method to get the cut out components on the right. The method produces a model
of foreground and background pixel appearance from the marked pixels, then uses this
information to decide a ﬁgure ground segmentation. This ﬁgure was originally published
as Figure 9 of “Interactive Image Segmentation via Adaptive Weighted Distances,” by
Protiere and Sapiro, IEEE Transactions on Image Processing, 2007 c _ IEEE, 2007.
etc.) in the next frame. If there are few potentially corresponding edges,
there is a shot boundary. A distance can be obtained by transforming the
number of corresponding edges.
These are relatively ad hoc methods, but are often suﬃcient to solve the problem
at hand.
9.2.3 Interactive Segmentation
People very often want to cut objects out of images and move them into other
images. There are lots of reasons to do this; we sketched some in Section 6.3, where
we described methods to ﬁll in the resulting hole in the source image. But to do
this eﬃciently, we need good ways to select the object we want to cut out. It is too
much work to have to paint the object’s pixels, or its boundary on the image.
This is fairly clearly a segmentation problem, but a special one in which there
are two segments, foreground and background. The foreground segment should be
coherent, but the background segment might not be. Diﬀerent ways to attack the
problem are built around diﬀerent types of interface. In an intelligent scissors
interface, the user sketches a curve fairly close to the boundary of the object;
this curve is then moved to the boundary using local information, typically image
gradient cues. In a painting interface, the user paints some pixels with a foreground
or background brush. These pixels are used to produce an appearance model of
the foreground and of the background. In turn, these models are fed into a fast
Section 9.2 Important Applications 266
FIGURE 9.12: In a grabcut interface for interactive segmentation, a user marks a box
around the object of interest; foreground and background models are then inferred by a
clustering method, and the object is segmented. If this segmentation isn’t satisfactory,
the user has the option of painting foreground and background strokes on pixels to help
guide the model. This ﬁgure was originally published as Figure 1 of “GrabCut Interactive
Foreground Extraction using Iterated Graph Cuts” by C. Rother, V. Kolmogorov, and A.
Blake, ACM Trans. on Graphics (ACM SIGGRAPH Proc), Vol. 23:3 c _ 2004, ACM,
Inc. http: // doi. acm. org/ 10. 1145/ 1186562. 1015720 Reprinted by permission.
graph-based segmenter (Section 9.4.3). Figure 9.11 illustrates the process. Finally,
in a grabcut interface, the user draws a box around the object. This box yields
an initial estimate of foreground and background pixels, and from this we get an
initial segmentation, which yields foreground and background models, which yield
an improved segmentation (Figure 9.12).
Quite often, pixels are neither pure background or pure foreground. For exam-
ple, in a picture of a face, the pixels around the boundary of the hair are somewhat
ambiguous; few pixels here contain only hair, or only background. Instead, because
pixels average light coming in through the lens, most have a value that is a weighted
average of hair and background. In this case, we could use interactive segmentations
to prepare a matte, a mask of values in the range [0−1]. The matte is traditionally
written as α, and our model of the ith pixel value is that it is αf + (1 − α)b,
where f and b are foreground and background values (Figure 9.13). Rotoscoping
is a process like matting, but applied to video; here one recovers a set of segments,
one per frame, corresponding to a moving object. These segments could then be
composited onto a new background, making it look as though the object is mov-
ing against the new background. Matting and rotoscoping methods are strongly
related to segmentation methods, but involve slightly diﬀerent representations. We
give some pointers in the notes.
9.2.4 Forming Image Regions
One application of segmentation is to decompose an image into regions that have
roughly coherent color and texture. Typically, the shape of these regions isn’t
particularly important, but the coherence is important. This process is quite widely
studied—it is often referred to as the exclusive meaning of the term segmentation—
and usually thought of as a ﬁrst step in recognition. Regions are a valuable image
representation in several applications. Regions can oﬀer a route to compressing an
image. Because each region has coherent appearance, we may be able to compress
the image by describing the shape and appearance of the regions separately (as
opposed to describing each pixel independently).
Regions can be used as the backbone of many other visual computations. For
Section 9.2 Important Applications 267
FIGURE 9.13: Matting methods produce a real-valued mask (rather than a foreground-
background mask) to try and compensate for eﬀects in hair, at occluding boundaries, and
so on, where some pixels consist of an average of foreground and background values. The
matte is bright for foreground pixels and dark for background pixels; for some pixels in
the hair, it is gray, meaning that when the foreground is transferred to a new image,
these pixels should become a weighted sum of foreground and background. The gray
value indicates the weight. This ﬁgure was originally published as Figure 6 of “Spectral
Matting,” by A. Levin, A. Rav-Acha, and D. Lischinski, IEEE Transactions on Pattern
Analysis and Machine Intelligence, 2008 c _ IEEE, 2008.
FIGURE 9.14: Superpixels often can expose structure in images that other representations
conceal. Human body segments tend to appear as long, thin segments. In the top row,
an image together with three diﬀerent edge maps (the edge detector of Section 5.2.1,
with two scales of smoothing, and the P
b
of Section 17.1.3) and superpixels computed
at two “scales” (in this case, the number of superpixels was constrained). Notice that
the coarser superpixels tend to expose limb segments in a straightforward way. On the
bottom row, another image, its superpixels, and two versions of the body layout inferred
from the superpixel representation. This ﬁgure was originally published as Figure 3 and
part of Figure 10 of “Recovering human body conﬁgurations: Combining Segmentation and
Recognition,” by G. Mori, X. Ren, A. Efros, and J. Malik, Proc. IEEE CVPR, 2004 c _
IEEE, 2004.
Section 9.3 Image Segmentation by Clustering Pixels 268
example, if we want to identify correspondences between two images—to compute
optic ﬂow or to register parts of the images, say—correspondences between regions
might be the place to start. As a second example, if we want to label images with
the names of the objects that are present, regions help us keep track of what has
been labelled, because we can use them to tell which image pixels correspond to a
particular label. As yet another example, regions could be matched to other regions
within an image to ﬁnd the kind of repetition that one sees in, for example, the
windows on the facade of a building—not quite a texture, but still repetitious.
In some applications, regions need to be quite large and may have a complex
shape. We might want regions to largely respect object boundaries (for example,
if we are labelling objects in images). Most clustering methods can be adapted to
produce segmenters that will construct regions like this.
In other applications, it is more useful to have small, compact regions. These
are usually called superpixels. Superpixels are particularly useful when we need
representation that is small compared to the pixel grid, but still very rich (such a
representation is sometimes called an oversegmentation). One example application
is in computing lightness (Section 2.2.3). If we wanted to represent the shading
ﬁeld, representing it on the pixel grid would be wasteful because it changes slowly;
instead, we might have one shading value per superpixel, and smooth the result.
Other applications are in recognition. For example, human arms and legs tend
to be long and straight; we could try to ﬁnd them by assembling superpixels to
form suggestive groups. This seems to be easier than cutting up large regions
(Figure 9.14).
9.3 IMAGE SEGMENTATION BY CLUSTERING PIXELS
Clustering is a process whereby a data set is replaced by clusters, which are collec-
tions of data points that belong together. It is natural to think of image segmenta-
tion as clustering; we would like to represent an image in terms of clusters of pixels
that belong together. The speciﬁc criterion to be used depends on the application.
Pixels may belong together because they have the same color, they have the same
texture, they are nearby, and so on.
The general recipe for image segmentation by clustering is as follows. We
represent each image pixel with a feature vector. This feature vector contains all
measurements that may be relevant in describing a pixel. Natural feature vectors
include: the intensity at the pixel; the intensity and location of the pixel; the color
of the pixel, represented in whatever color space seems appropriate; the color of the
pixel and its location; and the color of the pixel, its location, and a vector of ﬁlter
outputs from a local texture represenation (compare to Section 6.1). We cluster
these feature vectors. Every feature vector belongs to exactly one cluster, and
so each cluster represents an image segment. We can obtain the image segment
represented by a cluster by replacing the feature vector at each pixel with the
number of that feature vector’s cluster center. You should compare this procedure
with vector quantization (Section 6.2.1), which is what it is. Notice that this
description is extremely general; diﬀerent feature vectors will lead to diﬀerent kinds
of image segment, as will diﬀerent clusterers.
Whether a particular combination of feature vector and clusterer yields good
Section 9.3 Image Segmentation by Clustering Pixels 269
performance depends on what one needs. It is possible to make some general state-
ments, though. The general recipe doesn’t guarantee that segments are connected,
which may or may not matter. If one is segmenting images to compress them,
then encoding the US ﬂag as three segments (red, white and blue) might be a good
choice; if one is segmenting to represent objects, this is probably a poor represen-
tation, because it regards all white stars as a single segment. If the feature vector
contains a representation of the position of the pixel, the segments that result tend
to be “blobby,” because pixels that lie very far from the center of a segment will
tend to belong to other clusters. This is one way to ensure that segments are con-
nected. Representing color information tends to make segmenters better, because
in this case it’s hard to get easy images wrong (color doesn’t seem to make hard
images easier, though). For some applications, doing well at easy images is enough.
9.3.1 Basic Clustering Methods
There are two natural algorithms for clustering. In divisive clustering, the entire
data set is regarded as a cluster, and then clusters are recursively split to yield a
good clustering (Algorithm 9.4). In agglomerative clustering, each data item is
regarded as a cluster, and clusters are recursively merged to yield a good clustering
(Algorithm 9.3).
Make each point a separate cluster
Until the clustering is satisfactory
Merge the two clusters with the
smallest inter-cluster distance
end
Algorithm 9.3: Agglomerative Clustering or Clustering by Merging.
Construct a single cluster containing all points
Until the clustering is satisfactory
Split the cluster that yields the two
components with the largest inter-cluster distance
end
Algorithm 9.4: Divisive Clustering, or Clustering by Splitting.
There are two major issues in thinking about clustering:
• What is a good inter-cluster distance? Agglomerative clustering uses an inter-
cluster distance to fuse nearby clusters; divisive clustering uses it to split in-
suﬃciently coherent clusters. Even if a natural distance between data points
is available (which might not be the case for vision problems), there is no
canonical inter-cluster distance. Generally, one chooses a distance that seems
appropriate for the data set. For example, one might choose the distance be-
Section 9.3 Image Segmentation by Clustering Pixels 270
tween the closest elements as the inter-cluster distance, which tends to yield
extended clusters (statisticians call this method single-link clustering). An-
other natural choice is the maximum distance between an element of the ﬁrst
cluster and one of the second, which tends to yield rounded clusters (statis-
ticians call this method complete-link clustering). Finally, one could use an
average of distances between elements in the cluster, which also tends to yield
“rounded” clusters (statisticians call this method group average clustering).
• How many clusters are there? This is an intrinsically diﬃcult task if there
is no model for the process that generated the clusters. The algorithms we
have described generate a hierarchy of clusters. Usually, this hierarchy is
displayed to a user in the form of a dendrogram—a representation of the
structure of the hierarchy of clusters that displays inter-cluster distances—
and an appropriate choice of clusters is made from the dendrogram (see the
example in Figure 9.15).
d
i
s
t
a
n
c
e
1 2 3 4 5 6
1
2
3
4
5
6
FIGURE 9.15: Left, a data set; right, a dendrogram obtained by agglomerative clustering
using single-link clustering. If one selects a particular value of distance, then a horizontal
line at that distance splits the dendrogram into clusters. This representation makes it
possible to guess how many clusters there are and to get some insight into how good the
clusters are.
The main diﬃculty in using either agglomerative or divisive clustering meth-
ods directly is that there are an awful lot of pixels in an image. There is no rea-
sonable prospect of examining a dendrogram because the quantity of data means
that it will be too big. In practice, this means that the segmenters decide when
to stop splitting or merging by using a set of threshold tests. For example, an
agglomerative segmenter might stop merging when the distance between clusters
is suﬃciently low or when the number of clusters reaches some value. A divisive
clusterer might stop splitting when the resulting clusters meet some similarity test.
Section 9.3 Image Segmentation by Clustering Pixels 271
FIGURE 9.16: Segmentation results from the watershed algorithm, applied to an image
by Martin Brigdale. Center: watershed applied to the image intensity; notice some long
superpixels. Right: watershed applied to image gradient magnitude; this tends to produce
rounder superpixels. Martin Brigdale c _ Dorling Kindersley, used with permission.
It is straightforward to modify both divisive and agglomerative clusterers to
ensure that regions are connected. Agglomerative clusterers need to merge only
clusters with shared boundaries. It is more diﬃcult to modify divisive clusterers,
which need to ensure that the children of any split are connected. One way to do
this is to split along spatial boundaries in the segment being split. It is usually
impractical to look for the best split of a cluster (for a divisive method) or the
best merge (for an agglomerative method). Divisive methods are usually modiﬁed
by using some form of summary of a cluster to suggest a good split (for example,
a histogram of pixel colors). Agglomerative methods also need to be modiﬁed,
because the number of pixels means that one needs to be careful about the inter-
cluster distance (the distance between cluster centers of gravity is often used).
Finally, it can be useful to merge regions simply by scanning the image and merging
all pairs whose distance falls below a threshold, rather than searching for the closest
pair.
9.3.2 The Watershed Algorithm
An early segmentation algorithm that is still widely used is the watershed algorithm.
Assume we wish to segment image J. In this algorithm, we compute a map of the
image gradient magnitude, [[ ∇J [[. Zeros of this map are locally extreme intensity
values; we take each as a seed for a segment, and give each seed a unique label.
Now we assign pixels to seeds by a procedure that is, rather roughly, analogous to
ﬁlling a height map with water (hence the name). Imagine starting at pixel (i, j);
if we travel backward down the gradient of [[ ∇J [[, we will hit a unique seed. Each
pixel gets the label of the seed that is hit by this procedure.
You should recognize this description as a form of shortest path algorithm;
it can also be seen as a form of agglomerative clusterer. We start with seed clus-
ters, then agglomerate pixels to clusters when the path to the cluster is “downhill”
Section 9.3 Image Segmentation by Clustering Pixels 272
from the pixel. This means that one can produce rather more eﬃcient algorithms
than the one we sketched, and there is a considerable literature of these algo-
rithms. In this literature, authors tend to criticize the watershed algorithm for
oversegmentation—that is, for producing “too many” segments. More recently,
watershed algorithms are quite widely used because they produce tolerable super-
pixels, and are eﬃcient. Good implementations of watershed algorithms are widely
available; to produce Figure 9.16, we used the implementation in Matlab’s Image
processing toolbox. It is natural to use the gradient magnitude to drive a watershed
transform, because this divides the image up into regions of relatively small gradi-
ent; however, one could also use the image intensity, in which case each region is the
domain of attraction of an intensity minimum or maximum. Gradient watersheds
tend to produce more useful superpixels (Figure 9.16).
9.3.3 Segmentation Using K-means
There is a strong resonance between image segmentation as we have described
it, and vector quantization. In Chapter 6, we described vector quantization in
the context of texture representation and introduced the k-means algorithm. K-
means produces good image segments for some applications. Following the general
recipe, we compute a feature vector representing each pixel, we apply k-means, and
each pixel goes to the segment represented by the cluster center that claims its
feature vector. The main consequence of using k-means is that we know how many
segments there will be. For some applications, this is a good thing; for example,
the segmentations of Figure 9.17 use ﬁve segments, and essentially represent a
requantization of the image gray-levels (or colors, respectively) to ﬁve levels. This
can be useful for some coding and compression applications.
FIGURE 9.17: On the left, an image of mixed vegetables, which is segmented using k-means
to produce the images at center and on the right. We have replaced each pixel with the
mean value of its cluster; the result is somewhat like an adaptive requantization, as one
would expect. In the center, a segmentation obtained using only the intensity information.
At the right, a segmentation obtained using color information. Each segmentation assumes
ﬁve clusters.
One diﬃculty with using this approach for segmenting images is that segments
Section 9.3 Image Segmentation by Clustering Pixels 273
FIGURE 9.18: Here we show the image of vegetables segmented with k-means, assuming a
set of 11 components. The left ﬁgure shows all segments shown together, with the mean
value in place of the original image values. The other ﬁgures show four of the segments.
Note that this approach leads to a set of segments that are not necessarily connected.
For this image, some segments are actually quite closely associated with objects, but one
segment may represent many objects (the peppers); others are largely meaningless. The
absence of a texture measure creates serious diﬃculties, as the many diﬀerent segments
resulting from the slice of red cabbage indicate.
are not connected and can be very widely scattered (Figures 9.17 and 9.18). This
eﬀect can be reduced by using pixel coordinates as features—an approach that
results in large regions being broken up (Figure 9.19).
FIGURE 9.19: Five of the segments obtained by segmenting the image of vegetables with a
k-means segmenter that uses position as part of the feature vector describing a pixel, now
using 20 segments rather than 11. Note that the large background regions that should be
coherent have been broken up because points got too far from the center. The individual
peppers are now better separated, but the red cabbage is still broken up because there is
no texture measure.
9.3.4 Mean Shift: Finding Local Modes in Data
Clustering can be abstracted as a density estimation problem. We have a set of
sample points in some feature space, which came from some underlying probability
density. Comaniciu and Meer (2002) created an extremely important segmenter,
using the mean shift algorithm, which thinks of clusters as local maxima (local
modes) in this density. To do so, we need an approximate representation of the
density. One way to build an approximation is to use kernel smoothing. Here we
take a set of functions that look like “blobs” or “bumps,” place one over each data
point, and so produce a smooth function that is large when there are many data
points close together and small when the data points are widely separated.
Section 9.3 Image Segmentation by Clustering Pixels 274
This is a quite general strategy that applies to many diﬀerent bump functions.
We will use a speciﬁc kernel smoother, writing
K(x; h) =
(2π)
(−d/2)
h
d
exp
_
−
1
2
[[ x[[
2
h
_
for the bump function. We introduce a positive scale parameter h, which we can
adjust to get the best representation. Then, our model of the density is
f(x) =
_
1
n
_
n

i=1
K (x
i
−x; h)
(you should check that this is a density, i.e., that it is non-negative, and that its
integral is one). We can estimate h by maximizing the average likelihood of held-
out data. In this procedure, we repeat the following experiment numerous times:
hold out one data point at random, ﬁt a density model to the remaining data, and
then compute the likelihood of the held-out data point as a function of h (perhaps
by computing values at a set of diﬀerent sample points in h). We average the
likelihoods computed in this way, then use the h that maximizes that likelihood.
We can simplify notation by writing k(u) = exp
_
−
1
2
u
_
(this is called the
kernel proﬁle) and C =
(2π)
(−d/2)
nh
d
, so that
f(x) = C
n

i=1
k
_
[[
x −x
i
h
[[
2
_
(9.1)
Now write g =
d
du
k(u). Starting from some point x
0
, we should like to ﬁnd a
nearby point that maximizes the density value. We will use this local maximum
(local mode) as a cluster center. The mean shift procedure maximizes expressions
of the form of Equation 9.1. We are seeking y such that the gradient ∇f vanishes
at that point. We must require that
∇f(x) [
x=y
= 0
= C

i
g([[
x
i
−y
(j)
h
[[
2
)
.
The procedure gets its name from the fact that we are shifting to a point which
has the form of a weighted mean (see Algorithm 9.5).
Start with an estimate of the mode y
(0)
and a set of n data vectors x
i
of dimension d, a scaling constant h, and g the derivative of the kernel proﬁle
Until the update is tiny
Form the new estimate
y
(j+1)
=

i
x
i
g(||
x
i
−y
(j)
h
||
2
)

i
g(||
x
i
−y
(j)
h
||
2
)
Algorithm 9.5: Finding a Mode with Mean Shift.
9.3.5 Clustering and Segmentation with Mean Shift
Clustering with mean shift is, in principle, straightforward. We start the mean shift
procedure at every data point, producing a mode for each data point. Because we
are working with continuous variables, every one of these modes is diﬀerent, but
we expect the modes to be very tightly clustered. There should be a small set of
actual modes, and each of these estimates is very close to one of them. These esti-
mates are themselves useful, because they represent a form of ﬁltering of the image.
We could replace each pixel with its mode representation; this gives a signiﬁcant
smoothing of the image in a way that respects image boundaries (Figure 9.20). To
cluster the data, we apply, say, an agglomerative clusterer to the mode estimates.
Because we expect the modes to be very tightly clustered, group average distance
is a good choice of distance, and we can stop clustering when this distance exceeds
a small threshold. This will produce a set of small, tight clusters that are widely
separated. We now map each data point to the cluster center corresponding to its
mode (Algorithm 9.6).
This recipe can be applied nearly directly to image segmentation. We repre-
sent each pixel with a feature vector, then cluster the feature vectors; each cluster
center represents a segment, and we replace each pixel with the number of its cluster
center. Improved performance can be obtained by a representation that balances
spatial and appearance features more explicitly. In particular, we represent the ith
pixel with a feature vector x
i
which has two components: x
s
i
, which has dimension
Section 9.3 Image Segmentation by Clustering Pixels 276
= (8, 8)
(8, 16) (16, 8)
(h , h )
s r
(h , h )=
s r
(h , h )=
s r
FIGURE 9.20: An image (top left) and mean shift modes obtained with diﬀerent clustering
scales for space h
s
and appearance h
r
. If h
s
is small, the method must produce clusters
that are relatively small and compact spatially because the kernel function smoothes over
a relatively small radius and so will allow many distinct modes. If h
r
is small, the clusters
are compact in appearance; this means that small h
s
and large h
r
will produce small,
blobby clusters that could span a range of appearances, whereas large h
s
and small h
r
will
tend toward spatially complex and extended clusters with a small range of appearances.
Cluster boundaries will try harder to follow level curves of intensity. This ﬁgure was
originally published as Figure 5 of “Mean Shift: A Robust Approach Toward Feature Space
Analysis,” by D. Comaniciu and P. Meer, IEEE Transactions on Pattern Analysis and
Machine Intelligence, 2002 c _ IEEE, 2002.
For each data point x
i
Apply the mean shift procedure (Algorithm 9.5), starting with y
(0)
= x
i
Record the resulting mode as y
i
Cluster the y
i
, which should form small tight clusters.
A good choice is an agglomerative clusterer with group average distance,
stopping clustering when the group average distance exceeds a small threshold
The data point x
i
belongs to the cluster that its mode y
i
belongs to.
Algorithm 9.6: Mean Shift Clustering.
Section 9.4 Segmentation, Clustering, and Graphs 277
d
s
and represents the location of the pixel and x
r
i
, which has dimension d
r
and
represents everything else. Now we use two kernels and two smoothing parameters
in the density estimation procedure, writing
K(x; h
s
, h
r
) =
_
(2π)
(−d
s
/2)
h
d
s
s
k
_
x
s
h
s
__ _
(2π)
(−d
r
/2)
h
d
r
r
k
_
x
r
h
r
__
.
This means that we can balance spatial and appearance clustering and require, for
example, spatially tight clusters with a wide range of appearances, and so on. In
this case, the mean shift update equation changes slightly (Exercises).
For each pixel, p
i
, compute a feature vector x
i
= (x
s
i
, x
r
i
) representing
spatial and appearance components, respectively.
Choose h
s
, h
r
the spatial (resp. appearance) scale of the smoothing kernel.
Cluster the x
i
using this data and mean shift clustering (Algorithm 9.6).
(Optional) Merge clusters with fewer than t
min
pixels with a neighbor;
the choice of neighbor is not signiﬁcant, because the cluster is tiny.
The i’th pixel belongs to the segment corresponding to its cluster center
(for example, one could label the cluster centers 1 . . . r, and then
identify segments by computing a map of the labels corresponding to pixels).
Algorithm 9.7: Mean Shift Segmentation.
9.4 SEGMENTATION, CLUSTERING, AND GRAPHS
Clustering algorithms deal with similarity between data items. Some algorithms
may summarize data items (for example, the cluster centers in the k-means algo-
rithm), but the core issue is similarity between data items. It might not be useful
to compare all pairs of data items; for example, there might be little real advantage
in comparing very distant image pixels directly. All this rather naturally suggests a
graph. Each data item would be a vertex. There would be a weighted edge between
all pairs of data items that could usefully be compared. The process of clustering
the data then becomes one of segmenting the graph into connected components.
9.4.1 Terminology and Facts for Graphs
We review terminology here very brieﬂy, as it’s quite easy to forget.
• A graph is a set of vertices V and edges E that connect various pairs of
vertices. A graph can be written G = ¦V, E¦. Each edge can be represented
by a pair of vertices—that is, E ⊂ V V . Graphs are often drawn as a set
of points with curves connecting the points.
• The degree of a vertex is the number of edges incident on that vertex.
Section 9.4 Segmentation, Clustering, and Graphs 278
FIGURE 9.21: Segmentations of images obtained using the mean shift algorithm. This
ﬁgure was originally published as Figure 10 of “Mean Shift: A Robust Approach Toward
Feature Space Analysis,” by D. Comaniciu and P. Meer, IEEE Transactions on Pattern
Analysis and Machine Intelligence, 2002 c _ IEEE, 2002.
• A directed graph is one in which edges (a, b) and (b, a) are distinct; such a
graph is drawn with arrowheads indicating which direction is intended.
• An undirected graph is one in which no distinction is drawn between edges
(a, b) and (b, a).
• A weighted graph is one in which a weight is associated with each edge.
• Two edges are consecutive if they have a vertex in common.
• A path is a sequence of consecutive edges.
• A circuit is a path which ends at the vertex at which it begins.
• A self-loop is an edge that has the same vertex at each end; self-loops don’t
occur in our applications.
• Two vertices are said to be connected when there is a sequence of edges starting
at the one and ending at the other; if the graph is directed, then the arrows
in this sequence must point the right way.
• A connected graph is one where every pair of vertices is connected.
• A tree is a connected graph with no circuits.
Section 9.4 Segmentation, Clustering, and Graphs 279
• Given a connected graph G = ¦V, E¦, a spanning tree is a tree with vertices V
and edges a subset of E. By our deﬁnition, trees are connected, so a spanning
tree is connected.
• Every graph consists of a disjoint set of connected components—that is, G =
¦V
1
∪ V
2
. . . V
n
, E
1
∪ E
2
. . . E
n
¦, where ¦V
i
, E
i
¦ are all connected graphs and
there is no edge in E that connects an element of V
i
with one of V
j
for i ,= j.
• A forest is a graph whose connected components are trees.
In a weighted graph, there are eﬃcient algorithms for computing minimum
weight spanning trees (see, for example, Jungnickel (1999) or Cormen et al. (2009)).
Another very important problem that can be solved eﬃciently seeks to maximize
ﬂow in a directed graph. In particular, in a directed graph identify one vertex as a
source s and another as a target t. Associate with each directed edge e a capacity,
c(e), which is a non-negative number. A ﬂow is a non-negative value f(e) associated
with each edge with the following properties. First, 0 ≤ f(e) ≤ c(e). Second, at
any vertex v ∈ ¦V −s −t¦,

earriving at v
f(e) −

eleaving fromv
f(e) = 0
(i.e., all ﬂow arriving at a vertex leaves it; this is Kirchoﬀ’s law). The value of a
ﬂow is

earriving att
f(e).
There are eﬃcient algorithms to maximize the ﬂow in, for example, Ahuja et al.
(1993) or Cormen et al. (2009). A dual problem is also interesting. Decompose the
vertices into two disjoint sets o and T , such that s ∈ o and t ∈ T . This represents
a cut. Consider J ∈ E, the set of directed edges from o to T . The value of the
cut is

e∈W
c(e).
The value of the cut can again be minimized eﬃciently; algorithms appear in, for
example, Ahuja et al. (1993), Jungnickel (1999), or Schrijver (2003).
9.4.2 Agglomerative Clustering with a Graph
Felzenszwalb and Huttenlocher (2004) showed how to use graph theoretic ideas to
build a straightforward but very eﬀective segmenter based around an agglomer-
ative clusterer. Represent the image as a weighted graph. There are edges be-
tween any pair of pixels that are neighbors. Each edge has a weight that measures
dissimilarity—i.e., weights are large if pixels are very diﬀerent, and small if they
are similar. The weights could come from a variety of pixel representations. For
example, we could use the squared diﬀerence in intensity; we could represent the
color at each pixel with a vector, and use the length of the diﬀerence vector; we
could represent the texture at each pixel with a vector of ﬁlter outputs (after the
Section 9.4 Segmentation, Clustering, and Graphs 280
Start with a set of clusters (
i
, one cluster per pixel.
Sort the edges in order of non-decreasing edge weight, so that
w(e
1
) ≥ w(e
2
) ≥ . . . ≥ w(e
r
).
For i = 1 to r
If the edge e
i
lies inside a cluster
do nothing
Else
One end is in cluster (
l
and the other is in cluster (
m
If diff((
l
, (
m
) ≤ MInt((
l
, (
m
)
Merge (
l
and (
m
to produce a new set of clusters.
Report the remaining set of clusters.
Algorithm 9.8: Agglomerative Clustering with Graphs.
local texture representations of Section 6.1), then use the length of the diﬀerence
vector; or we could use a weighted sum of all these distances.
We will start with every pixel forming a cluster, then merge clusters until
there is no need to continue. To do this, we need some notion of the distance
between two clusters. Each cluster is a component of the graph, formed from all
the vertices (pixels) in the cluster, and all the edges that start and end inside the
cluster. Then the diﬀerence between two components is the minimum weight edge
connecting two components. Write (
1
, (
2
for the two components, c for the edges,
and w(v
1
, v
2
) for the weight of the edge joining v
1
and v
2
. Then, we have
diﬀ((
1
, (
2
) = min
v
1
∈C
1
,v
2
∈C
2
,(v
1
,v
2
)∈E
w(v1, v2).
It is also helpful to know how coherent a particular cluster is. This will help us
stop clustering. We deﬁne the internal diﬀerence of a component to be the largest
weight in the minimum spanning tree of the component. Write M(() = ¦V
C
, E
M
¦
for the minimum spanning tree of (. Then, we have
int(() = max
e∈M(C)
w(e).
We will start with a set of clusters (or segments) that consists of all the pixels,
one cluster per pixel. We then merge clusters iteratively. We do so by sorting
all edges in order of non-decreasing edge weight. For each edge, starting with the
smallest, we consider the clusters at either end of the edge. If both ends of the
edge lie in the same cluster, there is nothing to do. If there are distinct clusters
at each end, then we could merge them. We do so when the edge weight is small
compared to the internal diﬀerence of each of the clusters (this requires some care
for small clusters; details below). We now proceed through all the edges, merging
as necessary. The ﬁnal segmentation is the set of clusters once the last edge has
been visited (Algorithm 9.8).
Section 9.4 Segmentation, Clustering, and Graphs 281
FIGURE 9.22: Images segmented using Algorithm 9.8, shown next to segments. Figures
obtained from http://people.cs.uchicago.edu/
~
pff/segment/, by kind permission of
Pedro Felzenszwalb.
Comparing the edge weight to the internal diﬀerence of the clusters requires
some care, because in small clusters the internal distance might be zero (if there
is only one vertex), or implausibly small. To deal with this, Felzenszwalb and
Huttenlocher (2004) deﬁne a function of two clusters, MInt, as
MInt((
1
, (
2
) = min(int((
1
) +τ((
1
), int((
2
) +τ((
2
))
where τ(() is a term that biases the internal diﬀerence upward for small clusters;
Felzenszwalb and Huttenlocher (2004) use τ(() = k/ [ ( [, for k some constant
parameter. This algorithm is notably fast and relatively accurate (Figure 9.22).
9.4.3 Divisive Clustering with a Graph
As we have seen (Section 9.2.3), it is extremely useful to separate an image into
a foreground and background based on examples. Assume we have a map of pix-
els, one per image pixel, where each pixel in the map carries one of three labels:
foreground, background or unknown (these maps are sometimes known as trimaps)
depending on whether the corresponding image is in the foreground, background
or is unknown. We should like to take the foreground and background pixels, build
models from these, then label the unknown pixels with these models. There are two
important constraints on the labels. First, a pixel that looks like the foreground
examples should get a foreground label (similarly for background). Second, pixels
should tend to have labels that are the same as their neighbors’.
Boykov and Jolly (2001) phrase this problem as energy minimization. Write T
for the set of pixels with foreground labels, B for the set with background labels, and
| for the unknown pixels. We associate a binary variable δ
i
with the ith unknown
Section 9.4 Segmentation, Clustering, and Graphs 282
pixel. We adopt the convention that δ
i
= −1 if the ith pixel is background, and
δ
i
= 1 if the ith pixel is foreground. We will ﬁnd a value for this binary variable
that minimizes an energy function. The energy will have two types of term. The
ﬁrst type of term will encourage pixels similar to the foreground (resp. background)
model to have foreground (resp. background) labels. The second will encourage
pixels to have the same label as their neighbors.
Write p
i
for a vector representing the ith pixel. The vector could contain
intensity; intensity and color; intensity, color and texture; or other information.
Write d
f
(p) for a function that compares the pixel vector p with the foreground
model; this function is large when the pixel is not like the foreground, and small
when it is like the foreground. Similarly, write d
b
(p) for a function that compares
the pixel vector with the background. Write A(i) for the neighbors of pixel i. Write
B(p
i
, p
j
) for a non-negative, symmetric function that compares two pixels, which
we will use as a cost for assigning neighboring pixels to diﬀerent models. This could
be as simple as a constant, which just encourages neighboring pixels to have the
same label. More complicated B should be large for two pixels that are similar,
and small for diﬀerent pixels; in this case, we will encourage the label to change
between pixels that look diﬀerent.
Notice that (
1
2
)(1 − δ
i
δ
j
) has the value 1 when δ
i
and δ
j
are diﬀerent, and 0
otherwise. Write J for the set of all pixels, | for the set of unknown pixels, T for
the set of known foreground pixels, and B for the set of known background pixels.
Now we can write an energy function
E
∗
(δ) =

i∈I
d
f
(p
i
)
1
2
(1 +δ
i
) +d
b
(p
i
)
1
2
(1 −δ
i
) +

i∈I

j∈N(i)
B(p
i
, p
j
)(
1
2
)(1 −δ
i
δ
j
)
which we must minimize subject to δ
k
= 1 for k ∈ T and δ
k
= 0 for k ∈ B. Notice
that we can make this energy function small by labelling pixels that agree with the
foreground model with δ = 1, those that agree with the background model with
δ = −1, and ensuring that labels change at pixels that look diﬀerent (i.e., where
B is small). Minimizing this energy might be hard, because it is a combinatorial
problem (δ
j
can take only two values).
It turns out that minimizing E can be rephrased as minimizing a cut on a
graph. The easiest way to see this is with a ﬁgure. Imagine a cut on the graph of
Figure 9.23. In this graph, each pixel is represented by a vertex, the source ver-
tex corresponds to the foreground label, and the target vertex corresponds to the
background label. There is one edge connecting each pixel to the source and one
connecting it to the target; we can cut the graph by cutting only one of these two
edges, and if we cut both, the cut is not minimal. We can interpret a cut that cuts
only one of these edges as a map from a pixel to foreground (resp. background)
depending on whether the edge to the source (resp. target) remains uncut. Fur-
thermore, the value of a cut that cuts only one of these two edges for each pixel
is the same as the value of the energy function E for the corresponding labelling.
As a result, we can minimize the energy function by computing the minimum cut.
This is known to be polynomial (from the references in Section 9.4.1), but in fact
Section 9.4 Segmentation, Clustering, and Graphs 283
S
D
S
D
d
B
d
B
edge weight case
(i, j) B(p
i
, p
j
) i, j, neighbors
(S →i)
K
0
d
f
(i)
p ∈ T
p ∈ B
otherwise
(i →D)
K
0
d
b
(i)
p ∈ B
p ∈ T
otherwise
FIGURE 9.23: On the left, a graph derived from an image to set up foreground/background
segmentation as a graph cut problem. We interpret pixels linked to the source (S) as
foreground pixels, and pixels linked to the drain (D) as background pixels. Some pixels—
whose labels are known—are linked to only one of the two, and to their neighbors. Link
weights are given in the table. The links between neighbors have the same capacity in each
direction, which is why they are drawn without a direction. On the right, a cut of that
graph (edges that have been cut are grayed out). Notice that each pixel is linked to either
the foreground or to the background, but not to both (because otherwise we would not
have disconnected S and D) or to neither (because we could restore one of the two edges
and get a cut with a better value). Furthermore, the sum of weights of cut edges is equal
to the energy cost function. As a result, we can segment the image into foreground and
background by solving for the minimum cost cut. With the weights shown in the table, the
value of a cut on the graph is the same as the value of the energy function, as long as the
cut does not cut both (S → i) and (i → D), and K = 1 + max
p∈I

q:{p,q}∈N
B(p, q).
A minimum cut will not cut both, because a better cut will cut only one; this means that
the energy function in the text can be minimized by cutting the graph.
specialized algorithms are now very fast at cutting graphs from images.
This procedure gives us one way to deal with the problem of Section 6.3.2.
Here we had a hole in an image and a patch that matched the hole; but the patch
typically is square, and the hole typically is not. Place the patch over the hole. For
some pixels we know only one value (those inside the hole, and those outside the
patch), but for others we know two values. For these, we would like to choose which
pixel appears in the ﬁnal image. Again, we have a combinatorial problem. Write δ
i
for a variable that takes the value −1 if the ith pixel in the ﬁnal image should come
from the patch, and 1 otherwise. Write | for the pixels that could take either label,
T for the pixels that can take values only from the patch, and J for the pixels that
can take values only from the image. We do not have a foreground or background
model. Generally, we would like pixels to have a δ that agrees with their neighbors.
When two neighboring pixels have diﬀerent δ values (i.e., at a point where we cut
Section 9.4 Segmentation, Clustering, and Graphs 284
Property Aﬃnity function Notes
Distance exp
_
−
_
(x −y)
t
(x −y)/2σ
2
d
__
Intensity exp
_
−
_
(I(x) −I(y))
t
(I(x) −I(y))/2σ
2
I
__
I(x) is the intensity
of the pixel at x.
Color exp
_
−
_
dist(c(x), c(y))
2
/2σ
2
c
__
c(x) is the color
of the pixel at x.
Texture exp
_
−
_
(f(x) −f(y))
t
(f(x) −f(y))/2σ
2
I
__
f(x) is a vector
of ﬁlter outputs
describing the
pixel at x
computed as
in Section 6.1.
TABLE 9.1: Diﬀerent aﬃnity functions comparing pixels for a graph based segmenter.
Notice that aﬃnities can be combined. One attractive feature of the exponential form is
that, say, location, intensity and texture aﬃnities could be combined by multiplying them.
from patch to image), we would like the actual values of the pixels to be as similar
as possible; this is to ensure that we blend at places where the image agrees with the
patch. These criteria can be written into an energy function that can be minimized
with graph cuts.
9.4.4 Normalized Cuts
Segmenting an image by min-cut usually does not work well without good fore-
ground and background models. This is because one can get very good cut values
by cutting oﬀ small groups of pixels. The cut does not balance the diﬀerence be-
tween segments with the coherence within segments. Shi and Malik (2000) suggest
a normalized cut: cut the graph into two connected components such that the cost
of the cut is a small fraction of the total aﬃnity within each group.
To do this, we need a measure of aﬃnity between pixels. We will model the
image as a graph with one vertex at each pixel, and an edge from each pixel to
all its neighbors. We must place a weight on each edge, which we will call the
aﬃnity between the pixels. The detailed form of the aﬃnity measure depends on
the problem at hand. The weight of an arc connecting similar nodes should be
large, and the weight on an arc connecting diﬀerent nodes should be small (in the
last section, B was the cost of cutting an edge, and so was small when pixels were
similar, and large when they were diﬀerent). Table 9.1 gives some aﬃnity functions
in current use.
Recall that a normalized cut must cut the graph into two connected compo-
nents such that the cost of the cut is a small fraction of the total aﬃnity within
each group. We can formalize this as decomposing a weighted graph V into two
components A and B and scoring the decomposition with
cut(A, B)
assoc(A, V )
+
cut(A, B)
assoc(B, V )
Section 9.5 Image Segmentation in Practice 285
(where cut(A, B) is the sum of weights of all edges in V that have one end in A and
the other in B, and assoc(A, V ) is the sum of weights of all edges that have one end
in A). This score is small if the cut separates two components that have few edges
of low weight between them and many internal edges of high weight. We would like
to ﬁnd the cut with the minimum value of this criterion, called a normalized cut.
The criterion is successful in practice (Figure 9.24).
This problem is too diﬃcult to solve in this form, because we would need to
look at every graph cut. It is a combinatorial optimization problem, so we can’t
use continuity arguments to reason about how good a neighboring cut is given the
value of a particular cut. Worse, it’s an NP-complete problem, even for grid graphs.
However, Shi and Malik (2000) give an approximation algorithm that generates a
good cut.
FIGURE 9.24: The images on top are segmented using the normalized cuts framework,
described in the text, into the components shown. The aﬃnity measures used involved
intensity and texture, as in Table 9.1. The image of the swimming tiger yields one seg-
ment that is essentially tiger, one that is grass, and four components corresponding to the
lake. Similarly, the railing shows as three reasonably coherent segments. Note the im-
provement over k-means segmentation obtained by having a texture measure. This ﬁgure
was originally published as Figure 2 of “Image and video segmentation: the normalized
cut framework,” by J. Shi, S. Belongie, T. Leung, and J. Malik, Proc. IEEE Int. Conf.
Image Processing, 1998 c _ IEEE, 1998.
9.5 IMAGE SEGMENTATION IN PRACTICE
Code is now available for many important image segmenters. The EDISON codes
(from Rutgers’ computer vision group, available at http://coewww.rutgers.edu/
Section 9.5 Image Segmentation in Practice 286
FIGURE 9.25: Segmenters and edge detectors can be evaluated by comparing the predicted
boundaries to object boundaries that people mark on images. A natural comparison
involves precision (the percentage of the marked boundary points that are real ones)
and recall (the percentage of real boundary points that were marked); the F measure
summarizes precision and recall into a single number F = 2PR/(P + R). On the left,
these measures for various segmenters; on the right, for various edge detectors. This ﬁgure
was originally published as Figures 1 and 2 of “Contour Detection and Hierarchical Image
Segmentation” by P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2011, c _ IEEE, 2011.
riul/research/robust.html) implement mean shift image segmentation (Sec-
tion 9.3.5). The same web page distributes a variety of other mean shift codes.
Pedro Felzenszwalb distributes code for his segmenter (Section 9.4.2) at http:
//people.cs.uchicago.edu/
~
pff/segment/. Jianbo Shi distributes code for nor-
malized cuts at http://www.cis.upenn.edu/
~
jshi/software/. Greg Mori dis-
tributes code for computing superpixels using normalized-cut algorithms at http:
//www.cs.sfu.ca/
~
mori/research/superpixels/. Yuri Boykov distributes code
for min-cut problems at http://vision.csd.uwo.ca/code/; this includes codes
for extremely large grids. Vladimir Kolmogorov distributes a min-cut code at
http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software.html.
9.5.1 Evaluating Segmenters
Quantitative evaluation of segmenters is a somewhat vexed issue, because diﬀerent
methods have diﬀerent goals. One reasonable goal is predicting object boundaries
that people mark on images. This view yields a quantitative evaluation in terms of
recall and precision for boundary points that people have marked on a test set. A
natural comparison involves precision P (the percentage of the marked boundary
points that are real ones, i.e., were marked by people) and recall R (the percentage
of real boundary points that were marked); the F measure summarizes precision
and recall into a single number, F = 2PR/(P + R). In this framework, human
performance can be evaluated by holding out a test person, comparing the test
person’s markup to all the rest, and then averaging performance statistics over
Section 9.6 Notes 287
held out test people. Modern segmenters can do quite well at this test, but not as
well as people do (Figure 9.25).
The Berkeley Segmentation Data Set consists of 300 manually segmented im-
ages, and is distributed at http://www.eecs.berkeley.edu/Research/Projects/
CS/vision/bsds/. This page also maintains up-to-date benchmarks on that dataset.
A more recent version (BSDS-500) has 500 manually segmented images; see http://
www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.
html. Again, there is a set of benchmarks on that dataset available. The Lo-
tus Hill Institute provides a large dataset, free for academic use, at http://www.
imageparsing.com/. Annotations are much richer than just region structure, and
extend to a detailed semantic hierarchy of region relations.
9.6 NOTES
Segmentation is a diﬃcult topic, and there are a huge variety of methods. Surveys
of mainly historical interest are Riseman and Arbib (1977), Fu and Mui (1981),
Haralick and Shapiro (1985), Nevatia (1986), and Pal and Pal (1993).
One reason is that it is typically quite hard to assess the performance of a
segmenter at a level more useful than that of showing some examples. The original
clustering segmenter is Ohlander et al. (1978). Clustering methods tend to be rather
arbitrary—remember, this doesn’t mean they’re not useful—because there really
isn’t much theory available to predict what should be clustered and how. It is clear
that what we should be doing is forming clusters that are helpful to a particular
application, but this criterion hasn’t been formalized in any useful way. In this
chapter, we have attempted to give the big picture while ignoring detail, because a
detailed record of what has been done would be unenlightening. Everyone should
know about agglomerative clustering, divisive clustering, k-means, mean shift, and
at least one graph-based clustering algorithm (your choice!), because these ideas
are just so useful for so many applications; segmentation is just one application of
clustering.
There is a large literature on the role of grouping in human visual perception.
Standard Gestalt handbooks include Kanizsa (1979), and Koﬀka (1935). Subjec-
tive contours were ﬁrst described by Kanisza; there is a broad summary discus-
sion in Kanizsa (1976). The authoritative book by Palmer (1999) gives a much
broader picture than we can supply here. There is a great deal of information
about the development of diﬀerent theories of vision and the origins of Gestalt
thinking in Gordon (1997). Some groups appear to be formed remarkably early in
the visual process, a phenomenon known as pop out (Triesman 1982).
We believe the watershed is originally due to Digabel and Lantu´ejoul (1978);
see Vincent and Soille (1991). Fukunaga and Hostetler (1975) ﬁrst described mean
shift, but it was largely ignored until the work of Cheng (1995). It is now a main-
stay of computer vision research; as we shall see in the following chapters, it has
numerous applications.
A variety of graph theoretical clustering methods have been used in vision
(see Sarkar and Boyer (1998), and Wu and Leahy (1993); there is a summary
in Weiss (1999)).
Interactive segmentation became possible because of extremely fast min-cut
Section 9.6 Notes 288
algorithms that solve the relevant two-label Markov random ﬁeld (see Vogler et al.
(2000); Boykov and Jolly (2001); or Boykov and Funka Lea (2006)). There are now
many important variants. Grabcut is due to Rother et al. (2004); Objcut uses prior
information about object shapes to improve the cut (Kumar et al. 2010); and see
also Duchenne et al. (2008). There are numerous matting methods, which Wang
and Cohen (2007) survey in detail.
The normalized cuts formalism is due to Shi and Malik (1997) and (2000).
Variants include applications to motion segmentation Shi and Malik (1998a) and
methods for deducing similarity metrics from outputs Shi and Malik (1998b). There
are numerous alternate criteria (e.g., Cox et al. (1996), Perona and Freeman (1998)).
There is a considerable early literature on the evaluation of segmentation.
Useful references include: Zhang (1996a); Zhang (1997); Beauchemin and Thom-
son (1997); Zhang and Gerbrands (1994); Correia and Pereira (2003); Lei and
Udupa (2003); Warﬁeld et al. (2004); Paglieroni (2004); Cardoso and Corte Real
(2005); Cardoso and Corte Real (2006); Cardoso et al. (2009); Carleer et al. (2005);
and Crum et al. (2006). Evaluation is easier in the context of a speciﬁc task; pa-
pers dealing with assorted tasks include Yasnoﬀ et al. (1977), Hartley et al. (1982),
Zhang (1996b), and Ranade and Prewitt (1980). Martin et al. (2001) introduced
the Berkeley segmentation dataset, which is now a standard for evaluation, but
there are a variety of criteria one can use. Unnikrishnan et al. (2007) use the Rand
index; Polak et al. (2009) use multiple object boundaries; Polak et al. (2009) give a
detailed evaluation of four segmentation algorithms; Hanbury and Stottinger (2008)
compare metrics; and Zhang et al. (2008) give a recent survey of evaluation meth-
ods. Good image segments are most likely internally coherent, but making that
idea useful is hard (Bagon et al. 2008).
Since it is hard to get a segmentation right, Russell et al. (2006) suggest
working with multiple segmentations and then choosing good pieces. This idea is
now very inﬂuential. Multiple segmentations have been used to improve estimates
of support (Malisiewicz and Efros 2007), and to drive recognition (Pantofaru et
al. 2008) or (Malisiewicz and Efros 2008). One could organize the multiple segments
into an inclusion hierarchy (Tacc and Ahuja 1997); the hierarchies yield object
models (Todorovic and Ahuja 2008b), and can be matched (Todorovic and Ahuja
2008a).
We haven’t discussed some aspects of perceptual organization in great detail
mainly because our emphasis is on exposition rather than historical accuracy, and
these methods follow from the uniﬁed view. For example, there is a long thread of
literature on clustering image edge points or line segments into conﬁgurations that
are unlikely to have arisen by accident. We cover some of these ideas in the following
chapter, but also draw the readers attention to Amir and Lindenbaum (1996),
Huttenlocher and Wayner (1992), Lowe (1985), Mohan and Nevatia (1992), Sarkar
and Boyer (1993), and to Sarkar and Boyer (1994). In building user interfaces, it can
(as we hinted before) be helpful to know what is perceptually salient (e.g., Saund
and Moran (1995)).
Section 9.6 Notes 289
PROBLEMS
9.1. The mean shift procedure for ﬁnding a mode of a function
f(x) = C
n

i=1
_
(2π)
(−d
s
/2)
h
d
s
s
k
_
x
s
−x
s
i
h
s
___
(2π)
(−d
r
/2)
h
d
r
r
k
_
x
r
−x
r
i
h
r
__
.
What is the form of the mean shift estimate for this function?
PROGRAMMING EXERCISES
9.2. Implement a mean shift segmenter.
9.3. Implement a graph-based segmenter, after Section 9.4.2.
9.4. Implement a graph-based segmenter, after Section 9.4.3; for this, you should
use one of the fast graph cut packages available on the Web.
9.5. Use your graph-based segmenter to build an interactive segmentation system.
C H A P T E R 10
Grouping and Model Fitting
In the previous chapter, we collected together pixels that “looked like” one another,
using various clustering methods and various ways of measuring similarity. This
view could be applied to tokens (such as points, edge points, superpixels). It is an
intrinsically local view.
An alternative, which emphasizes a more global view, is to collect together
pixels, tokens, or whatever because they conform to some model. This approach
appears rather similar to the clustering approach in intent, but the mechanisms
and outcomes tend to be quite diﬀerent. In the clustering approach, the results we
produce can have local structure, but will not necessarily have a global structure.
For example, if we try to assemble tokens into a line agglomeratively by testing
whether the token to be added is close to the line formed by the previous two
tokens, we might get a shape that is quite curved. We really need to check whether
all tokens “agree” on the parameters of the line; local consistency is not enough.
These problems are rather diﬃcult, and strategies to attack them for one kind
of model tend to extend rather well to strategies for other kinds of model. In this
chapter, we mainly concentrate on one core problem, which is simple enough to
do in detail. We seek to ﬁnd all the lines represented by a set of tokens. This
problem is usually referred to as ﬁtting, or sometimes as grouping. There are three
important sub-problems here: If all the tokens agree on a model, what is the model?
Which tokens contribute to a particular model, and which do not? And how many
instances of the model are there?
10.1 THE HOUGH TRANSFORM
Assume we wish to ﬁt a structure to a set of tokens (say, a line to a set of points).
One way to cluster tokens that could lie on the same structure is to record all the
structures on which each token lies and then look for structures that get many votes.
This (quite general) technique is known as the Hough transform. To ﬁt a structure
with a Hough transform, we take each image token and determine all structures
that could pass through that token. We make a record of this set—you should think
of this as voting—and repeat the process for each token. We decide on what is
present by looking at the votes. For example, if we are grouping points that lie on
lines, we take each point and vote for all lines that could go through it; we now
do this for each point. The line (or lines) that are present should make themselves
obvious because they pass through many points and so have many votes.
10.1.1 Fitting Lines with the Hough Transform
A line is easily parametrized as a collection of points (x, y) such that
xcos θ +y sinθ +r = 0.
290
Section 10.1 The Hough Transform 291
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
FIGURE 10.1: The Hough transform maps each point like token to a curve of possible
lines (or other parametric curves) through that point. These ﬁgures illustrate the Hough
transform for lines. The left-hand column shows points, and the right-hand column
shows the corresponding accumulator arrays (the number of votes is indicated by the
gray level, with a large number of votes being indicated by bright points). The top row
shows what happens using a set of 20 points drawn from a line. On the top right, the
accumulator array for the Hough transform of these points. Corresponding to each point is
a curve of votes in the accumulator array; the largest set of votes is 20 (which corresponds
to the brightest point). The horizontal variable in the accumulator array is θ, and the
vertical variable is r; there are 200 steps in each direction, and r lies in the range [0, 1.55].
On the bottom, these points have been oﬀset by a random vector, each element of which
is uniform in the range [0, 0.05]. Note that this oﬀsets the curves in the accumulator
array shown next to the points and the maximum vote is now 6 (which corresponds to the
brightest value in this image; this value would be diﬃcult to see on the same scale as the
top image).
Now any pair of (θ, r) represents a unique line, where r ≥ 0 is the perpendicular
distance from the line to the origin and 0 ≤ θ < 2π. We call the set of pairs (θ, r)
line space; the space can be visualized as a half-inﬁnite cylinder. There is a family
Section 10.1 The Hough Transform 292
of lines that passes through any point token. In particular, the lines that lie on
the curve in line space given by r = −x
0
cos θ +y
0
sin θ all pass through the point
token at (x
0
, y
0
).
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
FIGURE 10.2: The Hough transform for a set of random points can lead to quite large sets
of votes in the accumulator array. As in Figure 10.1, the left-hand column shows points,
and the right-hand column shows the corresponding accumulator arrays (the number
of votes is indicated by the gray level, with a large number of votes being indicated by
bright points). In this case, the data points are noise points (both coordinates are uniform
random numbers in the range [0, 1]); the accumulator array in this case contains many
points of overlap, and the maximum vote is now 4 (compared with 6 in Figure 10.1).
Because the image has a known size, there is some R such that we are not
interested in lines for r > R. These lines are too far away, from the origin for us
to see them. This means that the lines we are interested in form a bounded subset
of the plane, and we discretize this with some convenient grid. The grid elements
can be thought of as buckets into which we place votes. This grid of buckets is
referred to as the accumulator array. For each point token, we add a vote to the
total formed for every grid element on the curve corresponding to the point token.
If there are many point tokens that are collinear, we expect there to be many votes
in the grid element corresponding to that line.
10.1.2 Using the Hough Transform
The Hough transform is an extremely general procedure. One could use the proce-
dure described to ﬁt, say, circles to points in the plane, or spheres or even ellipsoids
to three-dimensional data. This works in principle, but in practice, the Hough
transform as described is diﬃcult to use, even for ﬁnding lines. There are several
sources of diﬃculty.
• Grid dimension: The accumulator array for lines has dimension two, but
for circles in the plane, it has dimension three (center location and radius); for
axis-aligned ellipses in the plane it has dimension four; for general ellipses in
Section 10.2 Fitting Lines and Planes 293
the plane, ﬁve; for spheres in 3D, four; for axis-aligned ellipsoids in 3D, seven;
and for general ellipsoids in 3D, 10. Even quite simple structures could result
in high-dimensional accumulator arrays, which take unmanageable amounts
of storage.
• Quantization errors: An appropriate grid size is diﬃcult to pick. Too
coarse a grid can lead to large values of the vote being obtained falsely because
many quite diﬀerent structures correspond to a single bucket. Too ﬁne a grid
can lead to structures not being found because votes resulting from tokens
that are not exactly aligned end up in diﬀerent buckets, and no bucket has a
large vote (Figure 10.1).
• Noise: The attraction of the Hough transform is that it connects widely
separated tokens that lie close to some structure. This is also a weakness
because it is usually possible to ﬁnd many quite good phantom structures
in a large set of reasonably uniformly distributed tokens (Figure 10.2). This
means that regions of texture can generate peaks in the voting array that are
larger than those associated with the lines sought.
These diﬃculties can be avoided, to some extent, by recognizing the Hough
transform as an attempt to ﬁnd a mode in a distribution. The distribution is
represented by the voting array, and some of the problems are created by the cells
in that array. But to ﬁnd a mode, we do not necessarily need to use an accumulator
array; instead, we could apply mean shift. The algorithm of Section 9.3.4 can be
applied directly. It can also be useful to ensure the minimum of irrelevant tokens
by, for example, tuning the edge detector to smooth out texture, using lighting that
ensures high-contrast edges, or using tokens with a more complex structure with
edge points.
One natural application of the Hough transform is in object recognition. We
defer the details to Section 18.4.2, but the general picture is as follows. Imagine
an object is made out of known parts. These parts might move around a bit with
respect to one another, but are substantial enough to be detected on their own.
We can then expect each detected part to have an opinion about the location
(and, perhaps, the state) of the object. This means we could detect objects by
ﬁrst detecting parts, then allowing each detected part to vote for location (and
maybe state) of object instances, and ﬁnally using the Hough transform, most
likely in mean shift form, to ﬁnd instances on which many part detectors agree. This
approach has been successfully applied in numerous forms (Maji and Malik 2009).
10.2 FITTING LINES AND PLANES
In many applications, objects are characterized by the presence of straight lines. For
example, we might wish to build models of buildings using pictures of the buildings
(as in the application in Chapter 19). This application uses polyhedral models of
buildings, meaning that straight lines in the image are important. Similarly, many
industrial parts have straight edges of one form or another; if we wish to recognize
industrial parts in an image, straight lines could be helpful. In either case, a report
of all straight lines in the image is an extremely useful segmentation. All this means
that ﬁtting a line to a set of plane tokens is extremely useful. Fitting planes to
Section 10.2 Fitting Lines and Planes 294
tokens in 3D is also useful, and our methods for ﬁtting lines in the plane apply with
little change.
10.2.1 Fitting a Single Line
We ﬁrst assume that all the points that belong to a particular line are known, and
the parameters of the line must be found. We adopt the notation
u =

u
i
k
to simplify the presentation.
There is a simple strategy for ﬁtting lines, known as least squares. This
procedure has a long tradition (which is the only reason we describe it!), but has
a substantial bias. Most readers will have seen this idea, but many will not be
familiar with the problems that it leads to. For this approach, we represent a line
as y = ax + b. At each data point, we have (x
i
, y
i
); we decide to choose the line
that best predicts the measured y coordinate for each measured x coordinate. This
means we want to choose the line that minimizes

i
(y
i
−ax
i
−b)
2
.
By diﬀerentiation, the line is given by the solution to the problem
_
y
2
y
_
=
_
x
2
x
x 1
__
a
b
_
.
Although this is a standard linear solution to a classical problem, it’s not much help
in vision applications, because the model is an extremely poor one. The diﬃculty
is that the measurement error is dependent on coordinate frame—we are counting
vertical oﬀsets from the line as errors, which means that near vertical lines lead
to quite large values of the error and quite funny ﬁts (Figure 10.3). In fact, the
process is so dependent on coordinate frame that it doesn’t represent vertical lines
at all.
We could work with the actual distance between the point and the line (rather
than the vertical distance). This leads to a problem known as total least squares.
We can represent a line as the collection of points where ax + by + c = 0. Every
line can be represented in this way, and we can think of a line as a triple of values
(a, b, c). Notice that for λ ,= 0, the line given by λ(a, b, c) is the same as the line
represented by (a, b, c). In the exercises, you are asked to prove the simple, but
extremely useful, result that the perpendicular distance from a point (u, v) to a
line (a, b, c) is given by
abs(au +bv +c) if a
2
+b
2
= 1.
In our experience, this fact is useful enough to be worth memorizing. To minimize
the sum of perpendicular distances between points and lines, we need to minimize

i
(ax
i
+by
i
+c)
2
,
Section 10.2 Fitting Lines and Planes 295
FIGURE 10.3: Left: Least squares ﬁnds the line that minimizes the sum of squared vertical
distances between the line and the tokens (because it assumes that the error appears only
in the y coordinate). This yields a (very slightly) simpler mathematical problem at the
cost of a poor ﬁt. Right: Total least-squares ﬁnds the line that minimizes the sum of
squared perpendicular distances between tokens and the line; this means that, for example,
we can ﬁt near-vertical lines without diﬃculty.
subject to a
2
+b
2
= 1. Now using a Lagrange multiplier λ, we have a solution if
⎛
⎝
x
2
xy x
xy y
2
y
x y 1
⎞
⎠
⎛
⎝
a
b
c
⎞
⎠
= λ
⎛
⎝
2a
2b
0
⎞
⎠
.
This means that
c = −ax −by,
and we can substitute this back to get the eigenvalue problem
_
x
2
−x x xy −x y
xy −x y y
2
−y y
__
a
b
_
= μ
_
a
b
_
.
Because this is a 2D eigenvalue problem, two solutions up to scale can be obtained
in closed form (for those who care, it’s usually done numerically!). The scale is
obtained from the constraint that a
2
+ b
2
= 1. The two solutions to this problem
are lines at right angles; one maximizes the sum of squared distances and the other
minimizes it.
10.2.2 Fitting Planes
Fitting planes is very similar to ﬁtting lines. We could represent a plane as z =
ux + vy + w, then apply least squares. This will be biased, just like least squares
line ﬁtting, because it will not represent vertical planes well. Total least squares
is a better strategy, just as in line ﬁtting. We represent the plane as ax + by +
cz + d = 0; then the distance from a point x
i
= (x
i
, y
i
, z
i
) to the plane will be
(ax
i
+ by
i
+ cz
i
+ d)
2
if a
2
+ b
2
+ c
2
= 1, and we can now use the analysis above
with small changes.
Section 10.2 Fitting Lines and Planes 296
10.2.3 Fitting Multiple Lines
Now assume we have a set of tokens (say, points), and we want to ﬁt several lines to
this set of tokens. This problem can be diﬃcult because it can involve searching over
a large combinatorial space. One approach is to notice that we seldom encounter
isolated points; instead, in many problems, we are ﬁtting lines to edge points. We
can use the orientation of an edge point as a hint to the position of the next point
on the line. If we are stuck with isolated points, then k-means can be applied.
Incremental line ﬁtting algorithms take connected curves of edge points
and ﬁt lines to runs of points along the curve. Connected curves of edge points
are fairly easily obtained from an edge detector whose output gives orientation (see
exercises). An incremental ﬁtter then starts at one end of a curve of edge points and
walks along the curve, cutting oﬀ runs of pixels that ﬁt a line well (the structure of
the algorithm is shown in Algorithm 10.1). Incremental line ﬁtting can work well,
despite the lack of an underlying statistical model. One feature is that it reports
groups of lines that form closed curves. This is attractive when the lines of interest
can reasonably be expected to form a closed curve (e.g., in some object recognition
applications) because it means that the algorithm reports natural groups without
further fuss. When one uses this strategy, occlusion of an edge can lead to more
than one line segment being ﬁtted to the boundary. This diﬃculty can be addressed
by postprocessing the lines to ﬁnd pairs that (roughly) coincide, but the process is
somewhat unattractive because it is hard to give a sensible criterion by which to
decide when two lines do coincide.
Put all points on curve list, in order along the curve
Empty the line point list
Empty the line list
Until there are too few points on the curve
Transfer ﬁrst few points on the curve to the line point list
Fit line to line point list
While ﬁtted line is good enough
Transfer the next point on the curve
to the line point list and reﬁt the line
end
Transfer last point(s) back to curve
Reﬁt line
Attach line to line list
end
Algorithm 10.1: Incremental Line Fitting.
Now assume that points carry no hints about which line they lie on (i.e., there
is no color information or anything like it to help, and, crucially, the points are not
linked). Furthermore, assume that we know how many lines there are. We can
attempt to determine which point lies on which line using a modiﬁed version of
k-means. In this case, the model is that there are k lines, each of which generates
Section 10.3 Fitting Curved Structures 297
some subset of the data points. The best solution for lines and data points is
obtained by minimizing

l
i
∈lines

x
j
∈data due to ith line
dist(l
i
, x
j
)
2
over both correspondences and lines. Again, there are too many correspondences
to search this space.
It is easy to modify k-means to deal with this problem. The two phases are
as follows:
• Allocate each point to the closest line.
• Fit the best line to the points allocated to each line.
This results in Algorithm 10.2. Convergence can be tested by looking at the size
of the change in the lines, at whether labels have been ﬂipped (probably the best
test), or at the sum of perpendicular distances of points from their lines.
Hypothesize k lines (perhaps uniformly at random)
or
Hypothesize an assignment of lines to points
and then ﬁt lines using this assignment
Until convergence
Allocate each point to the closest line
Reﬁt lines
end
Algorithm 10.2: K-means Line Fitting.
10.3 FITTING CURVED STRUCTURES
Curves in 2D are diﬀerent from lines in 2D. For every token on the plane, there is
a unique, single point on a line that is closest to it. This is not true for a curve.
Because curves curve, there might be more than one point on the curve that looks
locally as though it is closest to the token (Figure 10.4). This means it can be
very hard to ﬁnd the smallest distance between a point and a curve. Similar eﬀects
occur for surfaces in 3D. If one ignores this diﬃculty, ﬁtting curves is similar to
ﬁtting lines. We minimize the sum of squared distances between the points and the
curve as a function of the choice of curve.
Assume that the curve is implicit, and so has the form φ(x, y) = 0. The vector
from the closest point on the implicit curve to the data point is normal to the curve,
so the closest point is given by ﬁnding all the (u, v) with the following properties:
1. (u, v) is a point on the curve, which means that φ(u, v) = 0.
2. s = (d
x
, d
y
) −(u, v) is normal to the curve.
Section 10.3 Fitting Curved Structures 298
FIGURE 10.4: There can be more than one point on a curve that looks locally as if it is
closest to a token. This makes ﬁtting curves to points very diﬃcult. On the left, a curve
and a token; dashed lines connect the token to the two points on the curve that, by a
local test, could be closest. The local test checks that the dashed line and the tangent to
the curve are at right angles. Center and right, we show copies of part of the curve; for
each, the closest point on the segment to the token is diﬀerent, because part of the curve
is missing. As a result, we cannot perform a local test that guarantees that a point is
closest. We must check all candidates.
Given all such s, the length of the shortest is the distance from the data point to
the curve.
The second criterion requires a little work to determine the normal. The
normal to an implicit curve is the direction in which we leave the curve fastest;
along this direction, the value of φ must change fastest, too. This means that the
normal at a point (u, v) is
_
∂φ
∂x
,
∂φ
∂y
_
,
evaluated at (u, v). If the tangent to the curve is T, then we must have T.s = 0.
Because we are working in 2D, we can determine the tangent from the normal, so
that we must have
ψ(u, v; d
x
, d
y
) =
∂φ
∂y
(u, v) ¦d
x
−u¦ −
∂φ
∂x
(u, v) ¦d
y
−v¦ = 0
at the point (u, v). We now have two equations in two unknowns, and, in principle
can solve them. However, this is very seldom as easy as it looks, because there
might be many solutions. We expect d
2
in the case that φ is a polynomial of degree
d, though some of them might be complex.
The situation is not improved by using a parametric curve. The coordinates of
a point on a parametric curve are functions of a parameter, so if t is the parameter,
the curve could be written as (x(t), y(t)). Assume we have a data point (d
x
, d
y
).
The closest point on a parametric curve can be identiﬁed by its parameter value,
which we shall write as τ. This point could lie at one or the other end of the
curve. Otherwise, the vector from our data point to the closest point is normal to
the curve. This means that s(τ) = (d
x
, d
y
) − (x(τ), y(τ)) is normal to the tangent
vector, so that s(τ).T = 0. The tangent vector is
_
dx
dt
(τ),
dy
dt
(τ)
_
,
Section 10.4 Robustness 299
which means that τ must satisfy the equation
dx
dt
(τ) ¦d
x
−x(τ)¦ +
dy
dt
(τ) ¦d
y
−y(τ)¦ = 0.
Now this is only one equation, rather than two, but the situation is not much better
than that for parametric curves. It is almost always the case that x(t) and y(t) are
polynomials because it is usually easier to do root ﬁnding for polynomials. At worst,
x(t) and y(t) are ratios of polynomials because we can rearrange the left-hand side
of our equation to come up with a polynomial in this case, too. However, we are still
faced with a possibly large number of roots. The underlying problem is geometric:
there may be many points on a curve that, locally, appear to be the closest point
to a given data point. This is because the curve is not ﬂat (Figure 10.4). There
is no way to tell which is closest without checking each in turn. In some cases
(for example, circles), one can duck around this problem. This diﬃculty applies to
ﬁtting curved surfaces to points in 3D as well.
There are two strategies for dealing with this quite general problem. One
is to substitute some approximation for the distance between a point and a curve
(or, in 3D, a point and a surface), which is sometimes eﬀective. The other is to
modify the representation of the curve or of the surface. For example, one might
represent a curve with a set of samples, or with a set of line segments. Similarly, a
surface might be represented with a set of samples, or a mesh. We could then use
the methods of Chapter 12 to register these representations to the data, or even to
deform them to ﬁt the data.
10.4 ROBUSTNESS
All of the line ﬁtting methods described involve squared error terms. This can
lead to poor ﬁts in practice because a single wildly inappropriate data point might
give errors that dominate those due to many good data points; these errors could
result in a substantial bias in the ﬁtting process (Figure 10.5). This eﬀect results
from the squaring of the error. It is diﬃcult to avoid such data points—usually
called outliers—in practice. Errors in collecting or transcribing data points is
one important source of outliers. Another common source is a problem with the
model. Perhaps some rare but important eﬀect has been ignored or the magnitude
of an eﬀect has been badly underestimated. Finally, errors in correspondence are
particularly prone to generating outliers. Practical vision problems usually involve
outliers.
This problem can be resolved either by reducing the inﬂuence of distant points
on the estimate (Section 10.4.1), or by identifying outlying points and ignoring
them. There are two methods to identify outlying points. We could search for good
points. A small set of good points will identify the thing we wish to ﬁt; other good
points will agree, and the points that disagree are bad points. This is the basis of an
extremely important approach, described in Section 10.4.2. Alternatively, we could
regard this as a problem with missing data, and use the EM algorithm described
in Section 10.5.
Section 10.4 Robustness 300
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
FIGURE 10.5: Line ﬁtting with a squared error is extremely sensitive to outliers, both in
x and y coordinates. We show an example using least squares. At the top left, a good
least-squares ﬁt of a line to a set of points. Top right shows the same set of points,
but with the x coordinate of one point corrupted; this means that the point has been
translated horizontally from where it should be. As a result, it contributes an enormous
error term to the true line, and a better least-squares ﬁt is obtained by making a signiﬁcant
change in the line’s orientation. Although this makes the errors at most points larger, it
reduces the very large error at the outlier. Bottom left shows the same set of points, but
with the y coordinate of one point corrupted. In this particular case, the x intercept has
changed. These three ﬁgures are on the same set of axes for comparison, but this choice
of axes does not clearly show how bad the ﬁt is for the third case. Bottom right shows
a detail of this case, in which the line is clearly a bad ﬁt.
10.4.1 M-Estimators
An M-estimator estimates parameters by replacing the squared error term with a
term that is better behaved. This means we minimize an expression of the form

i
ρ(r
i
(x
i
, θ); σ),
where θ are the parameters of the model being ﬁtted (for example, in the case of
the line, we might have the orientation and the y intercept), and r
i
(x
i
, θ) is the
residual error of the model on the ith data point. Using this notation, our least
Section 10.4 Robustness 301
-10 -8 -6 -4 -2 0 2 4 6 8 10
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
1
10
0.1
y=x
2
FIGURE 10.6: The function ρ(x; σ) = x
2
/(σ
2
+x
2
), plotted for σ
2
= 0.1, 1, and 10, with a
plot of y = x
2
for comparison. Replacing quadratic terms with ρ reduces the inﬂuence of
outliers on a ﬁt. A point that is several multiples of σ away from the ﬁtted curve is going
to have almost no eﬀect on the coeﬃcients of the ﬁtted curve, because the value of ρ will
be close to 1 and will change extremely slowly with the distance from the ﬁtted curve.
squares and total least squares line-ﬁtting errors—which diﬀer only in the form of
the residual error—both have ρ(u; σ) = u
2
. The trick to M-estimators is to make
ρ(u; σ) look like u
2
for part of its range and then ﬂattens out; we expect that ρ(u; σ)
increases monotonically, and is close to a constant value for large u. A common
choice is
ρ(u; σ) =
u
2
σ
2
+u
2
.
The parameter σ controls the point at which the function ﬂattens out, and we have
plotted a variety of examples in Figure 10.6. There are many other M-estimators
available. Typically, they are discussed in terms of their inﬂuence function, which
is deﬁned as
∂ρ
∂θ
.
This is natural because our minimization criterion yields

i
ρ(r
i
(x
i
, θ); σ)
∂ρ
∂θ
= 0
at the solution. For the kind of problems we consider, we would expect a good
inﬂuence function to be antisymmetric— there is no diﬀerence between a slight
overprediction and a slight underprediction—and to tail oﬀ with large values—
because we want to limit the inﬂuence of the outliers.
There are two tricky issues with using M-estimators. First, the minimization
problem is non-linear and must be solved iteratively. The standard diﬃculties
apply: there might be more than one local minimum, the method might diverge,
and the behavior of the method is likely to be quite dependent on the start point.
Section 10.4 Robustness 302
For s = 1 to s = k
Draw a subset of r distinct points, chosen uniformly at random
Fit to this set of points using least squares to obtain an initial
set of parameters θ
0
s
Estimate σ
0
s
using θ
0
s
Until convergence (usually [θ
n
s
−θ
n−1
s
[ is small):
Take a minimizing step using θ
n−1
s
, σ
n−1
s
to get θ
n
s
Now compute σ
n
s
end
end
Report the best ﬁt of this set of k trials, using the median of the residuals
as a criterion
Algorithm 10.3: Using an M-Estimator to Fit a Least Squares Model.
A common strategy for dealing with this problem is to draw a subsample of the
dataset, ﬁt to that subsample using least squares, and use this as a start point for
the ﬁtting process. We do this for a large number of diﬀerent subsamples, enough
to ensure that there is a high probability that there is at least one subsample that
consists entirely of good data points (Algorithm 10.3).
Second, as Figures 10.7 and 10.8 indicate, the estimators require a sensible
estimate of σ, which is often referred to as scale. Typically, the scale estimate is
supplied at each iteration of the solution method; a popular estimate of scale is
σ
(n)
= 1.4826 median
i
[r
(n)
i
(x
i
; θ
(n−1)
)[ .
We summarize a general M-estimator in Algorithm 10.3.
10.4.2 RANSAC: Searching for Good Points
An alternative to modifying the cost function is to search the collection of data
points for good points. This is quite easily done by an iterative process: First, we
choose a small subset of points and ﬁt to that subset, and then we see how many
other points ﬁt to the resulting object. We continue this process until we have a
high probability of ﬁnding the structure we are looking for.
For example, assume that we are ﬁtting a line to a dataset that consists of
about 50% outliers. We can ﬁt a line to only two points. If we draw pairs of points
uniformly and at random, then about a quarter of these pairs will consist entirely
of good data points. We can identify these good pairs by noticing that a large
collection of other points lie close to the line ﬁtted to such a pair. Of course, a
better estimate of the line could then be obtained by ﬁtting a line to the points
that lie close to our current line.
Fischler and Bolles (1981) formalized this approach into an algorithm—search
for a random sample that leads to a ﬁt on which many of the data points agree.
The algorithm is usually called RANSAC, for RANdom SAmple Consensus, and is
Section 10.4 Robustness 303
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
FIGURE 10.7: The top row shows lines ﬁtted to the second dataset of Figure 10.5 using a
weighting function that deemphasizes the contribution of distant points (the function φ of
Figure 10.6). On the left, μ has about the right value; the contribution of the outlier has
been down-weighted, and the ﬁt is good. In the center, the value of μ is too small so that
the ﬁt is insensitive to the position of all the data points, meaning that its relationship to
the data is obscure. On the right, the value of μ is too large, meaning that the outlier
makes about the same contribution as it does in least squares. The bottom row shows
closeups of the ﬁtted line and the non-outlying data points for the same cases.
displayed in Algorithm 10.4. To make this algorithm practical, we need to choose
three parameters.
The Number of Samples Required
Our samples consist of sets of points drawn uniformly and at random from
the dataset. Each sample contains the minimum number of points required to ﬁt
the abstraction of interest. For example, if we wish to ﬁt lines, we draw pairs of
points; if we wish to ﬁt circles, we draw triples of points, and so on. We assume
that we need to draw n data points, and that w is the fraction of these points that
are good (we need only a reasonable estimate of this number). Now the expected
value of the number of draws k required to get one point is given by
E[k] = 1P(one good sample in one draw) +
2P(one good sample in two draws) +. . .
= w
n
+ 2(1 −w
n
)w
n
+ 3(1 −w
n
)
2
w
n
+. . .
= w
−n
(where the last step takes a little manipulation of algebraic series). We would like
to be fairly conﬁdent that we have seen a good sample, so we wish to draw more
than w
−n
samples; a natural thing to do is to add a few standard deviations to this
Section 10.4 Robustness 304
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-14 -12 -10 -8 -6 -4 -2 0 2 4 6
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
FIGURE 10.8: The top row shows lines ﬁtted to the third dataset of Figure 10.5 using a
weighting function that deemphasizes the contribution of distant points (the function φ of
Figure 10.6). On the left, μ has about the right value; the contribution of the outlier has
been down-weighted, and the ﬁt is good. In the center, the value of μ is too small, so that
the ﬁt is insensitive to the position of all the data points, meaning that its relationship to
the data is obscure. On the right, the value of μ is too large, meaning that the outlier
makes about the same contribution as it does in least squares. The bottom row shows
close ups of the ﬁtted line and the non-outlying data points, for the same cases.
number. The standard deviation of k can be obtained as
SD(k) =
√
1 −w
n
w
n
.
An alternative approach to this problem is to look at a number of samples that
guarantees a low probability z of seeing only bad samples. In this case, we have
(1 −w
n
)
k
= z,
which means that
k =
log(z)
log(1 −w
n
)
.
It is common to have to deal with data where w is unknown. However, each ﬁtting
attempt contains information about w. In particular, if n data points are required,
then we can assume that the probability of a successful ﬁt is w
n
. If we observe
a long sequence of ﬁtting attempts, we can estimate w from this sequence. This
suggests that we start with a relatively low estimate of w, generate a sequence
of attempted ﬁts, and then improve our estimate of w. If we have more ﬁtting
attempts than the new estimate of w predicts, the process can stop. The problem
of updating the estimate of w reduces to estimating the probability that a coin
comes up heads or tails given a sequence of ﬁts.
Section 10.4 Robustness 305
Determine:
n—the smallest number of points required (e.g., for lines, n = 2,
for circles, n = 3)
k—the number of iterations required
t—the threshold used to identify a point that ﬁts well
d—the number of nearby points required
to assert a model ﬁts well
Until k iterations have occurred
Draw a sample of n points from the data
uniformly and at random
Fit to that set of n points
For each data point outside the sample
Test the distance from the point to the structure
against t; if the distance from the point to the structure
is less than t, the point is close
end
If there are d or more points close to the structure
then there is a good ﬁt. Reﬁt the structure using all
these points. Add the result to a collection of good ﬁts.
end
Use the best ﬁt from this collection, using the
ﬁtting error as a criterion
Algorithm 10.4: RANSAC: Fitting Structures Using Random Sample Consensus.
Telling Whether a Point Is Close
We need to determine whether a point lies close to a line ﬁtted to a sample.
We do this by determining the distance between the point and the ﬁtted line, and
testing that distance against a threshold d; if the distance is below the threshold,
the point lies close. In general, specifying this parameter is part of the modeling
process. Obtaining a value for this parameter is relatively simple. We generally need
only an order of magnitude estimate, and the same value applies to many diﬀerent
experiments. The parameter is often determined by trying a few values and seeing
what happens; another approach is to look at a few characteristic datasets, ﬁtting
a line by eye, and estimating the average size of the deviations.
The Number of Points That Must Agree
Assume that we have ﬁtted a line to some random sample of two data points,
and we need to know whether that line is good. We do this by counting the number
of points that lie within some distance of the line (the distance was determined in
the previous section). In particular, assume that we know the probability that an
outlier lies in this collection of points; write this probability as y. We would like to
choose some number of points t such that the probability that all points near the
line are outliers, y
t
, is small (say, less than 0.05). Notice that y ≤ (1 −w) (because
some outliers should be far from the line), so we could choose t such that (1 −w)
t
Section 10.5 Fitting Using Probabilistic Models 306
is small.
10.5 FITTING USING PROBABILISTIC MODELS
It is straightforward to build probabilistic models from the ﬁtting procedures we
have described. Doing so yields a new kind of model, and a new algorithm; both
are extremely useful in practice. The key is to view our observed data as having
been produced by a generative model. The generative model speciﬁes how each
data point was produced.
In the simplest case, line ﬁtting with least squares, we can recover the same
equations we worked with in Section 10.2.1 by using a natural generative model.
Our model for producing data is that the x coordinate is uniformly distributed
and the y coordinate is generated by (a) ﬁnding the point ax
i
+b on the line corre-
sponding to the x coordinate and then (b) adding a zero mean normally distributed
random variable. Now write x ∼ p to mean that x is a sample from the probability
distribution p; write U(R) for the uniform distribution over some range of values
R; and write N(μ, σ
2
) for the normal distribution with mean mu and variance σ
2
.
With our notation, we can write:
x
i
∼ U(R)
y
i
∼ N(ax
i
+b, σ
2
).
We can estimate the unknown parameters of this model in a straightforward way.
The important parameters are a and b (though knowing σ might be useful). The
usual way to estimate parameters in a probabilistic model is to maximize the like-
lihood of the data, typically by working with the negative log-likelihood and mini-
mizing that. In this case, the log-likelihood of the data is
/(a, b, σ) =

i∈data
log P(x
i
, y
i
[a, b, σ)
=

i∈data
log P(y
i
[x
i
, a, b, σ) + log P(x
i
)
=

i∈data
−
(y
i
−(ax
i
+b))
2
2σ
2
−
1
2
log 2πσ
2
+K
b
where K
b
is a constant representing log P(x
i
). Now, to minimize the negative log-
likelihood as a function of a and b we could minimize

i∈data
(y
i
− (ax
i
+ b))
2
as a function of a and b (which is what we did for least-squares line ﬁtting in
Section 10.2.1).
Now consider total least-squares line ﬁtting. Again, we can recover the equa-
tions we worked with in Section 10.2.1 from a natural generative model. In this
case, to generate a data point (x
i
, y
i
), we generate a point (u
i
, v
i
) uniformly at ran-
dom along the line (or rather, along a ﬁnite length segment of the line likely to be of
interest to us), then sample a distance ξ
i
(where ξ
i
∼ N(0, σ
2
), and move the point
(u
i
, v
i
) perpendicular to the line by that distance. If the line is ax + by + c = 0
and if a
2
+ b
2
= 1, we have that (x
i
, y
i
) = (u
i
, v
i
) + ξ
i
(a, b). We can write the
Section 10.5 Fitting Using Probabilistic Models 307
log-likelihood of the data under this model as
/(a, b, c, σ) =

i∈data
log P(x
i
, y
i
[a, b, c, σ)
=

i∈data
log P(ξ
i
[σ) + log P(u
i
, v
i
[a, b, c).
But P(u
i
, v
i
[a, b, c) is some constant, because this point is distributed uniformly
along the line. Since ξ
i
is the perpendicular distance from (x
i
, y
i
) to the line (which
is [[ (ax
i
+by
i
+c)[[ as long as a
2
+b
2
= 1), we must maximize

i∈data
log P(ξ
i
[σ) =

i∈data
−
ξ
2
i
2σ
2
−
1
2
log 2πσ
2
=

i∈data
−
(ax
i
+by
i
+c)
2
2σ
2
−
1
2
log 2πσ
2
(again, subject to a
2
+b
2
= 1). For ﬁxed (but perhaps unknown) σ this yields the
problem we were working with in Section 10.2.1. So far, generative models have
just reproduced what we know already, but a powerful trick makes them much more
interesting.
10.5.1 Missing Data Problems
A number of important vision problems can be phrased as problems that happen to
be missing useful elements of the data. For example, we can think of segmentation
as the problem of determining from which of a number of sources a measurement
came. This is a general view. More speciﬁcally, ﬁtting a line to a set of tokens
involves segmenting the tokens into outliers and inliers, then ﬁtting the line to
the inliers; segmenting an image into regions involves determining which source of
color and texture pixels generated the image pixels; ﬁtting a set of lines to a set
of tokens involves determining which tokens lie on which line; and segmenting a
motion sequence into moving regions involves allocating moving pixels to motion
models. Each of these problems would be easy if we happened to possess some data
that is currently missing (respectively, whether a point is an inlier or an outlier,
which region a pixel comes from, which line a token comes from, and which motion
model a pixel comes from).
A missing data problem is a statistical problem where some data is missing.
There are two natural contexts in which missing data are important: In the ﬁrst,
some terms in a data vector are missing for some instances and present for others
(perhaps someone responding to a survey was embarrassed by a question). In the
second, which is far more common in our applications, an inference problem can be
made much simpler by rewriting it using some variables whose values are unknown.
Fortunately, there is an eﬀective algorithm for dealing with missing data problems;
in essence, we take an expectation over the missing data. We demonstrate this
method and appropriate algorithms with two examples.
Section 10.5 Fitting Using Probabilistic Models 308
Example: Outliers and Line Fitting
We wish to ﬁt a line to a set of tokens that are at x
i
= (x
i
, y
i
). Some tokens
might be outliers, but we do not know which ones are. This means we can model the
process of generating a token as ﬁrst, choosing whether it will come from the line
or be an outlier, and then, choosing the token conditioned on the original choice.
The ﬁrst choice will be random, and we can write P(token comes from line) = π.
We have already given two models of how a point could be generated from a line
model. We model outliers as occuring uniformly and at random on the plane. This
means that we can write the probability of generating a token as
P(x
i
[a, b, c, π) = P(x
i
, line[a, b, c, π) +P(x
i
, outlier[a, b, c, π)
= P(x
i
[line, a, b, c)P(line) +P(x
i
[outlier, a, b, c)P(outlier)
= P(x
i
[line, a, b, c)π +P(x
i
[outlier, a, b, c)(1 −π).
If we knew for every data item whether it came from the line or was an outlier,
then ﬁtting the line would be simple; we would ignore all the outliers, and apply the
methods of Section 10.2.1 to the other points. Similarly, if we knew the line, then
estimating which point is an outlier and which is not would be straightforward (the
outliers are far from the line). The diﬃculty is that we do not; the key to resolving
this diﬃculty is repeated re-estimation (Section 10.5.3), which provides a standard
algorithm for this class of problem. Figure 10.9 shows typical results using the
standard algorithm.
By a very small manipulation of the equations above (replace “line” with
“background” and “outlier” with “foreground”), we can represent a background
subtraction problem, too. We model the image in each frame of video as the
same, multiplied by some constant to take account of automatic gain control, but
with noise added. We model the noise as coming from some uniform source. Fig-
ures 10.10 and 10.11 show results, obtained with the standard algorithm for these
problems (Section 10.5.3).
Example: Image Segmentation
At each pixel in an image, we compute a d-dimensional feature vector x,
which might contain position, color, and texture information. We believe the image
contains g segments, and each pixel is produced by one of these segments. Thus, to
produce a pixel, we choose an image segment and then generate the pixel from the
model of that segment. We assume that the lth segment is chosen with probability
π
l
, and we model the density associated with the lth segment as a Gaussian, with
known covariance Σ and unknown mean θ
l
= (μ
l
) that depends on the particular
segment. We encapsulate these parameters into a parameter vector to get Θ =
(π
1
, . . . , π
g
, θ
1
, . . . , θ
g
). This means that we can write the probability of generating
a pixel vector x as
p(x[Θ) =

i
p(x[θ
l
)π
l
.
Fitting this model would be simple if we knew which segment produced which pixel,
because then we could estimate the mean of each segment separately. Similarly