Section LC Linear Combinations

In Section VO we defined vector addition and scalar multiplication.
These two operations combine nicely to give us a construction known as a
linear combination, a construct that we will work with throughout this
course.

So this definition takes an equal number of scalars and vectors, combines them
using our two new operations (scalar multiplication and vector addition) and
creates a single brand-new vector, of the same size as the original vectors. When a
definition or theorem employs a linear combination, think about the nature of the
objects that go into its creation (lists of scalars and vectors), and the type of
object that results (a single vector). Computationally, a linear combination is
pretty easy.

Notice how we could keep our set of vectors fixed, and use different sets of scalars
to construct different vectors. You might build a few new linear combinations of
{u}_{1},\kern 1.95872pt {u}_{2},\kern 1.95872pt {u}_{3},\kern 1.95872pt {u}_{4} right
now. We’ll be right here when you get back. What vectors were you able to
create? Do you think you could create the vector

w = \left [\array{
13\cr
15
\cr
5\cr
−17
\cr
2\cr
25} \right ]

with a “suitable” choice of four scalars? Do you think you could create any possible
vector from {ℂ}^{6}
by choosing the proper scalars? These last two questions are very
fundamental, and time spent considering them now will prove beneficial later.
⊠

Our next two examples are key ones, and a discussion about decompositions
is timely. Have a look at Technique DC before studying the next two
examples.

We can now interpret the problem of solving the system of equations as
determining values for the scalar multiples that make the vector equation true. In
the analysis of Archetype B, we were able to determine that it had only one
solution. A quick way to see this is to row-reduce the coefficient matrix to the
3 × 3
identity matrix and apply Theorem NMRRI to determine that the coefficient
matrix is nonsingular. Then Theorem NMUS tells us that the system of equations
has a unique solution. This solution is

\eqalignno{
{x}_{1} = −3 & &{x}_{2} = 5 & &{x}_{3} = 2 & & & & & &
}

So, in the context of this example, we can express the fact that these
values of the variables are a solution by writing the linear combination,

Furthermore, these are the only three scalars that will accomplish this
equality, since they come from a unique solution.

Notice how the three vectors in this example are the columns of the coefficient
matrix of the system of equations. This is our first hint of the important interplay
between the vectors that form the columns of a matrix, and the matrix itself.
⊠

Notice that these three vectors are the columns of the coefficient matrix for
the system of equations in Archetype A. This equality says there is a linear
combination of those columns that equals the vector of all zeros. Give it some
thought, but this says that

\eqalignno{
{x}_{1} = −1 & &{x}_{2} = 1 & &{x}_{3} = 1 & & & & & &
}

is a nontrivial solution to the homogeneous system of equations with
the coefficient matrix for the original system in Archetype A. In
particular, this demonstrates that this coefficient matrix is singular.
⊠

There’s a lot going on in the last two examples. Come back to them
in a while and make some connections with the intervening material.
For now, we will summarize and explain some of this behavior with a
theorem.

Theorem SLSLCSolutions to Linear Systems are Linear Combinations Denote the columns of the m × n
matrix A as the
vectors {A}_{1},\kern 1.95872pt {A}_{2},\kern 1.95872pt {A}_{3},\kern 1.95872pt \mathop{\mathop{…}},\kern 1.95872pt {A}_{n}. Then
x ∈ ℂn is a solution to the linear
system of equations ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right )
if and only if b
equals the linear combination of the columns of
A formed with
the entries of x,

Notice then that the entry of the coefficient matrix
A in row
i and column
j has two names:
{a}_{ij} as the coefficient
of {x}_{j} in equation
i of the system
and {\left [{A}_{j}\right ]}_{i} as the
i-th entry of the column
vector in column j of
the coefficient matrix A.
Likewise, entry i of
b has two names:
{b}_{i} from the linear
system and {\left [b\right ]}_{i}
as an entry of a vector. Our theorem is an equivalence (Technique E) so we need
to prove both “directions.”

( ⇐)
Suppose we have the vector equality between
b and the linear combination
of the columns of A.
Then for 1 ≤ i ≤ m,

This says that the entries of x
form a solution to equation i
of ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right ) for all
1 ≤ i ≤ m, in other
words, x is a
solution to ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right ).

( ⇒) Suppose now
that x is a solution to
the linear system ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right ).
Then for all 1 ≤ i ≤ m,

Since the components of b
and the linear combination of the columns of
A agree for
all 1 ≤ i ≤ m,
Definition CVE tells us that the vectors are equal.
■

In other words, this theorem tells us that solutions
to systems of equations are linear combinations of the
n column vectors of the
coefficient matrix ({A}_{j}) which
yield the constant vector b.
Or said another way, a solution to a system of equations
ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right )
is an answer to the question “How can I form the vector
b as a linear combination
of the columns of A?”
Look through the archetypes that are systems of equations and examine a few of
the advertised solutions. In each case use the solution to form a linear
combination of the columns of the coefficient matrix and verify that the result
equals the constant vector (see Exercise LC.C21).

Subsection VFSS: Vector Form of Solution Sets

We have written solutions to systems of equations as
column vectors. For example Archetype B has the solution
{x}_{1} = −3,\kern 1.95872pt {x}_{2} = 5,\kern 1.95872pt {x}_{3} = 2 which
we now write as

Now, we will use column vectors and linear combinations to express all of the
solutions to a linear system of equations in a compact and understandable way.
First, here’s two examples that will motivate our next theorem. This is a valuable
technique, almost the equal of row-reducing a matrix, so be sure you get
comfortable with it over the course of this section.

and we see r = 2 nonzero rows.
Also, D = \left \{1,\kern 1.95872pt 2\right \} so the dependent
variables are then {x}_{1}
and {x}_{2}.
F = \left \{3,\kern 1.95872pt 4,\kern 1.95872pt 5\right \} so the two free
variables are {x}_{3}
and {x}_{4}.
We will express a generic solution for the system by two slightly different
methods, though both arrive at the same conclusion.

First, we will decompose (Technique DC) a solution vector. Rearranging
each equation represented in the row-reduced form of the augmented
matrix by solving for the dependent variable in each row yields the vector
equality,

Step 3. For each dependent variable, use the augmented matrix to
formulate an equation expressing the dependent variable as a constant
plus multiples of the free variables. Convert this equation into entries of
the vectors that ensure equality for each dependent variable, one at a
time.

This final form of a typical solution is especially pleasing and useful. For example,
we can build solutions quickly by choosing values for our free variables, and then
compute a linear combination. Such as

You’ll find the second solution listed in the write-up for Archetype D, and you
might check the first solution by substituting it back into the original
equations.

While this form is useful for quickly creating solutions, it’s even better because it
tells us exactly what every solution looks like. We know the solution set is infinite,
which is pretty big, but now we can say that a solution is some multiple of
\left [\array{
−3\cr
−1
\cr
1\cr
0 } \right ]plus a multiple
of \left [\array{
2\cr
3
\cr
0\cr
1 } \right ] plus the
fixed vector \left [\array{
4\cr
0
\cr
0\cr
0 } \right ].
Period. So it only takes us three vectors to describe the entire infinite solution set,
provided we also agree on how to combine the three vectors into a linear combination.
⊠

This is such an important and fundamental technique, we’ll do another
example.

Example VFSVector form of solutions Consider a linear system of m = 5
equations in n = 7 variables, having
the augmented matrix A.

and we see r = 4 nonzero rows.
Also, D = \left \{1,\kern 1.95872pt 2,\kern 1.95872pt 5,\kern 1.95872pt 6\right \} so the dependent
variables are then {x}_{1},\kern 1.95872pt {x}_{2},\kern 1.95872pt {x}_{5},
and {x}_{6}.
F = \left \{3,\kern 1.95872pt 4,\kern 1.95872pt 7,\kern 1.95872pt 8\right \} so the
n − r = 3 free
variables are {x}_{3},\kern 1.95872pt {x}_{4}
and {x}_{7}.
We will express a generic solution for the system by two different methods: both a
decomposition and a construction.

First, we will decompose (Technique DC) a solution vector. Rearranging
each equation represented in the row-reduced form of the augmented
matrix by solving for the dependent variable in each row yields the vector
equality,

Step 3. For each dependent variable, use the augmented matrix to
formulate an equation expressing the dependent variable as a constant
plus multiples of the free variables. Convert this equation into entries of
the vectors that ensure equality for each dependent variable, one at a
time.

This final form of a typical solution is especially pleasing and useful. For example,
we can build solutions quickly by choosing values for our free variables, and then
compute a linear combination. For example

So we can compactly express all of the solutions to this linear system with just 4
fixed vectors, provided we agree how to combine them in a linear combinations to
create solution vectors.

Suppose you were told that the vector
w below was
a solution to this system of equations. Could you turn the problem around and write
w as a linear combination
of the four vectors c,
{u}_{1},
{u}_{2},
{u}_{3}? (See
Exercise LC.M11.)

Proof First, ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right )
is equivalent to the linear system of equations that has the matrix
B as
its augmented matrix (Theorem REMES), so we need only show that
S is the solution set
for the system with B
as its augmented matrix. The conclusion of this theorem is that the solution set is equal
to the set S,
so we will apply Definition SE.

We begin by showing that every element of
S is indeed a solution
to the system. Let {α}_{1},\kern 1.95872pt {α}_{2},\kern 1.95872pt {α}_{3},\kern 1.95872pt \mathop{\mathop{…}},\kern 1.95872pt {α}_{n−r}
be one choice of the scalars used to describe elements of
S. So an arbitrary
element of S,
which we will consider as a proposed solution is

When r + 1 ≤ ℓ ≤ m,
row ℓ of the
matrix B
is a zero row, so the equation represented by that row is always true, no matter
which solution vector we propose. So concentrate on rows representing equations
1 ≤ ℓ ≤ r. We evaluate equation
ℓ of the system represented
by B with the proposed
solution vector x
and refer to the value of the left-hand side of the equation as
{β}_{ℓ},

So {β}_{ℓ} began as the left-hand
side of equation ℓ of the
system represented by B and
we now know it equals {\left [B\right ]}_{ℓ,n+1}, the
constant term for equation ℓ
of this system. So the arbitrarily chosen vector from
S makes
every equation of the system true, and therefore is a solution to the system. So all the
elements of S
are solutions to the system.

For the second half of the proof, assume that
x is a solution vector for
the system having B
as its augmented matrix. For convenience and clarity, denote the entries of
x by
{x}_{i}, in other
words, {x}_{i} ={ \left [x\right ]}_{i}.
We desire to show that this solution vector is also an element of the set
S.
Begin with the observation that a solution vector’s entries makes equation
ℓ of the system
true for all 1 ≤ ℓ ≤ m,

This tells us that the entries of the solution vector
x
corresponding to dependent variables (indices in
D), are equal to those
of a vector in the set S.
We still need to check the other entries of the solution vector
x corresponding to the free
variables (indices in F)
to see if they are equal to the entries of the same vector in the set
S. To this end,
suppose i ∈ F
and i = {f}_{j}.
Then

So entries of x
and c + {x}_{{f}_{1}}{u}_{1} + {x}_{{f}_{2}}{u}_{2} + \mathrel{⋯} + {x}_{{f}_{n−r}}{u}_{n−r}
are equal and therefore by Definition CVE they are equal vectors. Since
{x}_{{f}_{1}},\kern 1.95872pt {x}_{{f}_{2}},\kern 1.95872pt {x}_{{f}_{3}},\kern 1.95872pt \mathop{\mathop{…}},\kern 1.95872pt {x}_{{f}_{n−r}} are scalars, this shows
us that x qualifies for
membership in S. So the
set S contains all of the
solutions to the system. ■

Note that both halves of the proof of Theorem VFSLS indicate that
{α}_{i} ={ \left [x\right ]}_{{f}_{i}}. In other words, the
arbitrary scalars, {α}_{i}, in the
description of the set S
actually have more meaning — they are the values of the free variables
{\left [x\right ]}_{{f}_{i}},
1 ≤ i ≤ n − r.
So we will often exploit this observation in our descriptions of solution
sets.

Theorem VFSLS formalizes what happened in the three steps of
Example VFSAD. The theorem will be useful in proving other theorems, and it it
is useful since it tells us an exact procedure for simply describing an infinite
solution set. We could program a computer to implement it, once we have
the augmented matrix row-reduced and have checked that the system is
consistent. By Knuth’s definition, this completes our conversion of linear
equation solving from art into science. Notice that it even applies (but is
overkill) in the case of a unique solution. However, as a practical matter, I
prefer the three-step process of Example VFSAD when I need to describe
an infinite solution set. So let’s practice some more, but with a bigger
example.

Step 2. For each free variable, use 0’s and 1’s to ensure equality for the
corresponding entry of the vectors. Take note of the pattern of 0’s and 1’s at this
stage, because this is the best look you’ll have at it. We’ll state an important
theorem in the next section and the proof will essentially rely on this observation.

Step 3. For each dependent variable, use the augmented matrix to
formulate an equation expressing the dependent variable as a constant
plus multiples of the free variables. Convert this equation into entries of
the vectors that ensure equality for each dependent variable, one at a
time.

We can now use this final expression to quickly build solutions to the system. You
might try to recreate each of the solutions listed in the write-up for Archetype I.
(Hint: look at the values of the free variables in each solution, and notice that the
vector c
has 0’s in these locations.)

Even better, we have a description of the infinite solution set, based
on just 5 vectors, which we combine in linear combinations to produce
solutions.

Whenever we discuss Archetype I you know that’s your cue to go work
through Archetype J by yourself. Remember to take note of the 0/1 pattern at
the conclusion of Step 2. Have fun — we won’t go anywhere while you’re away.
⊠

This technique is so important, that we’ll do one more example. However, an
important distinction will be that this system is homogeneous.

We’ll interpret it here as the coefficient matrix of
a homogeneous system and reference this matrix as
L. So we are solving the
homogeneous system ℒS\kern -1.95872pt \left (L,\kern 1.95872pt 0\right )
having m = 5
equations in n = 5
variables. If we built the augmented matrix, we would add a sixth column to
L
containing all zeros. As we did row operations, this sixth column would remain all
zeros. So instead we will row-reduce the coefficient matrix, and mentally
remember the missing sixth column of zeros. This row-reduced matrix is

and we see r = 3
nonzero rows. The columns with leading 1’s are
D = \{1,\kern 1.95872pt 2,\kern 1.95872pt 3\} so the
r dependent variables
are {x}_{1},\kern 1.95872pt {x}_{2},\kern 1.95872pt {x}_{3}. The columns
without leading 1’s are F = \{4,\kern 1.95872pt 5\},
so the n − r = 2 free
variables are {x}_{4},\kern 1.95872pt {x}_{5}.
Notice that if we had included the all-zero vector of constants to form the
augmented matrix for the system, then the index 6 would have appeared in the
set F,
and subsequently would have been ignored when listing the free variables.

Step 2. For each free variable, use 0’s and 1’s to ensure equality for the
corresponding entry of the vectors. Take note of the pattern of 0’s and
1’s at this stage, even if it is not as illuminating as in other examples.

Step 3. For each dependent variable, use the augmented matrix to
formulate an equation expressing the dependent variable as a constant
plus multiples of the free variables. Don’t forget about the “missing”
sixth column being full of zeros. Convert this equation into entries of
the vectors that ensure equality for each dependent variable, one at a
time.

The vector c
will always have 0’s in the entries corresponding to free variables. However, since we are
solving a homogeneous system, the row-reduced augmented matrix has zeros in column
n + 1 = 6, and hence all
the entries of c
are zero. So we can write

It will always happen that the solutions to a homogeneous system has
c = 0
(even in the case of a unique solution?). So our expression for the
solutions is a bit more pleasing. In this example it says that the
solutions are all possible linear combinations of the two vectors
{u}_{1} = \left [\array{
−1\cr
2
\cr
−2\cr
1
\cr
0 } \right ]and
{u}_{2} = \left [\array{
2\cr
−2
\cr
1\cr
0
\cr
1 } \right ], with
no mention of any fixed vector entering into the linear combination.

This observation will motivate our next section and the main definition of that
section, and after that we will conclude the section by formalizing this situation.
⊠

Subsection PSHS: Particular Solutions, Homogeneous Solutions

The next theorem tells us that in order to find all of the solutions to a linear
system of equations, it is sufficient to find just one solution, and then find all of
the solutions to the corresponding homogeneous system. This explains part of
our interest in the null space, the set of all solutions to a homogeneous
system.

After proving Theorem NMUS we commented (insufficiently)
on the negation of one half of the theorem. Nonsingular coefficient
matrices lead to unique solutions for every choice of the vector of
constants. What does this say about singular matrices? A singular matrix
A has a
nontrivial null space (Theorem NMTNS). For a given vector of constants,
b, the
system ℒS\kern -1.95872pt \left (A,\kern 1.95872pt b\right )
could be inconsistent, meaning there are no solutions. But if there is at least one
solution (w),
then Theorem PSPHS tells us there will be infinitely many solutions because of the
role of the infinite null space for a singular matrix. So a system of equations with a
singular coefficient matrix never has a unique solution. Either there are no solutions,
or infinitely many solutions, depending on the choice of the vector of constants
(b).

We will choose to have {y}_{1}
play the role of w
in the statement of Theorem PSPHS, any one of the three vectors listed
here (or others) could have been chosen. To illustrate the theorem, we
should be able to write each of these three solutions as the vector
w plus
a solution to the corresponding homogeneous system of equations. Since
0 is
always a solution to a homogeneous system we can easily write

{y}_{1} = w = w + 0.

The vectors {y}_{2}
and {y}_{3}
will require a bit more effort. Solutions to the homogeneous system
ℒS\kern -1.95872pt \left (A,\kern 1.95872pt 0\right ) are
exactly the elements of the null space of the coefficient matrix, which by an
application of Theorem VFSLS is

is obviously a solution of the homogeneous system since it is written as a linear
combination of the vectors describing the null space of the coefficient matrix (or
as a check, you could just evaluate the equations in the homogeneous system with
{z}_{2}).

is obviously a solution of the homogeneous system since it is written as a linear
combination of the vectors describing the null space of the coefficient matrix (or
as a check, you could just evaluate the equations in the homogeneous system with
{z}_{2}).

Here’s another view of this theorem, in the context of this example. Grab two
new solutions of the original system of equations, say

It is no accident that u
is a solution to the homogeneous system (check this!). In other words,
the difference between any two solutions to a linear system of equations
is an element of the null space of the coefficient matrix. This is an
equivalent way to state Theorem PSPHS. (See Exercise MM.T50).
⊠

The ideas of this subsection will appear again in Chapter LT when we discuss
pre-images of linear transformations (Definition PI).

Subsection READ: Reading Questions

Earlier, a reading question asked you to solve the system of equations

Subsection EXC: Exercises

C21 Consider each archetype that is a system of equations. For individual
solutions listed (both for the original system and the corresponding homogeneous
system) express the vector of constants as a linear combination of the columns of
the coefficient matrix, as guaranteed by Theorem SLSLC. Verify this equality by
computing the linear combination. For systems with no solutions, recognize that it
is then impossible to write the vector of constants as a linear combination of the
columns of the coefficient matrix. Note too, for homogeneous systems,
that the solutions give rise to linear combinations that equal the zero
vector.Archetype AArchetype BArchetype CArchetype DArchetype EArchetype FArchetype GArchetype HArchetype IArchetype J

Can it? Can any vector in {ℂ}^{6}
be written as a linear combination of the four vectors
{u}_{1},\kern 1.95872pt {u}_{2},\kern 1.95872pt {u}_{3},\kern 1.95872pt {u}_{4}?
Contributed by Robert BeezerSolution [340]

M11 At the end of Example VFS, the vector
w is
claimed to be a solution to the linear system under discussion. Verify that
w
really is a solution. Then determine the four scalars that express
w as a linear
combination of c,
{u}_{1},
{u}_{2},
{u}_{3}.
Contributed by Robert BeezerSolution [340]

M10 Contributed by Robert BeezerStatement [336]No, it is not possible to create w
as a linear combination of the four vectors
{u}_{1},\kern 1.95872pt {u}_{2},\kern 1.95872pt {u}_{3},\kern 1.95872pt {u}_{4}.
By creating the desired linear combination with unknowns as scalars,
Theorem SLSLC provides a system of equations that has no solution. This one
computation is enough to show us that it is not possible to create all the vectors
of {ℂ}^{6}
through linear combinations of the four vectors
{u}_{1},\kern 1.95872pt {u}_{2},\kern 1.95872pt {u}_{3},\kern 1.95872pt {u}_{4}.

M11 Contributed by Robert BeezerStatement [337]The coefficient of c is
1. The coefficients of {u}_{1},
{u}_{2},
{u}_{3}
lie in the third, fourth and seventh entries of
w. Can you see why?
(Hint: F = \left \{3,\kern 1.95872pt 4,\kern 1.95872pt 7,\kern 1.95872pt 8\right \}, so the
free variables are {x}_{3},\kern 1.95872pt {x}_{4}
and {x}_{7}.)