Orthonormal Sets

We begin with a review of what we have said about least squares problems in an inner product space, in particular a Hilbert space $\mathcal H$. Throughout, we will let $\{u_j\}_{j=1}^\infty$ be an o.n. set in $\mathcal H$.

Least squares. Let $V$ be a subspace of $\mathcal H$ and let
$f\in \mathcal H$. A vector $p \in V$ minimizes $\|f-v\|$ over all
$v\in V$ if and only if $f-p$ is orthogonal to $V$. If $p$ exists, it
is unique. Moreover, if $p$ exists for every $f\in \mathcal H$, then
there is an orthogonal projection $P_V$ such that $P_Vf=p$. Finally,
if $V$ is finite dimensional, the problem always has a solution, and
so $P_V$ always exists in that case.

Complete orthonormal sets. The question that now comes up is:
What happens when the o.n. set is infinite? Can we actually represent
every $f\in \mathcal H$ as $f = \sum_{j=1}^\infty\langle f,u_j\rangle
u_j$, where the series converges in $\mathcal H$? The answer is
``sometimes.'' When it is possible, we say that the o.n. set
$U:=\{u_j\}_{j=1}^\infty$ is complete. (Don't confuse this
completeness with completeness for a normed space. They are different
things.)

There are two equivalent definitions of completeness for an o.n. set:

Every vector in $\mathcal H$ may be uniquely represented as the
series $f=\sum_{j=1}^\infty \langle f, u_j\rangle u_j$.

$U$ is maximal in the sense that there is no non-zero vector in
$\mathcal H$ that is orthogonal to $U$. (Equivalently, $U$ is not a
proper subset of any other o.n. set in $\mathcal H$.)

We leave it as an exercise (
homework problem 4.6) to show the equivalence of these
statements. In the next few paragraphs we will explore aspects of an
o.n. set $U$, including conditions under which the set is complete.

Bessel's inequality . The equation $(\ast)$ applies if the
space for $V=\text{span}\{u_j\}_{j=1}^n$. Since $\|f-P_Vf\|^2\ge 0$,
$(\ast)$ implies that $\|f\|^2 - \sum_{j=1}^n\big |\langle
f,u_j\rangle \big|^2 \ge 0$. Consequently, for every $n$ we have
$\sum_{j=1}^n\big |\langle f,u_j\rangle \big|^2 \le \|f\|^2$. Recall
that if the partial sums of a series with non negative terms are
uniformly bounded, then that series is convergent. This implies the
inequality below holds for all $f\in \mathcal H$. \[ (\text{Bessel's
inequality})\quad \sum_{j=1}^\infty \big |\langle f,u_j\rangle \big|^2
\le \|f\|^2. \]

Parseval's equation. We now come to our first necessary and
sufficient condition for completeness.

Subspace generated by an o.n. set. Whether or not $U$ is
complete, we can still look at the subspace $\mathcal H_U : = \{g\in
\mathcal H : g = \sum_{j=1}^\infty \alpha_j u_j\}$. There are some
simple, but important consequences of $g$ being in $\mathcal H_U$. To
begin with, just using a simple limiting argument shows that $\langle
g,u_j\rangle = \alpha_j$. Put another way, the $\alpha_j$'s are
uniquely given by the previous formula.

The next thing involves $\mathcal H_U$ as a subspace of $\mathcal
H$. Recall that a subset of $\mathcal H$ is closed if it contains all
of its limit points. We want to prove the following result:

Proposition. $\mathcal H_U$ is a closed subspace of $\mathcal H$.

Proof. Let $\{g_\ell\}$ be a sequence in $\mathcal H_U$
convergent to $g\in \mathcal H$. We want to show that $g \in \mathcal
H_U$. To simplify the notation, we will let $P_n$ be the orthogonal
projection for $\text{span}\{u_j\}_{j=1}^n$. This allows us to write
$P_nf=\sum_{j=1}^n \langle f, u_j\rangle u_j$, so that we don't have
to use the sum in every case. Thus, our objective is to show that
$g=\lim_{n\to \infty}P_ng$. Indeed, $g\in \mathcal H_U$ if and only
$g=\lim_{n\to \infty}P_ng$.

First of all, $P_n g_\ell =\sum_{j=1}^n \langle g_\ell, u_j\rangle
u_j$ is in the span of the first $n$ of the $u_j$'s. From the
minimization properties of the projection (or, equivalently equation
$(\ast \ast)$), we have that
\[
\|g -P_ng\|\le \|g - P_ng_\ell\| \le \|g-g_\ell\| + \|g_\ell - P_n g_\ell\|.
\]
Now we employ the standard "up, over, and around" argument. For
$\epsilon>0$, we can find $L$ such that $\|g-g_\ell\|
The following corollary immediately follows from the proposition above
and the definition of completeness for an o.n. set.

Corollary. An o.n. set $U$ is complete if and only if $\mathcal
H_U = \mathcal H$.

Checking for completeness. In all of the results above,
checking whether an o.n. set is or is not complete involves the whole
space $\mathcal H$. Usually this is difficult to do. We now present an
alternative method that allows dealing only with a dense subset of
$\mathcal H$.

Proposition. Let $D$ be a dense subset of $\mathcal H$. An
o.n. set $U$ is complete if and only if $D\subset \mathcal H_U$.

Proof. Obviously, if $U$ is complete, $\mathcal H_U = \mathcal
H \supset D$. To show the converse, suppose that $D \subset
H_U$. Since $H_U$ is closed, it contains all of its limit points. But
$D$ is dense in $\mathcal H$, so its limit points comprise all of
$\mathcal H$. It follows that $\mathcal H_U= \mathcal H$ and that $U$
is complete.

We can restate the proposition above in an equivalent way, which will
provide a direct check on showing the completeness of $U$.

Corollary Let $D$ be a dense subset of $\mathcal H$. An
o.n. set $U$ is complete if and only if every vector $f\in D$ can be
represented a series of the form $f=\sum_{j=1}^\infty \alpha_j u_j$.

Proof. The condition on $f\in D$ is simply a statement that
$f\in \mathcal H_U$, so the result follows from the previous
proposition.

Orthogonal Polynomials

Let's look at the specific case of $L^2[a,b]$, where $a$ and $b$ are
finite. The inner product for this space is $\langle f,g\rangle =
\int_a^b f(x)\overline{g(x)}dx$. We also are given that $C[a,b]$ is
dense in $L^2[a,b]$. The result below is a consequence of this fact.

Proposition. The polynomials are dense in $L^2[a,b]$.

Proof. Let $f\in L^2$. Because the continuous functions are
dense in $L^2$, for every $\epsilon >0$ we have that there is a $g\in
C[a,b]$ such that $\|f-g\|_{L^2} < \frac12 \epsilon$. By the
Weierstrass approximation theorem, which we will discuss next, we can
also find a polynomial $p$ such that $\|g - p\|_{C[a,b]}
We point out that most sets of orthogonal polynomials are obtained by
using the Gram-Schmidt process on the set $\{1,x,x^2, \ldots\}$
relative to some weighted inner product, $\langle f,g\rangle
= \int_a^b f(x)\overline{g(x)}w(x)dx$. In the case where $w(x)=1$, the
result above proves that the polynomials orthogonal relative to the
inner product $\langle f,g\rangle = \int_a^b f(x)\overline{g(x)}dx$
form a complete set.

Let $a=-1$ and $b=1$, so that $\langle f,g\rangle = \int_{-1}^1
f(x)\overline{g(x)}dx$. The polynomials generated from this inner
product via the Gram-Schmidt process are called the Legendre
polynomials. The $\ell^{th}$ degree Legendre polynomial is
usually denoted by $P_\ell(x)$ and is normalized so that
$P_\ell(1)=1$; for example, we have
\[
P_0(x)=1,\ P_1(x) = x,\ P_2(x) =
\tfrac12\!(3x^2-1),\ P_3(x)=\tfrac12 \!(5x^3-3x).
\]
The $P_\ell$'s are orthogonal, but not orthonormal. It turns out that
$\|P_\ell\|_{L^2[-1,1]} = \sqrt{\frac{2}{2\ell+1}}$. Letting $p_\ell =
\sqrt{\frac{2\ell+1}{2}}P_\ell$ results in an orthonormal set,
$\{p_0,p_1, p_2\ldots\}$. For $\ell= 0$ to $\ell=4$, they are
\[
p_0(x)=\sqrt{\tfrac12},\ p_1(x) = \sqrt{\tfrac32} x,\ p_2(x)
=\sqrt{\tfrac58}(3x^2-1),\ p_3(x)=\sqrt{\tfrac78}(5x^3-3x).
\]
We can turn this around and write powers of $x$ in terms of the
$p_\ell $'s. For example,
\[
1=\sqrt{2}p_0,\ x = \sqrt{\tfrac23}\,p_1, \ x^2 =
\sqrt{\tfrac{2}{9}}p_0+ \sqrt{\tfrac{8}{45}}p_2,\ x^3 =
\sqrt{\tfrac{6}{25}}p_1+ \sqrt{\tfrac{8}{175}}p_3.
\]
The point is that every polynomial can be written as a linear
combination of the $p_\ell$'s, so they all belong to $\mathcal H_U$,
with $U=\{p_\ell\}_{\ell=0}^\infty$. By the two previous propositions,
we obtain this:

Corollary. The Legendre polynomials form a complete set in $L^2[-1,1]$.