Contents / Linear Algebra / Inner Product Spaces

Chapter 10

Inner Product Spaces

Abstract inner products, norms, orthogonal polynomials, Fourier approximation, and complex Hermitian and unitary matrices.

Introduction

A vector space knows about addition and scaling, and nothing else. It cannot tell you how long a vector is, whether two vectors are perpendicular, or which vector in a subspace is closest to a given point. Those are metric questions, and in $\mathbb{R}^n$ they are all answered by one gadget: the dot product.

The subject of this chapter is what happens when you take the algebraic properties of the dot product, promote them to axioms, and impose them on an arbitrary vector space. The payoff is enormous and slightly unreasonable. Length, angle, perpendicularity, projection and the Pythagorean theorem all survive the move — and they survive into spaces whose "vectors" are polynomials, matrices, or continuous functions. Fourier series, least-squares curve fitting, Legendre and Chebyshev polynomials and the quantum-mechanical notion of a state all turn out to be the same geometry, done in a space where you cannot draw a picture.

The plan: the axioms and a catalogue of examples; the norm they induce; Cauchy–Schwarz and the triangle inequality, proved once and for all; angles and orthogonality; orthonormal sets and Gram–Schmidt; orthogonal complements; orthogonal projection and best approximation; Bessel and Parseval; Fourier series as a projection; and finally the complex case, where conjugation enters, together with the adjoint of an operator — the doorway to the spectral theorem.

10.1Inner products: the axioms

Take the dot product $x \cdot y = x_1y_1 + \cdots + x_ny_n$ on $\mathbb{R}^n$ and ask what it actually does, forgetting the formula. It is symmetric, it is linear in each slot, and $x \cdot x$ is a sum of squares, hence positive unless $x = 0$ . That short list is enough to rebuild all of Euclidean geometry, so we make it the definition.

Definition 10.1 (Real inner product space). Let $V$ be a vector space over $\mathbb{R}$ . An inner product on $V$ is a function $\langle\,\cdot\,,\cdot\,\rangle : V \times V \to \mathbb{R}$ such that for all $u, v, w \in V$ and all $c \in \mathbb{R}$ :

Symmetry: $\langle u, v\rangle = \langle v, u\rangle$ .
Additivity in the first slot: $\langle u + v, w\rangle = \langle u, w\rangle + \langle v, w\rangle$ .
Homogeneity in the first slot: $\langle cu, v\rangle = c\,\langle u, v\rangle$ .
Positive definiteness: $\langle v, v\rangle \ge 0$ , with $\langle v, v\rangle = 0$ only when $v = 0$ .

A vector space carrying an inner product is an inner product space.

Three remarks on the fine print, because each axiom is doing work.

Symmetry plus linearity in the first slot gives linearity in the second slot for free: $\langle u, cv + w\rangle = \langle cv + w, u\rangle = c\langle v,u\rangle + \langle w,u\rangle = c\langle u,v\rangle + \langle u,w\rangle$ . So a real inner product is bilinear, and we never have to state the second-slot rules separately. (Over $\mathbb{C}$ this breaks, and the repair occupies the last two sections of the chapter.)

Positive definiteness is the axiom with teeth. Without the strictness — if we allowed $\langle v,v\rangle = 0$ for some $v \ne 0$ — we would have a semi-inner product, in which a nonzero vector has length zero, and nothing about distance or approximation would work. Note also that axioms 2 and 3 already force $\langle 0, v\rangle = \langle 0\cdot 0, v\rangle = 0\,\langle 0,v\rangle = 0$ , so the zero vector is orthogonal to everything, including itself.

Finally, notice what is not assumed: nothing about coordinates, nothing about dimension, no basis. That is why the definition transplants into spaces of functions.

Intuition. Think of $\langle u, v\rangle$ as a machine that measures agreement. Feed it two vectors pointing the same way and it returns a large positive number; feed it two opposed vectors and it returns a large negative one; feed it two perpendicular vectors and it returns zero. Feeding it the same vector twice asks "how much does $v$ agree with itself?", and the answer had better be positive — that is axiom 4, and it is what makes $\sqrt{\langle v,v\rangle}$ deserve the name length.

Here is the catalogue. Each of these is an inner product space you will meet again.

Example 10.2 (The standard inner product on $\mathbb{R}^n$ ). Verify that $\langle x, y\rangle = x_1y_1 + \cdots + x_ny_n$ satisfies the axioms.

Solution. Symmetry is clear since $x_iy_i = y_ix_i$ . Additivity and homogeneity in the first slot follow term by term from the distributive law in $\mathbb{R}$ . For positivity,

\langle x, x\rangle = x_1^2 + \cdots + x_n^2 \ge 0,

a sum of squares, and it vanishes only when every $x_i = 0$ , i.e. when $x = 0$ . In matrix notation $\langle x,y\rangle = x^{\mathsf T}y$ , which is the form we use whenever matrices are nearby.□

Example 10.3 (A weighted inner product). Fix constants $w_1, \dots, w_n$ and define $\langle x, y\rangle_w = \sum_{i=1}^n w_i x_i y_i$ on $\mathbb{R}^n$ . For which weights is this an inner product?

Solution. Symmetry and bilinearity hold for any real weights, since each term is symmetric and linear in $x_i$ . Everything depends on axiom 4:

\langle x, x\rangle_w = \sum_{i=1}^n w_i x_i^2.

If every $w_i > 0$ this is a sum of nonnegative terms, zero only when all $x_i = 0$ ; so the axioms hold. If some $w_k \le 0$ , take $x = e_k$ : then $\langle e_k, e_k\rangle_w = w_k \le 0$ with $e_k \ne 0$ , and positive definiteness fails.

So $\langle\,\cdot\,,\cdot\,\rangle_w$ is an inner product exactly when all weights are strictly positive. Weighted inner products are how statistics encodes "these measurements are more reliable than those": a large $w_i$ makes discrepancies in coordinate $i$ expensive.□

Example 10.4 (The integral inner product on $C[a,b]$ ). Let $C[a,b]$ be the space of continuous real functions on $[a,b]$ and set

\langle f, g\rangle = \int_a^b f(t)g(t)\,dt.

Show this is an inner product.

Solution. Symmetry holds because $f(t)g(t) = g(t)f(t)$ pointwise. Linearity in the first slot is linearity of the integral. For positivity,

\langle f, f\rangle = \int_a^b f(t)^2\,dt \ge 0

since the integrand is nonnegative. The only delicate point is strictness. Suppose $f$ is continuous and $\int_a^b f^{2} = 0$ but $f(t_0) \ne 0$ for some $t_0$ . By continuity $f(t)^2 > f(t_0)^2/2$ on some interval of positive length $\delta$ around $t_0$ , so

\int_a^b f(t)^2\,dt \ge \frac{f(t_0)^2}{2}\,\delta > 0,

a contradiction. Hence $f \equiv 0$ . Continuity is essential here: on a space of merely integrable functions, a function that is $1$ at a single point and $0$ elsewhere is nonzero yet has integral zero, and we would have only a semi-inner product.□

Example 10.5 (The trace inner product on matrices). On $M_{m\times n}(\mathbb{R})$ define $\langle A, B\rangle = \operatorname{tr}(A^{\mathsf T}B)$ . Show this is an inner product, and compute $\langle A,B\rangle$ for

A = \begin{pmatrix} 1 & 2 \\ 0 & -1\end{pmatrix}, \qquad B = \begin{pmatrix} 3 & 1 \\ 4 & 2\end{pmatrix}.

Solution. Write out the trace. The $(j,j)$ entry of $A^{\mathsf T}B$ is $\sum_i (A^{\mathsf T})_{ji}B_{ij} = \sum_i a_{ij}b_{ij}$ , so

\operatorname{tr}(A^{\mathsf T}B) = \sum_{j}\sum_{i} a_{ij}b_{ij} = \sum_{i,j} a_{ij}b_{ij}.

That is exactly the dot product of $A$ and $B$ regarded as vectors of length $mn$ . All four axioms therefore transfer from the standard inner product on $\mathbb{R}^{mn}$ ; in particular $\langle A,A\rangle = \sum a_{ij}^2 \ge 0$ with equality only for $A = 0$ .

For the given matrices, $\langle A,B\rangle = 1\cdot 3 + 2\cdot 1 + 0\cdot 4 + (-1)\cdot 2 = 3 + 2 + 0 - 2 = 3$ . Sanity check by the trace route:

A^{\mathsf T}B = \begin{pmatrix}1 & 0\\ 2 & -1\end{pmatrix}\begin{pmatrix}3&1\\4&2\end{pmatrix} = \begin{pmatrix}3 & 1\\ 2 & 0\end{pmatrix},

whose trace is $3 + 0 = 3$ . The associated norm $\|A\| = \sqrt{\sum a_{ij}^2}$ is the Frobenius norm.□

Example 10.6 (An evaluation inner product on polynomials). Let $\mathcal{P}_n$ be the polynomials of degree at most $n$ , fix distinct reals $t_0, t_1, \dots, t_n$ , and define

\langle p, q\rangle = \sum_{k=0}^{n} p(t_k)\,q(t_k).

Why is positive definiteness true here, and why does it fail if you use only $n$ points?

Solution. Symmetry and bilinearity are immediate. For positivity, $\langle p,p\rangle = \sum_k p(t_k)^2 \ge 0$ , and it is zero only if $p(t_k) = 0$ for all $k$ — that is, only if $p$ has $n+1$ distinct roots. A nonzero polynomial of degree at most $n$ has at most $n$ roots, so $p$ must be the zero polynomial. The axioms hold.

With only $n$ sample points $t_1,\dots,t_n$ , the nonzero polynomial $p(t) = (t-t_1)\cdots(t-t_n)$ , of degree $n$ , satisfies $\langle p,p\rangle = 0$ . Positive definiteness collapses. The moral: sampling is an inner product precisely when you sample often enough to determine the polynomial.□

Pitfall. Positive definiteness is the axiom that fails in practice, and it fails quietly. A symmetric bilinear form such as $\langle x,y\rangle = x_1y_1 - x_2y_2$ on $\mathbb{R}^2$ looks harmless, but $\langle (1,1),(1,1)\rangle = 0$ with $(1,1)\ne 0$ — this is the Minkowski form of special relativity, and it is not an inner product. Always test axiom 4 before assuming the geometry of this chapter applies.

Remark. A symmetric bilinear form on $\mathbb{R}^n$ can always be written $\langle x,y\rangle = x^{\mathsf T}\!Ay$ for a symmetric matrix $A$ , and it is an inner product exactly when $A$ is positive definite, i.e. $x^{\mathsf T}\!Ax > 0$ for all $x \ne 0$ . The weighted inner products are the case where $A$ is diagonal with positive entries. The chapter on eigenvalues gives a test: $A$ is positive definite precisely when all its eigenvalues are positive.

Exercises for §10.1 →

10.2Norms, distance, and the norm axioms

Once you can measure agreement you can measure size: a vector's length is how much it agrees with itself.

Definition 10.7 (Induced norm and distance). In an inner product space $V$ , the norm of $v \in V$ is

\|v\| = \sqrt{\langle v, v\rangle},

which is well defined because $\langle v,v\rangle \ge 0$ . The distance between $u$ and $v$ is $d(u,v) = \|u - v\|$ . A vector with $\|v\| = 1$ is a unit vector, and for $v \ne 0$ the vector $v/\|v\|$ is the normalisation of $v$ .

Two immediate consequences, both used constantly. First, $\|cv\| = |c|\,\|v\|$ , since $\langle cv,cv\rangle = c^{2}\langle v,v\rangle$ ; note the absolute value, which appears because the square root is nonnegative. Second, $\|v\| = 0$ if and only if $v = 0$ , directly from axiom 4. The normalisation $v/\|v\|$ really is a unit vector: its norm is $\|v\|/\|v\| = 1$ .

Expanding a norm of a sum is the single most used computation in the chapter, so record it:

Proposition 10.8 (Expansion of $\|u+v\|^2$ ). For all $u,v$ in a real inner product space,

\|u+v\|^2 = \|u\|^2 + 2\langle u,v\rangle + \|v\|^2, \qquad \|u-v\|^2 = \|u\|^2 - 2\langle u,v\rangle + \|v\|^2.

Proof. By bilinearity and symmetry,

\langle u+v, u+v\rangle = \langle u,u\rangle + \langle u,v\rangle + \langle v,u\rangle + \langle v,v\rangle = \|u\|^2 + 2\langle u,v\rangle + \|v\|^2 .

Replacing $v$ by $-v$ and using $\langle u,-v\rangle = -\langle u,v\rangle$ gives the second identity.∎

Example 10.9 (Norm and distance in function space). In $C[0,1]$ with $\langle f,g\rangle = \int_0^1 fg$ , compute $\|f\|$ for $f(t) = t$ , and the distance from $f$ to the constant function $g(t) = \tfrac12$ .

Solution. First the norm:

\|f\|^2 = \int_0^1 t^2\,dt = \frac13, \qquad \|f\| = \frac{1}{\sqrt3}.

For the distance, $(f-g)(t) = t - \tfrac12$ , so

d(f,g)^2 = \int_0^1 \left(t - \tfrac12\right)^2 dt = \left[\frac{(t-\frac12)^3}{3}\right]_0^1 = \frac{1}{24} + \frac{1}{24} = \frac{1}{12},

giving $d(f,g) = 1/(2\sqrt3)$ . Sanity check: $f$ and $g$ agree at $t = \tfrac12$ and differ by at most $\tfrac12$ anywhere, so the distance should be comfortably below $\tfrac12$ , and $1/(2\sqrt3) \approx 0.289$ is.□

Example 10.10 (Normalising a polynomial). In $\mathcal{P}_2$ with $\langle p,q\rangle = \int_{-1}^{1} p(t)q(t)\,dt$ , normalise $p(t) = t^{2}$ .

Solution.

\|p\|^2 = \int_{-1}^{1} t^4\,dt = \left[\frac{t^5}{5}\right]_{-1}^{1} = \frac{2}{5},

so $\|p\| = \sqrt{2/5}$ and the normalisation is

\frac{p}{\|p\|} = \sqrt{\frac{5}{2}}\;t^2 .

Check: $\int_{-1}^1 \frac52 t^{4}\,dt = \frac52\cdot\frac25 = 1$ .□

The word norm has a life independent of inner products, and it is worth knowing which properties are which.

Definition 10.11 (Norm axioms). A norm on a real vector space $V$ is a function $\|\cdot\| : V \to \mathbb{R}$ with

$\|v\| \ge 0$ , and $\|v\| = 0$ only for $v = 0$ ;
$\|cv\| = |c|\,\|v\|$ for all scalars $c$ ;
the triangle inequality $\|u+v\| \le \|u\| + \|v\|$ .

The first two we have already verified for the induced norm. The third takes work, and the work is Cauchy–Schwarz, which is the next section. Meanwhile, not every norm comes from an inner product: on $\mathbb{R}^n$ the taxicab norm $\|x\|_1 = \sum |x_i|$ and the max norm $\|x\|_\infty = \max_i |x_i|$ satisfy all three axioms but, as the parallelogram law will show, are not induced by any inner product.

Pitfall. $\|u + v\| \ne \|u\| + \|v\|$ in general, and the failure is the whole point. Equality holds only when $u$ and $v$ point the same way. Students who "distribute" a norm over a sum — or, worse, write $\|u+v\|^2 = \|u\|^2 + \|v\|^2$ without checking orthogonality — are committing the same error twice.

Exercises for §10.2 →

10.3Cauchy–Schwarz and the triangle inequality

Everything metric in this chapter rests on one inequality, and it has a one-line proof once you see the right vector to square.

Theorem 10.12 (Cauchy–Schwarz inequality). For all $u, v$ in a real inner product space,

|\langle u, v\rangle| \le \|u\|\,\|v\|,

with equality if and only if $u$ and $v$ are linearly dependent.

Proof. If $v = 0$ both sides are $0$ and $u,v$ are dependent, so assume $v \ne 0$ and set

c = \frac{\langle u,v\rangle}{\|v\|^2}, \qquad w = u - cv .

The vector $w$ is $u$ with its $v$ -component removed, and it is orthogonal to $v$ : indeed $\langle w, v\rangle = \langle u,v\rangle - c\|v\|^2 = 0$ . Now expand $\|u\|^2 = \|w + cv\|^2$ using the expansion proposition, whose cross term $2c\langle w,v\rangle$ vanishes:

\|u\|^2 = \|w\|^2 + c^2\|v\|^2 = \|w\|^2 + \frac{\langle u,v\rangle^2}{\|v\|^2}.

Since $\|w\|^2 \ge 0$ this gives $\langle u,v\rangle^{2} \le \|u\|^2\|v\|^2$ , and taking square roots yields the inequality.

Equality forces $\|w\| = 0$ , hence $w = 0$ by positive definiteness, hence $u = cv$ : the vectors are dependent. Conversely if $u = cv$ then $|\langle u,v\rangle| = |c|\,\|v\|^2 = \|u\|\|v\|$ .∎

The proof is worth rereading, because it is the chapter in miniature: split $u$ into a piece along $v$ and a piece orthogonal to $v$ , then observe that the orthogonal piece only ever adds to the length. That decomposition reappears as orthogonal projection, as Bessel's inequality, and as the best-approximation theorem.

Corollary 10.13 (Triangle inequality). $\|u+v\| \le \|u\| + \|v\|$ for all $u,v$ , so the induced norm really is a norm.

Proof. Expand and apply Cauchy–Schwarz to the cross term:

\|u+v\|^2 = \|u\|^2 + 2\langle u,v\rangle + \|v\|^2 \le \|u\|^2 + 2\|u\|\|v\| + \|v\|^2 = \big(\|u\| + \|v\|\big)^2 .

Both sides are nonnegative, so taking square roots preserves the inequality.∎

Corollary 10.14 (Reverse triangle inequality).

\big|\,\|u\| - \|v\|\,\big| \le \|u - v\|

Proof. Write $u = (u-v) + v$ and apply the triangle inequality: $\|u\| \le \|u-v\| + \|v\|$ , so $\|u\| - \|v\| \le \|u-v\|$ . Swapping $u$ and $v$ gives $\|v\| - \|u\| \le \|v-u\| = \|u-v\|$ . Together these bound the absolute value.∎

Intuition. Cauchy–Schwarz says that the "agreement" between two vectors can never exceed the product of their sizes — you cannot get more overlap out of two arrows than their lengths allow. In the function space $C[0,1]$ it says something far less obvious: for any two continuous functions,

\left(\int_0^1 fg\right)^{\!2} \le \int_0^1 f^2 \cdot \int_0^1 g^2 .

That is a genuine theorem about integrals, and we got it for free from four axioms about arrows.

Example 10.15 (Cauchy–Schwarz on a concrete pair). Verify Cauchy–Schwarz for $u = (1,2,2)$ and $v = (3,0,4)$ in $\mathbb{R}^3$ , and report the ratio $|\langle u,v\rangle| / (\|u\|\|v\|)$ .

Solution. $\langle u,v\rangle = 3 + 0 + 8 = 11$ . Also $\|u\| = \sqrt{1+4+4} = 3$ and $\|v\| = \sqrt{9+0+16} = 5$ . So $11 \le 15$ , as promised, and the ratio is $11/15 \approx 0.733$ . The vectors are not parallel, so strict inequality is expected.□

Example 10.16 (An integral inequality from the axioms). Show that for every continuous $f$ on $[0,1]$ ,

\left(\int_0^1 f(t)\,dt\right)^{\!2} \le \int_0^1 f(t)^2\,dt .

Solution. Apply Cauchy–Schwarz in $C[0,1]$ with $g(t) = 1$ . Then $\langle f,g\rangle = \int_0^1 f$ , $\|g\|^2 = \int_0^1 1\,dt = 1$ , and $\|f\|^2 = \int_0^1 f^{2}$ . The inequality $\langle f,g\rangle^{2} \le \|f\|^2\|g\|^2$ reads exactly as claimed.

Equality holds iff $f$ and $1$ are dependent, i.e. iff $f$ is constant — which matches intuition: averaging destroys information unless there was no variation to destroy.□

Example 10.17 (A sharp numerical bound). Among all $(x,y,z)$ with $x^{2} + y^{2} + z^{2} = 1$ , how large can $x + 2y + 2z$ be?

Solution. Write $x + 2y + 2z = \langle (x,y,z), (1,2,2)\rangle$ . By Cauchy–Schwarz this is at most $\|(x,y,z)\|\cdot\|(1,2,2)\| = 1 \cdot 3 = 3$ .

Equality requires $(x,y,z)$ parallel to $(1,2,2)$ and of unit norm, so $(x,y,z) = \tfrac13(1,2,2)$ , which indeed gives $\tfrac13(1+4+4) = 3$ . The maximum is $3$ . This is the standard trick for bounding a linear expression on a sphere, and it works verbatim in function space.□

Pitfall. Cauchy–Schwarz bounds the absolute value of the inner product. Forgetting the bars loses the lower bound $-\|u\|\|v\| \le \langle u,v\rangle$ , which is the half you need when you are minimising rather than maximising.

Exercises for §10.3 →

10.4Angles, orthogonality and the Pythagorean theorem

Cauchy–Schwarz has a consequence that is easy to overlook: for nonzero $u,v$ the quotient

\frac{\langle u,v\rangle}{\|u\|\,\|v\|}

always lies in $[-1,1]$ . Every number in that range is the cosine of exactly one angle in $[0,\pi]$ , so the quotient defines an angle — in any inner product space whatsoever.

Definition 10.18 (Angle between vectors). For nonzero $u,v$ in a real inner product space, the angle $\theta \in [0,\pi]$ between them is the unique solution of

\cos\theta = \frac{\langle u, v\rangle}{\|u\|\,\|v\|}.

In $\mathbb{R}^2$ and $\mathbb{R}^3$ this agrees with the angle you would measure with a protractor — that is the content of the law of cosines. In $C[a,b]$ it is a definition with no prior meaning, and it is none the worse for that: it is how one says that two signals are "nearly aligned" or "in quadrature".

Definition 10.19 (Orthogonality). Vectors $u$ and $v$ are orthogonal, written $u \perp v$ , if $\langle u,v\rangle = 0$ . A set $S \subseteq V$ is orthogonal if its vectors are pairwise orthogonal.

Orthogonality, not angle, is what gets used. The angle is a convenience; the vanishing of an inner product is a structural fact, and the rest of the chapter is a study of its consequences.

Theorem 10.20 (Pythagorean theorem). If $u \perp v$ then $\|u + v\|^2 = \|u\|^2 + \|v\|^2$ . More generally, if $v_1,\dots,v_k$ are pairwise orthogonal then

\|v_1 + \cdots + v_k\|^2 = \|v_1\|^2 + \cdots + \|v_k\|^2 .

Proof. Expand the square: $\|u+v\|^2 = \|u\|^2 + 2\langle u,v\rangle + \|v\|^2$ , and the cross term is zero by hypothesis. For $k$ vectors, bilinearity gives

\Big\langle \sum_i v_i, \sum_j v_j\Big\rangle = \sum_{i,j} \langle v_i, v_j\rangle,

and every term with $i \ne j$ vanishes, leaving $\sum_i \|v_i\|^2$ .∎

Over $\mathbb{R}$ the converse holds too: if $\|u+v\|^2 = \|u\|^2+\|v\|^2$ then $2\langle u,v\rangle = 0$ , so $u \perp v$ . (This converse fails over $\mathbb{C}$ , as we note later.)

Orthogonality also buys independence, which is why orthogonal sets are such convenient bases.

Theorem 10.21 (Orthogonal sets are independent). An orthogonal set of nonzero vectors is linearly independent.

Proof. Suppose $v_1,\dots,v_k$ are pairwise orthogonal, all nonzero, and $c_1v_1 + \cdots + c_kv_k = 0$ . Take the inner product of both sides with $v_j$ :

0 = \Big\langle \sum_i c_i v_i,\; v_j\Big\rangle = \sum_i c_i \langle v_i, v_j\rangle = c_j\|v_j\|^2,

because every term with $i \ne j$ dies. Since $v_j \ne 0$ we have $\|v_j\|^2 > 0$ , so $c_j = 0$ . This holds for each $j$ , so the only relation is the trivial one.∎

That proof also reveals the trick that makes orthogonal bases so pleasant: to extract the coefficient of $v_j$ , pair with $v_j$ and everything else disappears. We exploit this systematically two sections from now.

Theorem 10.22 (Parallelogram law). In any inner product space,

\|u+v\|^2 + \|u-v\|^2 = 2\|u\|^2 + 2\|v\|^2 .

Proof. Add the two expansions from the expansion proposition: the cross terms $+2\langle u,v\rangle$ and $-2\langle u,v\rangle$ cancel, leaving $2\|u\|^2 + 2\|v\|^2$ .∎

Geometrically: the sum of the squares of a parallelogram's diagonals equals the sum of the squares of its four sides. Its real use, though, is as a test. Any norm induced by an inner product must satisfy it, so a norm that violates it cannot come from one.

Example 10.23 (The taxicab norm has no inner product behind it). Show that $\|x\|_1 = |x_1| + |x_2|$ on $\mathbb{R}^2$ is not induced by any inner product.

Solution. Take $u = (1,0)$ and $v = (0,1)$ . Then $u + v = (1,1)$ and $u - v = (1,-1)$ , so

\|u+v\|_1^2 + \|u-v\|_1^2 = 2^2 + 2^2 = 8,

while

2\|u\|_1^2 + 2\|v\|_1^2 = 2 + 2 = 4 .

Remark. The converse is also true and is a classical theorem (Jordan–von Neumann): a norm satisfying the parallelogram law is induced by an inner product, and the inner product can be recovered from the norm by the polarisation identity

\langle u,v\rangle = \frac{\|u+v\|^2 - \|u-v\|^2}{4},

which follows by subtracting the two expansions. So "inner product" and "norm obeying the parallelogram law" are the same information packaged two ways.

Example 10.24 (An angle in function space). In $C[0,1]$ with the integral inner product, find the angle between $f(t) = 1$ and $g(t) = t$ .

Solution. Compute the three quantities:

\langle f,g\rangle = \int_0^1 t\,dt = \frac12, \qquad \|f\| = 1, \qquad \|g\| = \sqrt{\int_0^1 t^2\,dt} = \frac{1}{\sqrt3}.

Hence

\cos\theta = \frac{1/2}{1\cdot 1/\sqrt3} = \frac{\sqrt3}{2},

so $\theta = \pi/6$ . The constant function and the identity function "agree" strongly, which is why fitting data with a line rather than a constant often changes the intercept so much: the two basis functions are far from perpendicular.□

Example 10.25 (Making two functions orthogonal). Find the constant $c$ for which $t - c$ is orthogonal to $1$ in $C[0,1]$ .

Solution. Require

\langle t - c, 1\rangle = \int_0^1 (t - c)\,dt = \frac12 - c = 0,

so $c = \tfrac12$ . The function $t - \tfrac12$ is the "mean-zero part" of $t$ ; subtracting the average is exactly the act of making something orthogonal to the constants. This is the first step of Gram–Schmidt, performed by hand.□

Example 10.26 (Orthogonality in the trace inner product). For which $x$ are $A = \begin{pmatrix}1&2\\3&x\end{pmatrix}$ and $B = \begin{pmatrix}2&-1\\0&4\end{pmatrix}$ orthogonal in $M_2(\mathbb{R})$ ?

Solution. $\langle A,B\rangle = 1\cdot2 + 2\cdot(-1) + 3\cdot 0 + x\cdot 4 = 2 - 2 + 0 + 4x = 4x$ . This vanishes exactly when $x = 0$ .□

Pitfall. The Pythagorean theorem needs pairwise orthogonality, not just "the vectors look spread out". For three vectors, $\|v_1+v_2+v_3\|^2 = \sum\|v_i\|^2$ requires all three cross terms to vanish. Two orthogonal pairs and one non-orthogonal pair is not enough.

Exercises for §10.4 →

10.5Orthonormal sets and coordinates

Normalising an orthogonal set costs nothing and simplifies every formula that follows.

Definition 10.27 (Orthonormal set). A set $\{e_1,\dots,e_k\}$ is orthonormal if

\langle e_i, e_j\rangle = \delta_{ij} = \begin{cases} 1, & i = j,\\ 0, & i \ne j.\end{cases}

An orthonormal basis of $V$ is an orthonormal set that is also a basis.

Since orthonormal vectors are nonzero and pairwise orthogonal, they are automatically independent. So in an $n$ -dimensional space, any orthonormal set of $n$ vectors is already a basis — no further checking required.

The reason to want such a basis is that coordinates become inner products, computed one at a time, with no linear system to solve.

Theorem 10.28 (Coordinates in an orthonormal basis). Let $\{e_1,\dots,e_n\}$ be an orthonormal basis of $V$ . Then for every $v \in V$ ,

v = \langle v,e_1\rangle e_1 + \cdots + \langle v,e_n\rangle e_n, \qquad \|v\|^2 = \sum_{i=1}^{n} \langle v, e_i\rangle^2 .

More generally, for $u,v \in V$ , $\;\langle u,v\rangle = \sum_i \langle u,e_i\rangle\langle v,e_i\rangle$ .

Proof. Since the $e_i$ form a basis, $v = \sum_i c_i e_i$ for some scalars. Pair with $e_j$ :

\langle v, e_j\rangle = \sum_i c_i \langle e_i,e_j\rangle = c_j,

which is the first formula. For the norm, apply the Pythagorean theorem to the pairwise orthogonal vectors $c_ie_i$ : $\|v\|^2 = \sum_i \|c_ie_i\|^2 = \sum_i c_i^{2}$ since $\|e_i\| = 1$ . For the last identity, write $u = \sum_i \langle u,e_i\rangle e_i$ and pair with $v$ using bilinearity.∎

The third formula deserves a name: it says that the coordinate map $v \mapsto (\langle v,e_1\rangle,\dots,\langle v,e_n\rangle)$ carries the inner product of $V$ to the standard dot product on $\mathbb{R}^n$ . Every $n$ -dimensional real inner product space is, in this precise sense, a copy of $\mathbb{R}^n$ with the dot product. The abstraction costs nothing in finite dimensions and buys everything in infinite ones, where no finite basis exists.

Intuition. An orthonormal basis is a set of independent measuring instruments. To know a vector you interrogate it once per instrument — "how much $e_1$ are you?", "how much $e_2$ ? " — and the answers never interfere. With a skewed basis, by contrast, the coefficient of $e_1$ depends on all the other basis vectors at once, and you must solve a system to get it. That is the whole practical difference.

Example 10.29 (Coordinates against an orthonormal basis of $\mathbb{R}^2$ ). Let $e_1 = \tfrac{1}{\sqrt5}(2,1)$ and $e_2 = \tfrac{1}{\sqrt5}(-1,2)$ . Check orthonormality and express $v = (4,3)$ in this basis.

Solution. $\langle e_1,e_2\rangle = \tfrac15(2\cdot(-1) + 1\cdot 2) = 0$ , and $\|e_1\|^2 = \tfrac15(4+1) = 1$ , likewise $\|e_2\| = 1$ . So the set is orthonormal, hence a basis of $\mathbb{R}^2$ .

\langle v,e_1\rangle = \frac{8+3}{\sqrt5} = \frac{11}{\sqrt5}, \qquad \langle v,e_2\rangle = \frac{-4+6}{\sqrt5} = \frac{2}{\sqrt5}.

So $v = \frac{11}{\sqrt5}e_1 + \frac{2}{\sqrt5}e_2$ . Check with the norm formula: $\frac{121}{5} + \frac{4}{5} = \frac{125}{5} = 25 = 4^2 + 3^2$ . Correct.□

Example 10.30 (An orthonormal basis of a matrix space). In $M_2(\mathbb{R})$ with the trace inner product, show that the four matrix units $E_{11},E_{12},E_{21},E_{22}$ form an orthonormal basis and find the coordinates of $A = \begin{pmatrix}5&-2\\1&7\end{pmatrix}$ .

Solution. Since $\langle A,B\rangle = \sum_{i,j}a_{ij}b_{ij}$ , we get $\langle E_{ij},E_{k\ell}\rangle = 1$ if $(i,j) = (k,\ell)$ and $0$ otherwise — exactly orthonormality. Four orthonormal vectors in a $4$ -dimensional space form a basis.

The coordinates are the entries themselves: $5, -2, 1, 7$ . Then $\|A\|^2 = 25 + 4 + 1 + 49 = 79$ , so $\|A\| = \sqrt{79}$ , matching the Frobenius norm.□

Example 10.31 (Orthonormal functions). Show that $\{1/\sqrt{2\pi},\; \cos t/\sqrt{\pi},\; \sin t/\sqrt{\pi}\}$ is orthonormal in $C[-\pi,\pi]$ with $\langle f,g\rangle = \int_{-\pi}^{\pi} fg$ .

Solution. Three norms and three cross terms. First $\int_{-\pi}^{\pi} 1\,dt = 2\pi$ , so $\|1/\sqrt{2\pi}\| = 1$ . Next, using $\cos^{2} t = \tfrac12(1+\cos 2t)$ ,

\int_{-\pi}^{\pi}\cos^2 t\,dt = \pi, \qquad \text{and similarly} \qquad \int_{-\pi}^{\pi}\sin^2 t\,dt = \pi,

so the other two are unit vectors. For the cross terms: $\int_{-\pi}^{\pi}\cos t\,dt = 0$ and $\int_{-\pi}^{\pi}\sin t\,dt = 0$ (both integrands integrate to zero over a full period), and $\int_{-\pi}^{\pi}\sin t\cos t\,dt = \tfrac12\int_{-\pi}^{\pi}\sin 2t\,dt = 0$ , the integrand being odd. All three pairs are orthogonal, so the set is orthonormal.□

Pitfall. The formula $v = \sum \langle v,e_i\rangle e_i$ is only valid for an orthonormal basis. For a merely orthogonal basis you must divide by the squared norms:

v = \sum_i \frac{\langle v, v_i\rangle}{\|v_i\|^2}\, v_i .

Dropping those denominators is the most common arithmetic slip in this chapter.

Exercises for §10.5 →

10.6Gram–Schmidt in a general inner product space

Orthonormal bases are so useful that one wants to know they always exist. They do, and the proof is an algorithm.

Theorem 10.32 (Gram–Schmidt). Let $v_1,\dots,v_k$ be linearly independent vectors in an inner product space $V$ . Define recursively

w_1 = v_1, \qquad w_m = v_m - \sum_{i=1}^{m-1} \frac{\langle v_m, w_i\rangle}{\|w_i\|^2}\, w_i \quad (m = 2,\dots,k).

Then $w_1,\dots,w_k$ are nonzero, pairwise orthogonal, and

\operatorname{span}\{w_1,\dots,w_m\} = \operatorname{span}\{v_1,\dots,v_m\} \quad\text{for every } m .

Normalising, $e_m = w_m/\|w_m\|$ gives an orthonormal set with the same spans. In particular, every finite-dimensional inner product space has an orthonormal basis.

Proof. Induct on $m$ . For $m = 1$ the claims are trivial: $w_1 = v_1 \ne 0$ and the spans agree.

Suppose $w_1,\dots,w_{m-1}$ are nonzero, pairwise orthogonal, and span the same subspace as $v_1,\dots,v_{m-1}$ — so all the denominators $\|w_i\|^2$ in the formula for $w_m$ are nonzero and the recursion is legal. Fix $j < m$ and pair $w_m$ with $w_j$ :

\langle w_m, w_j\rangle = \langle v_m, w_j\rangle - \sum_{i=1}^{m-1}\frac{\langle v_m,w_i\rangle}{\|w_i\|^2}\langle w_i, w_j\rangle = \langle v_m,w_j\rangle - \frac{\langle v_m,w_j\rangle}{\|w_j\|^2}\,\|w_j\|^2 = 0,

since only the $i = j$ term survives the sum. So $w_m$ is orthogonal to all earlier $w_j$ .

Next, $w_m \ne 0$ . If $w_m$ were $0$ , then $v_m$ would equal a combination of $w_1,\dots,w_{m-1}$ , which by the inductive hypothesis lie in $\operatorname{span}\{v_1,\dots,v_{m-1}\}$ — contradicting the independence of the $v_i$ .

Finally the spans. By construction $w_m \in \operatorname{span}\{v_m, w_1,\dots,w_{m-1}\} \subseteq \operatorname{span}\{v_1,\dots,v_m\}$ , and conversely $v_m = w_m + \sum_i c_i w_i \in \operatorname{span}\{w_1,\dots,w_m\}$ . Combined with the inductive hypothesis, the two spans coincide.

For the last sentence, apply the process to any basis of a finite-dimensional $V$ ; the result is an orthogonal basis, and normalising gives an orthonormal one.∎

The subtracted sum is the projection of $v_m$ onto everything built so far; $w_m$ is what is left over, the genuinely new direction in $v_m$ . Nothing in the argument mentions $\mathbb{R}^n$ , which is why the same three lines produce orthogonal polynomials, orthogonal wavelets, and the $QR$ factorisation.

Intuition. Gram–Schmidt is the act of un-tilting a basis, one vector at a time, without changing what you can reach. Vector $1$ is kept as-is. Vector $2$ has its shadow along vector $1$ removed, so only its perpendicular part survives. Vector $3$ has its shadows along both previous directions removed. At every stage you can still build exactly the same subspaces as before — you have only replaced a leaning scaffold with a square one.

Method 10.33 (Running Gram–Schmidt by hand).

Set $w_1 = v_1$ and compute $\|w_1\|^2$ once; you will reuse it.
For each new $v_m$ , compute $\langle v_m, w_i\rangle$ against every previous $w_i$ .
Subtract $\dfrac{\langle v_m,w_i\rangle}{\|w_i\|^2}w_i$ for each $i < m$ .
Check orthogonality of the new $w_m$ against one earlier $w_i$ before moving on — an error here poisons everything after it.
Normalise only at the end, to keep the arithmetic in fractions rather than surds.

Example 10.34 (Gram–Schmidt in $\mathbb{R}^3$ ). Orthogonalise $v_1 = (1,1,0)$ , $v_2 = (1,0,1)$ , $v_3 = (0,1,1)$ .

Solution. Take $w_1 = (1,1,0)$ , with $\|w_1\|^2 = 2$ .

For $w_2$ : $\langle v_2,w_1\rangle = 1 + 0 + 0 = 1$ , so

w_2 = (1,0,1) - \tfrac12(1,1,0) = \left(\tfrac12, -\tfrac12, 1\right),

with $\|w_2\|^2 = \tfrac14 + \tfrac14 + 1 = \tfrac32$ .

For $w_3$ : $\langle v_3, w_1\rangle = 0 + 1 + 0 = 1$ and $\langle v_3,w_2\rangle = 0 - \tfrac12 + 1 = \tfrac12$ . Hence

w_3 = (0,1,1) - \tfrac12(1,1,0) - \frac{1/2}{3/2}\left(\tfrac12,-\tfrac12,1\right) = (0,1,1) - \left(\tfrac12,\tfrac12,0\right) - \left(\tfrac16,-\tfrac16,\tfrac13\right),

which is $\left(-\tfrac23, \tfrac23, \tfrac23\right)$ .

Check: $\langle w_3,w_1\rangle = -\tfrac23 + \tfrac23 = 0$ and $\langle w_3,w_2\rangle = -\tfrac13 - \tfrac13 + \tfrac23 = 0$ . Good. Normalising, $e_3 = \tfrac{1}{\sqrt3}(-1,1,1)$ .□

Example 10.35 (Gram–Schmidt with a weighted inner product). In $\mathbb{R}^2$ with $\langle x,y\rangle = 2x_1y_1 + 3x_2y_2$ , orthogonalise $v_1 = (1,0)$ , $v_2 = (1,1)$ .

Solution. $w_1 = (1,0)$ and $\|w_1\|^2 = 2\cdot 1 + 3\cdot 0 = 2$ . Then $\langle v_2, w_1\rangle = 2\cdot 1\cdot 1 + 3\cdot 1\cdot 0 = 2$ , so

w_2 = (1,1) - \frac{2}{2}(1,0) = (0,1).

Check: $\langle (1,0),(0,1)\rangle = 0$ , as required. Here the answer happens to coincide with the standard one, but the norms do not: $\|w_1\| = \sqrt2$ and $\|w_2\| = \sqrt3$ , so the orthonormal basis is $\left(\tfrac{1}{\sqrt2},0\right)$ and $\left(0,\tfrac{1}{\sqrt3}\right)$ .□

Example 10.36 (Orthogonalising polynomials on $[0,1]$ ). In $\mathcal{P}_1$ with $\langle p,q\rangle = \int_0^1 pq$ , orthogonalise $1, t$ and normalise.

Solution. $w_1 = 1$ , $\|w_1\|^2 = 1$ . Then $\langle t, 1\rangle = \int_0^1 t\,dt = \tfrac12$ , so $w_2 = t - \tfrac12$ , and

\|w_2\|^2 = \int_0^1\left(t-\tfrac12\right)^2 dt = \frac{1}{12}.

Normalising, the orthonormal basis is $e_1 = 1$ and $e_2 = 2\sqrt3\,(t - \tfrac12) = \sqrt3\,(2t-1)$ . Check: $\int_0^1 3(2t-1)^2 dt = 3\cdot\tfrac13 = 1$ .□

Pitfall. Gram–Schmidt's output depends on the order of the input vectors. Feeding $v_2, v_1, v_3$ gives a genuinely different orthogonal basis of the same space. There is no "the" orthonormal basis, which is why exam questions ask for a normalised vector with a stated sign convention, or for something order-independent like a projection.

Exercises for §10.6 →

10.7Orthogonal polynomials: Legendre and Chebyshev

Run Gram–Schmidt on the monomials $1, t, t^{2}, \dots$ inside $C[a,b]$ and you manufacture a family of polynomials, orthogonal by construction, one of each degree. Different intervals and different weights produce the classical families, and these are among the most used objects in applied mathematics.

Definition 10.37 (Legendre polynomials). The Legendre polynomials $P_0, P_1, P_2, \dots$ are the polynomials obtained by applying Gram–Schmidt to $1, t, t^{2}, \dots$ in $C[-1,1]$ with

\langle p,q\rangle = \int_{-1}^{1} p(t)q(t)\,dt,

and then rescaling each so that $P_n(1) = 1$ .

The rescaling at the end is a convention that fixes the arbitrary constant Gram–Schmidt leaves behind; the directions are what matter, and those are forced.

Before computing, note a symmetry that halves the work. On the symmetric interval $[-1,1]$ , the integral of an odd function is zero. Since $t^m$ is odd exactly when $m$ is odd, $\langle t^j, t^k\rangle = 0$ whenever $j+k$ is odd. Consequently an even monomial is automatically orthogonal to every odd one, and Gram–Schmidt need only subtract terms of matching parity.

Example 10.38 (Legendre polynomials from $\{1,t,t^{2}\}$ ). Apply Gram–Schmidt to $1, t, t^{2}$ in $C[-1,1]$ , and identify $P_0, P_1, P_2$ .

Solution. Step 1. $w_0 = 1$ , with $\|w_0\|^2 = \int_{-1}^1 1\,dt = 2$ .

Step 2. $\langle t, 1\rangle = \int_{-1}^{1} t\,dt = 0$ (odd integrand), so nothing is subtracted:

w_1 = t, \qquad \|w_1\|^2 = \int_{-1}^{1}t^2\,dt = \frac23 .

Step 3. For $t^{2}$ : $\langle t^{2}, 1\rangle = \int_{-1}^{1}t^{2} dt = \tfrac23$ and $\langle t^{2}, t\rangle = \int_{-1}^{1}t^{3}dt = 0$ (odd). So

w_2 = t^2 - \frac{2/3}{2}\cdot 1 - 0 = t^2 - \frac13 .

Check orthogonality: $\int_{-1}^{1}(t^{2} - \tfrac13)\,dt = \tfrac23 - \tfrac23 = 0$ . Also

\|w_2\|^2 = \int_{-1}^{1}\left(t^4 - \tfrac23 t^2 + \tfrac19\right)dt = \frac25 - \frac49 + \frac29 = \frac25 - \frac29 = \frac{8}{45}.

Normalisation to the Legendre convention. We need $P_n(1) = 1$ . Here $w_0(1) = 1$ , $w_1(1) = 1$ , and $w_2(1) = \tfrac23$ , so multiply the last by $\tfrac32$ :

P_0(t) = 1, \qquad P_1(t) = t, \qquad P_2(t) = \frac{3t^2 - 1}{2}.

These are the standard Legendre polynomials. Continuing the process gives $P_3(t) = \tfrac12(5t^{3} - 3t)$ .□

Theorem 10.39 (Orthogonal polynomials have distinct real roots in the interval). Let $p$ be a nonzero polynomial of degree $n \ge 1$ that is orthogonal in $C[a,b]$ to every polynomial of degree less than $n$ . Then $p$ has $n$ distinct roots, all lying in the open interval $(a,b)$ .

Proof. Let $x_1,\dots,x_m$ be the points of $(a,b)$ at which $p$ changes sign, and set $q(t) = (t-x_1)\cdots(t-x_m)$ , with $q = 1$ if $m = 0$ . The product $p\,q$ does not change sign anywhere on $[a,b]$ — at each $x_i$ both factors flip — and it is not identically zero, so

\left|\int_a^b p(t)q(t)\,dt\right| > 0 .

If $m < n$ then $\deg q < n$ , and orthogonality forces $\int_a^b pq = 0$ , a contradiction. Hence $m \ge n$ . But $p$ has degree $n$ , so it has at most $n$ roots; therefore $m = n$ , the sign changes account for all the roots, and each is simple and lies in $(a,b)$ .∎

That theorem is why Gaussian quadrature works: the roots of $P_n$ are the sample points that integrate polynomials of degree up to $2n-1$ exactly. The roots of $P_2(t) = (3t^{2}-1)/2$ are $t = \pm 1/\sqrt3$ , and the two-point rule $\int_{-1}^1 f \approx f(-1/\sqrt3) + f(1/\sqrt3)$ is exact for every cubic.

Definition 10.40 (Chebyshev polynomials). The Chebyshev polynomials $T_n$ are orthogonal in $C[-1,1]$ with respect to the weighted inner product

\langle p, q\rangle_w = \int_{-1}^{1} \frac{p(t)q(t)}{\sqrt{1-t^2}}\,dt,

and satisfy $T_n(\cos\theta) = \cos(n\theta)$ , so $T_0 = 1$ , $T_1 = t$ , $T_2 = 2t^{2} - 1$ , $T_3 = 4t^{3} - 3t$ .

Example 10.41 (Chebyshev orthogonality by substitution). Verify that $T_1$ and $T_2$ are orthogonal in the Chebyshev inner product.

Solution. Substitute $t = \cos\theta$ , so $dt = -\sin\theta\,d\theta$ and $\sqrt{1-t^{2}} = \sin\theta$ for $\theta \in [0,\pi]$ . The weight and the Jacobian cancel exactly:

\int_{-1}^{1}\frac{T_m(t)T_n(t)}{\sqrt{1-t^2}}\,dt = \int_0^{\pi} \cos(m\theta)\cos(n\theta)\,d\theta .

For $m = 1, n = 2$ use $\cos A\cos B = \tfrac12[\cos(A-B) + \cos(A+B)]$ :

\frac12\int_0^\pi\big(\cos\theta + \cos 3\theta\big)\,d\theta = \frac12\left(\big[\sin\theta\big]_0^\pi + \tfrac13\big[\sin 3\theta\big]_0^\pi\right) = 0 .

So $T_1 \perp T_2$ . The same computation gives $\langle T_m,T_n\rangle_w = 0$ for all $m \ne n$ , and $\|T_0\|_w^{2} = \pi$ , $\|T_n\|_w^{2} = \pi/2$ for $n \ge 1$ .□

Remark. Changing the weight changes the geometry, and therefore changes which polynomials count as perpendicular. Weight $1$ on $[-1,1]$ gives Legendre; weight $1/\sqrt{1-t^{2}}$ gives Chebyshev; weight $e^{-t}$ on $[0,\infty)$ gives Laguerre; weight $e^{-t^{2}}$ on $\mathbb{R}$ gives Hermite. Each family is the "right" basis for a different approximation problem, and all four come out of the same three lines of Gram–Schmidt.

Exercises for §10.7 →

10.8Orthogonal complements

Definition 10.42 (Orthogonal complement). Let $S$ be any subset of an inner product space $V$ . Its orthogonal complement is

S^{\perp} = \{\, v \in V : \langle v, s\rangle = 0 \text{ for every } s \in S \,\}.

Note that $S$ need not be a subspace — it can be a single vector or a random collection — but $S^{\perp}$ always is.

Proposition 10.43 ( $S^\perp$ is a subspace, and only the span matters). For any $S \subseteq V$ , the set $S^\perp$ is a subspace of $V$ , and $S^\perp = (\operatorname{span} S)^\perp$ .

Proof. The zero vector is orthogonal to everything, so $0 \in S^\perp$ . If $u, v \in S^\perp$ and $c$ is a scalar, then for every $s \in S$ ,

\langle u + cv, s\rangle = \langle u,s\rangle + c\langle v,s\rangle = 0 + 0 = 0,

so $u + cv \in S^\perp$ : it is a subspace.

For the second claim, $\operatorname{span}S \supseteq S$ gives $(\operatorname{span}S)^\perp \subseteq S^\perp$ . Conversely if $v \perp s$ for all $s \in S$ and $x = \sum c_i s_i$ is any combination, then $\langle v, x\rangle = \sum c_i\langle v,s_i\rangle = 0$ , so $v \in (\operatorname{span}S)^\perp$ .∎

The practical content of that last sentence: to check membership in $W^\perp$ you only test against a spanning set — a basis, or the rows of a matrix — never against all of $W$ .

Proposition 10.44 (Trivial intersection). If $W$ is a subspace then $W \cap W^\perp = \{0\}$ .

Proof. If $v$ lies in both, then $v$ is orthogonal to itself: $\langle v,v\rangle = 0$ . Positive definiteness forces $v = 0$ .∎

Example 10.45 (A complement in $\mathbb{R}^4$ ). Let $W = \operatorname{span}\{(1,1,0,0),(0,1,1,0)\} \subseteq \mathbb{R}^4$ . Find $\dim W^\perp$ and a basis for it.

Solution. A vector $x$ lies in $W^\perp$ exactly when it is orthogonal to both spanning vectors:

x_1 + x_2 = 0, \qquad x_2 + x_3 = 0 .

This is a homogeneous system with matrix $\begin{pmatrix}1&1&0&0\\0&1&1&0\end{pmatrix}$ , of rank $2$ , so the solution space has dimension $4 - 2 = 2$ . Taking $x_3 = s$ and $x_4 = u$ as free variables gives $x_2 = -s$ and $x_1 = s$ , so

x = s(1,-1,1,0) + u(0,0,0,1).

A basis is $\{(1,-1,1,0),(0,0,0,1)\}$ , and $\dim W^\perp = 2$ . Note $\dim W + \dim W^\perp = 2 + 2 = 4 = \dim \mathbb{R}^4$ , which the next theorem explains.□

Theorem 10.46 (Orthogonal decomposition). Let $W$ be a finite-dimensional subspace of an inner product space $V$ . Then

V = W \oplus W^{\perp},

meaning every $v \in V$ can be written uniquely as $v = w + z$ with $w \in W$ and $z \in W^\perp$ . If $V$ itself is finite-dimensional, $\dim W + \dim W^\perp = \dim V$ .

Proof. Existence. By Gram–Schmidt, $W$ has an orthonormal basis $e_1,\dots,e_k$ . Given $v \in V$ , set

w = \sum_{i=1}^{k}\langle v, e_i\rangle e_i \in W, \qquad z = v - w .

To see $z \in W^\perp$ , pair with each $e_j$ :

\langle z, e_j\rangle = \langle v,e_j\rangle - \sum_{i}\langle v,e_i\rangle\langle e_i,e_j\rangle = \langle v,e_j\rangle - \langle v,e_j\rangle = 0 .

So $z$ is orthogonal to a basis of $W$ , hence to all of $W$ by the previous proposition. Thus $v = w + z$ as required.

Uniqueness. If $w + z = w' + z'$ with $w,w' \in W$ and $z,z' \in W^\perp$ , then $w - w' = z' - z$ lies in $W \cap W^\perp = \{0\}$ , so $w = w'$ and $z = z'$ .

Dimensions. Concatenating a basis of $W$ with a basis of $W^\perp$ gives a spanning set of $V$ by existence, and it is independent because a dependence would produce a nonzero vector of $W \cap W^\perp$ . Counting gives $\dim W + \dim W^\perp = \dim V$ .∎

Corollary 10.47 (Double complement). If $W$ is a finite-dimensional subspace of $V$ , then $(W^\perp)^\perp = W$ .

Proof. If $w \in W$ then $w$ is orthogonal to every element of $W^\perp$ by definition, so $W \subseteq (W^\perp)^\perp$ .

Conversely take $v \in (W^\perp)^\perp$ . By the decomposition theorem write $v = w + z$ with $w \in W$ , $z \in W^\perp$ . Then $z = v - w$ is a difference of two vectors of $(W^\perp)^\perp$ (using $W \subseteq (W^\perp)^\perp$ ), so $z \in (W^\perp)^\perp$ . But $z \in W^\perp$ as well, so $z$ lies in $W^\perp \cap (W^\perp)^\perp = \{0\}$ . Hence $v = w \in W$ .∎

Pitfall. $(W^\perp)^\perp = W$ can fail in infinite dimensions if $W$ is not closed. In $C[0,1]$ the subspace of polynomials has orthogonal complement $\{0\}$ — a continuous function orthogonal to every polynomial is zero, by the Weierstrass approximation theorem — so $(W^\perp)^\perp$ is all of $C[0,1]$ , strictly larger than $W$ . The finite-dimensionality hypothesis is not decoration.

Intuition. Picture a plane $W$ through the origin in $\mathbb{R}^3$ . Its complement $W^\perp$ is the single line perpendicular to it. Every vector in space splits, in exactly one way, into "the part lying in the plane" plus "the part sticking straight out of it" — a shadow and a height. The decomposition theorem says this happens in every inner product space, with subspaces of any dimension, including spaces of functions where "sticking out" has no visual meaning at all.

Example 10.48 (The four fundamental subspaces, in one line). For a real $m \times n$ matrix $A$ , show that $(\operatorname{Row} A)^\perp = \operatorname{Nul} A$ and $(\operatorname{Col} A)^\perp = \operatorname{Nul} A^{\mathsf T}$ .

Solution. A vector $x \in \mathbb{R}^n$ satisfies $Ax = 0$ exactly when every row of $A$ dots to zero with $x$ — that is, exactly when $x$ is orthogonal to every row, hence to their span. So $\operatorname{Nul}A = (\operatorname{Row}A)^\perp$ .

Applying this to $A^{\mathsf T}$ , whose rows are the columns of $A$ , gives $\operatorname{Nul}A^{\mathsf T} = (\operatorname{Col}A)^\perp$ .

Combining with the decomposition theorem recovers the rank–nullity theorem in geometric form: $\operatorname{rank}A + \dim\operatorname{Nul}A = n$ , because $\dim\operatorname{Row}A = \operatorname{rank}A$ .□

Example 10.49 (Complement of a line in function space). In $\mathcal{P}_2$ with $\langle p,q\rangle = \int_{-1}^{1}pq$ , find a basis of $W^\perp$ where $W = \operatorname{span}\{1\}$ .

Solution. A polynomial $p(t) = a + bt + ct^{2}$ lies in $W^\perp$ iff

\int_{-1}^{1}(a + bt + ct^2)\,dt = 2a + 0 + \frac{2c}{3} = 0,

i.e. iff $a = -c/3$ . So $p = c\left(t^{2} - \tfrac13\right) + bt$ , and a basis is $\left\{t,\; t^{2} - \tfrac13\right\}$ — the Legendre directions $P_1$ and $P_2$ , as expected. Here $\dim W + \dim W^\perp = 1 + 2 = 3 = \dim\mathcal{P}_2$ .□

Exercises for §10.8 →

10.9Orthogonal projection and best approximation

The $w$ produced by the decomposition theorem deserves its own name, because it answers an optimisation problem.

Definition 10.50 (Orthogonal projection onto a subspace). Let $W$ be a finite-dimensional subspace with orthonormal basis $e_1,\dots,e_k$ . The orthogonal projection of $v$ onto $W$ is

\operatorname{proj}_W v = \sum_{i=1}^{k}\langle v, e_i\rangle\,e_i .

With a merely orthogonal basis $w_1,\dots,w_k$ the formula reads $\operatorname{proj}_W v = \sum_i \dfrac{\langle v,w_i\rangle}{\|w_i\|^2}\,w_i$ .

The definition looks basis-dependent, but it is not: by the uniqueness half of the decomposition theorem, $\operatorname{proj}_W v$ is the component of $v$ in $W$ , and any orthonormal basis of $W$ produces it. That is worth remembering when a question asks for a projection but not for a basis — the answer is order-independent even though Gram–Schmidt's output is not.

Theorem 10.51 (Best approximation). Let $W$ be a finite-dimensional subspace of $V$ and $v \in V$ . Then for every $w \in W$ with $w \ne \operatorname{proj}_W v$ ,

\|v - \operatorname{proj}_W v\| < \|v - w\| .

So $\operatorname{proj}_W v$ is the unique closest point of $W$ to $v$ .

Proof. Write $\hat v = \operatorname{proj}_W v$ and let $w \in W$ . Split

v - w = (v - \hat v) + (\hat v - w).

The first summand lies in $W^\perp$ and the second lies in $W$ , so they are orthogonal, and the Pythagorean theorem gives

\|v - w\|^2 = \|v - \hat v\|^2 + \|\hat v - w\|^2 .

The last term is strictly positive unless $w = \hat v$ . Hence $\|v-w\|^2 > \|v - \hat v\|^2$ for every other $w$ , and taking square roots finishes the proof.∎

That single computation is the reason projections matter in every applied field: least squares, regression, filtering, data compression and numerical approximation are all "project onto the subspace of things you can build". The error $v - \hat v$ is orthogonal to the model space — the residual contains nothing the model could have captured.

Intuition. Stand a lamp directly above a table. A pencil held in the air casts a shadow on the tabletop, and of all the points on the table, the tip of the shadow is the one nearest the pencil tip. Why? Because the line from pencil tip to shadow tip is vertical — orthogonal to the table — and any other route to the table is the hypotenuse of a right triangle with that vertical as one leg. The best-approximation theorem is this observation, with "table" replaced by any subspace.

Example 10.52 (Projection onto a plane in $\mathbb{R}^3$ ). Project $v = (1,2,3)$ onto $W = \operatorname{span}\{(1,1,0),(1,0,1)\}$ and find the distance from $v$ to $W$ .

Solution. The spanning vectors are not orthogonal, so orthogonalise first. Take $w_1 = (1,1,0)$ , $\|w_1\|^2 = 2$ ; then $\langle (1,0,1),w_1\rangle = 1$ , so

w_2 = (1,0,1) - \tfrac12(1,1,0) = \left(\tfrac12,-\tfrac12,1\right), \qquad \|w_2\|^2 = \tfrac32 .

Now project: $\langle v,w_1\rangle = 1+2 = 3$ and $\langle v,w_2\rangle = \tfrac12 - 1 + 3 = \tfrac52$ . So

\operatorname{proj}_W v = \frac{3}{2}(1,1,0) + \frac{5/2}{3/2}\left(\tfrac12,-\tfrac12,1\right) = \left(\tfrac32,\tfrac32,0\right) + \left(\tfrac56,-\tfrac56,\tfrac53\right) = \left(\tfrac73,\tfrac23,\tfrac53\right).

The residual is $v - \operatorname{proj}_W v = \left(-\tfrac43,\tfrac43,\tfrac43\right)$ , with norm

\sqrt{3\cdot\frac{16}{9}} = \frac{4}{\sqrt3}.

Sanity check: the residual should be orthogonal to both spanning vectors. $\left(-\tfrac43,\tfrac43,\tfrac43\right)\cdot(1,1,0) = 0$ and $\cdot\,(1,0,1) = -\tfrac43 + \tfrac43 = 0$ . Correct.□

Example 10.53 (Least-squares line fit to a function). Find the polynomial of degree at most $1$ that best approximates $f(t) = e^{t}$ on $[-1,1]$ in the least-squares sense, i.e. minimising $\int_{-1}^{1}\big(e^{t} - p(t)\big)^2dt$ .

Solution. The subspace is $W = \mathcal{P}_1 = \operatorname{span}\{1, t\}$ , and on $[-1,1]$ these two are already orthogonal: $\int_{-1}^{1}t\,dt = 0$ . Their squared norms are $\|1\|^2 = 2$ and $\|t\|^2 = \tfrac23$ .

Two integrals are needed. First

\langle e^t, 1\rangle = \int_{-1}^{1}e^t\,dt = e - e^{-1}.

Second, by parts with $u = t$ , $dv = e^t dt$ :

\langle e^t, t\rangle = \int_{-1}^{1} te^t\,dt = \Big[(t-1)e^t\Big]_{-1}^{1} = 0 - (-2e^{-1}) = \frac{2}{e}.

Therefore

\operatorname{proj}_W f = \frac{e - e^{-1}}{2}\cdot 1 + \frac{2/e}{2/3}\cdot t = \frac{e - e^{-1}}{2} + \frac{3}{e}\,t .

Numerically that is $\approx 1.1752 + 1.1036\,t$ . Sanity check: the constant term is the average value of $e^t$ over $[-1,1]$ , namely $\sinh 1 \approx 1.175$ , which is what a mean-square fit should give for the constant when the basis is orthogonal. Note that the fitted line is not the tangent line at $0$ , which would be $1 + t$ : least squares balances error across the whole interval rather than matching perfectly at one point.□

Example 10.54 (Best linear approximation to a cubic). Find the best approximation to $f(t) = t^{3}$ in $\mathcal{P}_1$ on $[-1,1]$ , and compute the resulting minimum error $\|f - \hat f\|$ .

Solution. Again use the orthogonal basis $\{1,t\}$ . By parity, $\langle t^{3}, 1\rangle = \int_{-1}^{1}t^{3}dt = 0$ , and

\langle t^3, t\rangle = \int_{-1}^{1}t^4\,dt = \frac25 .

\hat f = \frac{2/5}{2/3}\,t = \frac35\,t .

For the error, use the Pythagorean split $\|f\|^2 = \|\hat f\|^2 + \|f - \hat f\|^2$ . Here $\|f\|^2 = \int_{-1}^{1}t^{6}dt = \tfrac27$ and $\|\hat f\|^2 = \tfrac{9}{25}\cdot\tfrac23 = \tfrac{6}{25}$ , so

\|f-\hat f\|^2 = \frac27 - \frac{6}{25} = \frac{50 - 42}{175} = \frac{8}{175},

giving $\|f - \hat f\| = \sqrt{8/175} \approx 0.2138$ . Sanity check: the error must be smaller than $\|f\| = \sqrt{2/7} \approx 0.535$ , and it is.□

Pitfall. Project onto an orthogonal basis or not at all. Writing $\operatorname{proj}_W v = \sum_i \frac{\langle v,v_i\rangle}{\|v_i\|^2}v_i$ for a basis $\{v_i\}$ that has not been orthogonalised gives a vector in $W$ , but not the nearest one, and the residual will not be orthogonal to $W$ . If in doubt, verify at the end that $v - \hat v$ is orthogonal to every basis vector of $W$ — that check is both necessary and sufficient.

Exercises for §10.9 →

10.10Bessel's inequality and Parseval's identity

Projecting onto a subspace can only shorten a vector, and saying so quantitatively gives an inequality that controls every approximation scheme built on orthonormal sets.

Theorem 10.55 (Bessel's inequality). Let $e_1,\dots,e_k$ be an orthonormal set in an inner product space $V$ (not necessarily a basis). Then for every $v \in V$ ,

\sum_{i=1}^{k}\langle v, e_i\rangle^2 \;\le\; \|v\|^2 .

Proof. Let $W = \operatorname{span}\{e_1,\dots,e_k\}$ and $\hat v = \operatorname{proj}_W v = \sum_i \langle v,e_i\rangle e_i$ . Then $v = \hat v + (v - \hat v)$ with the two pieces orthogonal, so by the Pythagorean theorem

\|v\|^2 = \|\hat v\|^2 + \|v - \hat v\|^2 \ge \|\hat v\|^2 .

By the coordinate formula, $\|\hat v\|^2 = \sum_i \langle v,e_i\rangle^{2}$ , which gives the inequality. Equality holds precisely when $\|v - \hat v\| = 0$ , i.e. when $v \in W$ .∎

The proof also shows the deficit exactly: $\|v\|^2 - \sum_i\langle v,e_i\rangle^{2}$ is the squared distance from $v$ to the span. So Bessel is a bookkeeping statement — the coefficients account for part of the energy of $v$ , never more.

Corollary 10.56 (Parseval's identity). If $\{e_1,\dots,e_n\}$ is an orthonormal basis of $V$ , then for all $u,v \in V$

\|v\|^2 = \sum_{i=1}^{n}\langle v,e_i\rangle^2, \qquad \langle u,v\rangle = \sum_{i=1}^{n}\langle u,e_i\rangle\langle v,e_i\rangle .

Proof. When the orthonormal set is a basis, $W = V$ , so $v = \hat v$ and Bessel's inequality becomes an equality — that is the first formula (already obtained in the coordinates theorem). The second follows by expanding $u = \sum_i\langle u,e_i\rangle e_i$ and pairing with $v$ .∎

Intuition. Think of $\langle v,e_i\rangle^{2}$ as the energy that $v$ carries in channel $i$ . Bessel says you can never extract more energy than the signal contains: a partial set of channels captures part of it. Parseval says that a complete set of channels loses nothing — every bit of the signal shows up somewhere. That is exactly the claim an audio engineer makes when asserting that a spectrum contains the same information as the waveform.

Example 10.57 (Bessel with an incomplete set). In $\mathbb{R}^3$ let $e_1 = \tfrac{1}{\sqrt2}(1,1,0)$ and $e_2 = \tfrac{1}{\sqrt2}(1,-1,0)$ . Verify Bessel's inequality for $v = (2,0,5)$ and identify the deficit.

Solution. $\langle v,e_1\rangle = \tfrac{2}{\sqrt2} = \sqrt2$ and $\langle v,e_2\rangle = \tfrac{2}{\sqrt2} = \sqrt2$ . So the coefficient sum is $2 + 2 = 4$ , while $\|v\|^2 = 4 + 0 + 25 = 29$ . Indeed $4 \le 29$ .

The deficit is $29 - 4 = 25$ , which is the squared distance from $v$ to the $xy$ -plane spanned by $e_1,e_2$ — namely $5^2$ . Adding $e_3 = (0,0,1)$ contributes $\langle v,e_3\rangle^{2} = 25$ and turns Bessel into Parseval.□

Example 10.58 (Parseval in a rotated basis). Let $e_1 = \tfrac{1}{\sqrt5}(1,2)$ , $e_2 = \tfrac{1}{\sqrt5}(2,-1)$ and $v = (3,4)$ . Confirm Parseval's identity.

Solution. $\langle v,e_1\rangle = \tfrac{3+8}{\sqrt5} = \tfrac{11}{\sqrt5}$ , $\langle v,e_2\rangle = \tfrac{6-4}{\sqrt5} = \tfrac{2}{\sqrt5}$ . Then

\frac{121}{5} + \frac{4}{5} = \frac{125}{5} = 25 = \|v\|^2 .

Parseval holds, as it must for an orthonormal basis: the norm does not care which orthonormal ruler you measure with.□

Example 10.59 (Bessel in function space). In $C[-1,1]$ with $\langle f,g\rangle = \int_{-1}^1 fg$ , take the orthonormal set $e_0 = 1/\sqrt2$ , $e_1 = \sqrt{3/2}\,t$ . Compute the Bessel sum for $f(t) = t^{2}$ and compare with $\|f\|^2$ .

Solution. First normalise correctly: $\|1\|^2 = 2$ so $e_0 = 1/\sqrt2$ ; $\|t\|^2 = 2/3$ so $e_1 = t/\sqrt{2/3} = \sqrt{3/2}\,t$ .

Now $\langle t^{2}, e_0\rangle = \tfrac{1}{\sqrt2}\int_{-1}^{1}t^{2}dt = \tfrac{1}{\sqrt2}\cdot\tfrac23 = \tfrac{2}{3\sqrt2}$ , and $\langle t^{2},e_1\rangle = \sqrt{3/2}\int_{-1}^1 t^{3}dt = 0$ by parity.

So the Bessel sum is $\tfrac{4}{18} = \tfrac29$ , while $\|f\|^2 = \int_{-1}^1 t^{4}dt = \tfrac25$ . Since $\tfrac29 \approx 0.222 < 0.4$ , the inequality holds. The deficit $\tfrac25 - \tfrac29 = \tfrac{8}{45}$ is exactly $\|P_2\|^2$ computed earlier — the missing energy of $t^{2}$ is its Legendre- $P_2$ component, as it should be.□

Pitfall. Bessel's inequality does not need the orthonormal set to be a basis, but it does need it to be orthonormal. With a merely orthogonal set $\{w_i\}$ the correct statement is $\sum_i \langle v,w_i\rangle^{2}/\|w_i\|^2 \le \|v\|^2$ . Forgetting the denominators can make the left side exceed the right and look like a counterexample to a true theorem.

Exercises for §10.10 →

10.11Fourier series as orthogonal projection

Everything in the last three sections applies verbatim when the subspace is spanned by sines and cosines, and what comes out is the classical Fourier series. This is the single best advertisement for the abstract point of view: an entire branch of analysis is the best-approximation theorem in a particular inner product space.

Work in $C[-\pi,\pi]$ with

\langle f, g\rangle = \int_{-\pi}^{\pi} f(t)g(t)\,dt .

Proposition 10.60 (The trigonometric system is orthogonal). The functions $1, \cos t, \sin t, \cos 2t, \sin 2t, \dots$ are pairwise orthogonal in $C[-\pi,\pi]$ , with

\|1\|^2 = 2\pi, \qquad \|\cos mt\|^2 = \|\sin mt\|^2 = \pi \quad (m \ge 1).

Proof. Use the product-to-sum identities. For $m \ne n$ ,

\int_{-\pi}^{\pi}\cos mt\cos nt\,dt = \frac12\int_{-\pi}^{\pi}\big[\cos(m-n)t + \cos(m+n)t\big]dt = 0,

since $\int_{-\pi}^{\pi}\cos kt\,dt = 0$ for every nonzero integer $k$ . The same identity with $\sin$ gives orthogonality of distinct sines. For a sine against a cosine,

\int_{-\pi}^{\pi}\sin mt\cos nt\,dt = 0

because the integrand is odd. Orthogonality to the constant $1$ is the case $n = 0$ . For the norms, $\cos^{2} mt = \tfrac12(1+\cos 2mt)$ integrates to $\pi$ over $[-\pi,\pi]$ , and likewise $\sin^{2} mt = \tfrac12(1-\cos 2mt)$ .∎

Definition 10.61 (Fourier coefficients and the $N$ -th partial sum). Let $W_N = \operatorname{span}\{1, \cos t, \sin t, \dots, \cos Nt, \sin Nt\}$ . For $f \in C[-\pi,\pi]$ , the Fourier coefficients are

a_m = \frac{1}{\pi}\int_{-\pi}^{\pi} f(t)\cos mt\,dt \;\;(m \ge 0), \qquad b_m = \frac{1}{\pi}\int_{-\pi}^{\pi} f(t)\sin mt\,dt \;\;(m \ge 1),

and the $N$ -th Fourier partial sum is

S_N f(t) = \frac{a_0}{2} + \sum_{m=1}^{N}\big(a_m\cos mt + b_m\sin mt\big).

Theorem 10.62 (Fourier partial sums are orthogonal projections). $S_N f = \operatorname{proj}_{W_N} f$ . Consequently $S_N f$ is the unique element of $W_N$ minimising $\int_{-\pi}^{\pi}\big(f(t) - g(t)\big)^2 dt$ over $g \in W_N$ .

Proof. The trigonometric system is an orthogonal basis of $W_N$ , so the projection formula gives

\operatorname{proj}_{W_N} f = \frac{\langle f,1\rangle}{\|1\|^2}\cdot 1 + \sum_{m=1}^{N}\left(\frac{\langle f,\cos mt\rangle}{\|\cos mt\|^2}\cos mt + \frac{\langle f,\sin mt\rangle}{\|\sin mt\|^2}\sin mt\right).

Substituting the norms from the previous proposition, the constant term is $\frac{1}{2\pi}\int_{-\pi}^{\pi}f = \frac{a_0}{2}$ — which is why the definition carries that awkward factor of one half — and the $m$ -th coefficients are $\frac1\pi\int f\cos mt = a_m$ and $\frac1\pi\int f\sin mt = b_m$ . That is exactly $S_Nf$ . The minimising property is the best-approximation theorem.∎

Intuition. A Fourier coefficient is the answer to one question: "how much pure tone of frequency $m$ is in this signal?" You get the answer by multiplying the signal against that tone and integrating — the tone resonates with its own component and averages to nothing against every other. Truncating at $N$ is not an approximation you chose for convenience; it is provably the closest that any combination of the first $N$ harmonics can come.

Example 10.63 (The Fourier series of $f(t) = t$ ). Compute the Fourier coefficients of $f(t) = t$ on $[-\pi,\pi]$ .

Solution. Parity does most of the work. $f$ is odd and $\cos mt$ is even, so every product $t\cos mt$ is odd and

a_m = \frac1\pi\int_{-\pi}^{\pi} t\cos mt\,dt = 0 \quad\text{for all } m \ge 0 .

For the sine coefficients, $t\sin mt$ is even, so integrate over half the range and double. Integrating by parts with $u = t$ , $dv = \sin mt\,dt$ :

\int_0^{\pi} t\sin mt\,dt = \left[-\frac{t\cos mt}{m}\right]_0^{\pi} + \frac1m\int_0^{\pi}\cos mt\,dt = -\frac{\pi\cos m\pi}{m} + 0 .

Since $\cos m\pi = (-1)^m$ , this is $\pi(-1)^{m+1}/m$ . Therefore

b_m = \frac{2}{\pi}\cdot\frac{\pi(-1)^{m+1}}{m} = \frac{2(-1)^{m+1}}{m},

and

t \sim 2\sin t - \sin 2t + \frac23\sin 3t - \frac12\sin 4t + \cdots .

Sanity check at $t = \pi/2$ : every even harmonic vanishes there, and the odd ones alternate, so the series collapses to

2\left(1 - \frac13 + \frac15 - \frac17 + \cdots\right) = 2\cdot\frac{\pi}{4} = \frac{\pi}{2},

which is $f(\pi/2)$ , as it should be.□

Example 10.64 (A Fourier coefficient of a step function). Let $f(t) = 1$ for $0 < t < \pi$ and $f(t) = -1$ for $-\pi < t < 0$ . Compute $b_1$ and $b_2$ .

Solution. $f$ is odd, so all $a_m$ vanish and $b_m = \frac2\pi\int_0^\pi \sin mt\,dt$ . Now

\int_0^{\pi}\sin mt\,dt = \left[-\frac{\cos mt}{m}\right]_0^{\pi} = \frac{1 - (-1)^m}{m},

which is $2/m$ for odd $m$ and $0$ for even $m$ . Hence

b_1 = \frac2\pi\cdot 2 = \frac4\pi, \qquad b_2 = 0 .

The square wave contains only odd harmonics — a fact audible in the timbre of a square-wave synthesiser.□

Example 10.65 (Parseval, and the value of $\sum 1/n^{2}$ ). Use Parseval's identity for $f(t) = t$ on $[-\pi,\pi]$ to evaluate $\sum_{n=1}^{\infty}\frac{1}{n^{2}}$ .

Solution. For the trigonometric system, Parseval in its classical form reads

\int_{-\pi}^{\pi}f(t)^2\,dt = \frac{a_0^2}{2}\pi + \pi\sum_{m\ge 1}\big(a_m^2 + b_m^2\big),

which is the orthogonal-basis identity $\|f\|^2 = \sum_i \langle f,u_i\rangle^{2}/\|u_i\|^2$ written out with $\|\cos mt\|^2 = \|\sin mt\|^2 = \pi$ .

Here $a_m = 0$ and $b_m = 2(-1)^{m+1}/m$ , so $b_m^{2} = 4/m^{2}$ . The left side is

\int_{-\pi}^{\pi}t^2\,dt = \frac{2\pi^3}{3}.

Equating,

\frac{2\pi^3}{3} = \pi\sum_{m=1}^{\infty}\frac{4}{m^2} \quad\Longrightarrow\quad \sum_{m=1}^{\infty}\frac{1}{m^2} = \frac{\pi^2}{6}.

Euler's celebrated sum, obtained from the norm of the identity function. Sanity check: $\pi^{2}/6 \approx 1.6449$ , and $1 + \tfrac14 + \tfrac19 + \tfrac1{16} = 1.4236$ is already climbing towards it.□

Pitfall. The $a_0/2$ in the partial sum is not a typo and not a convention you may drop. It arises because $\|1\|^2 = 2\pi$ while $\|\cos mt\|^2 = \pi$ — the constant function is "twice as long" as the harmonics. Defining $a_0$ by the same formula as the other $a_m$ and then writing $a_0$ rather than $a_0/2$ doubles the mean of your approximation.

Exercises for §10.11 →

10.12Complex inner product spaces

Over $\mathbb{C}$ the naive definition collapses. If we demanded symmetry and bilinearity, then for $v \ne 0$ ,

\langle iv, iv\rangle = i^2\langle v,v\rangle = -\langle v,v\rangle,

so $v$ and $iv$ could not both have positive "length". Positive definiteness and bilinearity are incompatible over $\mathbb{C}$ , and the fix — forced, not chosen — is to conjugate.

Definition 10.66 (Complex inner product space). Let $V$ be a vector space over $\mathbb{C}$ . A complex inner product is a map $\langle\,\cdot\,,\cdot\,\rangle : V\times V \to \mathbb{C}$ such that for all $u,v,w \in V$ and $c \in \mathbb{C}$ :

Conjugate symmetry: $\langle v, u\rangle = \overline{\langle u, v\rangle}$ .
Linearity in the first slot: $\langle u + cv, w\rangle = \langle u,w\rangle + c\langle v,w\rangle$ .
Positive definiteness: $\langle v,v\rangle$ is real and $\ge 0$ , with equality only for $v = 0$ .

Conjugate symmetry makes axiom 3 sensible: $\langle v,v\rangle = \overline{\langle v,v\rangle}$ forces $\langle v,v\rangle$ to be real before we can ask it to be positive. The price is that the second slot is only conjugate-linear:

\langle u, cv\rangle = \overline{\langle cv, u\rangle} = \overline{c\langle v,u\rangle} = \bar c\,\overline{\langle v,u\rangle} = \bar c\,\langle u,v\rangle .

A map with these properties is called sesquilinear — "one and a half times linear". Everything else in the chapter survives with $\langle v,v\rangle^{2}$ replaced by $|\langle u,v\rangle|^2$ at the appropriate places.

Example 10.67 (The standard inner product on $\mathbb{C}^n$ ). Define $\langle z, w\rangle = \sum_{k=1}^{n} z_k\overline{w_k}$ . Verify the axioms and compute $\langle z,w\rangle$ and $\|z\|$ for $z = (1+i,\,2)$ , $w = (3,\,i)$ .

Solution. Conjugate symmetry: $\langle w,z\rangle = \sum w_k\overline{z_k} = \overline{\sum \overline{w_k}z_k} = \overline{\langle z,w\rangle}$ . Linearity in the first slot is clear. Positivity:

\langle z,z\rangle = \sum_k z_k\overline{z_k} = \sum_k |z_k|^2 \ge 0,

zero only if every $z_k = 0$ . Note how the conjugate rescues this: without it we would get $\sum z_k^{2}$ , which for $z = (i,0)$ is $-1$ .

For the numbers: $\langle z,w\rangle = (1+i)\overline{3} + 2\overline{i} = 3(1+i) + 2(-i) = 3 + 3i - 2i = 3 + i$ . And

\|z\|^2 = |1+i|^2 + |2|^2 = 2 + 4 = 6, \qquad \|z\| = \sqrt6 .

Check conjugate symmetry numerically: $\langle w,z\rangle = 3\overline{(1+i)} + i\cdot\overline{2} = 3 - 3i + 2i = 3 - i = \overline{3+i}$ . Correct.□

Theorem 10.68 (Complex Cauchy–Schwarz). In a complex inner product space, $|\langle u,v\rangle| \le \|u\|\,\|v\|$ , with equality iff $u,v$ are linearly dependent.

Proof. Assume $v \ne 0$ and put $c = \langle u,v\rangle/\|v\|^2$ , $w = u - cv$ . Then

\langle w, v\rangle = \langle u,v\rangle - c\|v\|^2 = 0,

so $w \perp v$ , and also $\langle v, w\rangle = \overline{\langle w,v\rangle} = 0$ . Expanding $\|u\|^2 = \|w + cv\|^2$ with the sesquilinear rules, the cross terms $\langle w, cv\rangle = \bar c\langle w,v\rangle$ and $\langle cv, w\rangle = c\langle v,w\rangle$ both vanish, leaving

\|u\|^2 = \|w\|^2 + |c|^2\|v\|^2 = \|w\|^2 + \frac{|\langle u,v\rangle|^2}{\|v\|^2}.

Drop $\|w\|^2 \ge 0$ and rearrange. Equality forces $w = 0$ , i.e. $u = cv$ .∎

The triangle inequality follows as before, using $\operatorname{Re}\langle u,v\rangle \le |\langle u,v\rangle|$ in the cross term of

\|u+v\|^2 = \|u\|^2 + 2\operatorname{Re}\langle u,v\rangle + \|v\|^2 .

That expansion also shows why the converse of Pythagoras fails over $\mathbb{C}$ : $\|u+v\|^2 = \|u\|^2+\|v\|^2$ only forces the real part of $\langle u,v\rangle$ to vanish. In $\mathbb{C}$ , the vectors $u = 1$ and $v = i$ satisfy it without being orthogonal.

Definition 10.69 (Conjugate transpose). For a complex matrix $A$ , the conjugate transpose (or adjoint) is $A^{*} = \overline{A}^{\mathsf T}$ , so $(A^*)_{ij} = \overline{a_{ji}}$ .

A matrix is Hermitian (self-adjoint) if $A^* = A$ , skew-Hermitian if $A^* = -A$ , and unitary if $A^*A = I$ .

Proposition 10.70 (The defining property of $A^*$ ). For all $z \in \mathbb{C}^n$ and $w \in \mathbb{C}^m$ , and any $m\times n$ complex matrix $A$ ,

\langle Az, w\rangle = \langle z, A^{*}w\rangle .

Moreover $(AB)^* = B^*A^*$ and $(A^*)^* = A$ .

Proof. Writing the standard inner product as $\langle x,y\rangle = y^{*}x$ (a $1\times1$ matrix),

\langle Az, w\rangle = w^{*}Az = (A^{*}w)^{*}z = \langle z, A^{*}w\rangle,

using $(A^*w)^* = w^*A^{**} = w^*A$ . The product rule follows from $\overline{AB} = \overline A\,\overline B$ and $(AB)^{\mathsf T} = B^{\mathsf T}A^{\mathsf T}$ ; the involution rule from $\overline{\overline A} = A$ .∎

Example 10.71 (Computing a conjugate transpose). Find $A^{*}$ for $A = \begin{pmatrix} 1+2i & 3 \\ -i & 4-i\end{pmatrix}$ , and determine whether $A$ is Hermitian.

Solution. Transpose and conjugate every entry:

A^{\mathsf T} = \begin{pmatrix}1+2i & -i\\ 3 & 4-i\end{pmatrix}, \qquad A^{*} = \begin{pmatrix}1-2i & i\\ 3 & 4+i\end{pmatrix}.

Since $(A^*)_{11} = 1 - 2i \ne 1 + 2i = A_{11}$ , the matrix is not Hermitian. In fact no Hermitian matrix can have a non-real diagonal entry: $a_{kk} = \overline{a_{kk}}$ is forced.□

Proposition 10.72 (Hermitian matrices have real diagonal and real quadratic form). If $A$ is Hermitian then every diagonal entry $a_{kk}$ is real, and $\langle Az, z\rangle$ is real for every $z \in \mathbb{C}^n$ .

Proof. Setting $i = j = k$ in $a_{ij} = \overline{a_{ji}}$ gives $a_{kk} = \overline{a_{kk}}$ , so $a_{kk} \in \mathbb{R}$ . For the quadratic form,

\overline{\langle Az,z\rangle} = \langle z, Az\rangle = \langle A^{*}z, z\rangle = \langle Az, z\rangle,

using conjugate symmetry and then $A^* = A$ . A complex number equal to its own conjugate is real.∎

That proposition is the engine behind one of the headline facts of the next chapter: the eigenvalues of a Hermitian matrix are real, because if $Az = \lambda z$ with $z \ne 0$ then $\langle Az,z\rangle = \lambda\|z\|^2$ is real and $\|z\|^2 > 0$ .

Theorem 10.73 (Characterisations of unitary matrices). For a square complex matrix $U$ the following are equivalent:

$U^{*}U = I$ (that is, $U$ is unitary);
the columns of $U$ form an orthonormal basis of $\mathbb{C}^n$ ;
$\|Uz\| = \|z\|$ for every $z$ ;
$\langle Uz, Uw\rangle = \langle z,w\rangle$ for all $z,w$ .

Proof. $(1)\Leftrightarrow(2)$ : the $(i,j)$ entry of $U^{*}U$ is $\sum_k \overline{u_{ki}}u_{kj} = \langle u_j, u_i\rangle$ where $u_j$ is the $j$ -th column. So $U^*U = I$ says precisely that the columns are orthonormal; $n$ orthonormal vectors in $\mathbb{C}^n$ form a basis.

$(1)\Rightarrow(4)$ : $\langle Uz,Uw\rangle = \langle z, U^{*}Uw\rangle = \langle z,w\rangle$ .

$(4)\Rightarrow(3)$ : take $w = z$ and square-root.

$(3)\Rightarrow(1)$ : $\|Uz\|^2 = \langle U^{*}Uz, z\rangle$ and $\|z\|^2 = \langle z,z\rangle$ , so $\langle (U^{*}U - I)z, z\rangle = 0$ for all $z$ . The matrix $H = U^*U - I$ is Hermitian, and a Hermitian matrix with $\langle Hz,z\rangle \equiv 0$ is zero: taking $z = e_j$ gives $h_{jj} = 0$ , and taking $z = e_j + e_k$ and then $z = e_j + ie_k$ gives $h_{jk} + h_{kj} = 0$ and $i(h_{jk} - h_{kj}) \cdot$ (sign) $= 0$ , whence $h_{jk} = 0$ . So $U^{*}U = I$ .∎

Unitary matrices are the complex rotations: they move vectors without changing any length or any angle. Their real counterparts, with $Q^{\mathsf T}Q = I$ , are the orthogonal matrices.

Example 10.74 (Testing a matrix for unitarity). Is $U = \dfrac{1}{\sqrt2}\begin{pmatrix} 1 & i \\ i & 1\end{pmatrix}$ unitary?

Solution. Compute $U^{*}$ : conjugate and transpose gives $U^{*} = \frac{1}{\sqrt2}\begin{pmatrix}1 & -i\\ -i & 1\end{pmatrix}$ . Then

U^{*}U = \frac12\begin{pmatrix}1 & -i\\ -i & 1\end{pmatrix}\begin{pmatrix}1 & i\\ i & 1\end{pmatrix} = \frac12\begin{pmatrix}1 - i^2 & i - i\\ -i+i & -i^2+1\end{pmatrix} = \frac12\begin{pmatrix}2&0\\0&2\end{pmatrix} = I .

So yes. Equivalently, the columns $\tfrac{1}{\sqrt2}(1,i)$ and $\tfrac{1}{\sqrt2}(i,1)$ each have norm $\tfrac12(1+1) = 1$ and inner product $\tfrac12\big(1\cdot\overline{i} + i\cdot\overline{1}\big) = \tfrac12(-i + i) = 0$ .□

Pitfall. In $\mathbb{C}^n$ the quantity $\sum z_k w_k$ — no conjugate — is a perfectly good bilinear form but it is not the inner product, and it can be zero for a nonzero vector: $(1,i)$ satisfies $\sum z_k^{2} = 1 - 1 = 0$ . Whenever complex entries appear, conjugate the second argument, and remember that $\langle u, cv\rangle = \bar c\langle u,v\rangle$ pulls scalars out of the second slot conjugated.

Exercises for §10.12 →

10.13The adjoint of a linear operator

The conjugate transpose was defined by a formula. The right definition is coordinate-free, and it makes sense in any finite-dimensional inner product space, real or complex.

Definition 10.75 (Adjoint). Let $T : V \to W$ be linear, with $V,W$ finite-dimensional inner product spaces. The adjoint of $T$ is the map $T^{*}: W \to V$ satisfying

\langle T v, w\rangle_W = \langle v, T^{*}w\rangle_V \qquad \text{for all } v \in V,\; w \in W .

Theorem 10.76 (Existence and uniqueness of the adjoint). Every linear $T : V \to W$ between finite-dimensional inner product spaces has exactly one adjoint, and $T^{*}$ is linear.

Proof. Uniqueness. If $S_1$ and $S_2$ both satisfy the defining identity, then $\langle v, S_1w - S_2w\rangle = 0$ for every $v \in V$ . Taking $v = S_1w - S_2w$ gives $\|S_1w - S_2w\|^2 = 0$ , so $S_1w = S_2w$ for every $w$ .

Existence. Choose an orthonormal basis $e_1,\dots,e_n$ of $V$ (Gram–Schmidt supplies one) and define

T^{*}w = \sum_{i=1}^{n}\overline{\langle Te_i, w\rangle}\;e_i .

This is a well-defined element of $V$ . To check the identity it suffices to verify it for $v = e_j$ , since both sides are linear in $v$ (over $\mathbb{R}$ ; over $\mathbb{C}$ , both sides are linear in $v$ as well, the conjugation sitting in the second slot). Using the coordinate formula,

\langle e_j, T^{*}w\rangle = \overline{\overline{\langle Te_j,w\rangle}} = \langle Te_j, w\rangle,

because pairing $e_j$ against the sum picks out the $j$ -th coefficient conjugated. Linearity of $T^*$ follows from uniqueness applied to $\langle Tv, w_1 + cw_2\rangle = \langle v, T^*w_1\rangle + \bar c\,\langle v, T^*w_2\rangle$ rearranged, or directly from the displayed formula.∎

Theorem 10.77 (Matrix of the adjoint). Let $\mathcal{B} = \{e_1,\dots,e_n\}$ be an orthonormal basis of $V$ and $\mathcal{C} = \{f_1,\dots,f_m\}$ an orthonormal basis of $W$ . Then

[T^{*}]_{\mathcal{C}}^{\mathcal{B}} = \left([T]_{\mathcal{B}}^{\mathcal{C}}\right)^{*},

the conjugate transpose of the matrix of $T$ .

Proof. Because $\mathcal{C}$ is orthonormal, the $(i,j)$ entry of $A = [T]_{\mathcal{B}}^{\mathcal{C}}$ is the $i$ -th coordinate of $Te_j$ , namely $a_{ij} = \langle Te_j, f_i\rangle$ . Likewise the $(j,i)$ entry of $B = [T^{*}]_{\mathcal{C}}^{\mathcal{B}}$ is

b_{ji} = \langle T^{*}f_i, e_j\rangle = \overline{\langle e_j, T^{*}f_i\rangle} = \overline{\langle Te_j, f_i\rangle} = \overline{a_{ij}} .

So $B = \overline{A}^{\mathsf T} = A^{*}$ .∎

Orthonormality is essential: in a skewed basis the coordinates are not inner products, and the matrix of $T^*$ is $G^{-1}A^{*}G$ for the Gram matrix $G$ of the basis, not $A^*$ itself.

Definition 10.78 (Self-adjoint operator). An operator $T : V\to V$ is self-adjoint (Hermitian) if $T^{*} = T$ , i.e. $\langle Tv, w\rangle = \langle v, Tw\rangle$ for all $v,w$ .

Example 10.79 (The adjoint of a projection). Let $P = \operatorname{proj}_W$ be the orthogonal projection of $V$ onto a finite-dimensional subspace $W$ . Show $P$ is self-adjoint and $P^{2} = P$ .

Solution. Write $v = Pv + v'$ and $u = Pu + u'$ with $v', u' \in W^{\perp}$ . Then

\langle Pv, u\rangle = \langle Pv, Pu + u'\rangle = \langle Pv, Pu\rangle,

since $Pv \in W$ is orthogonal to $u' \in W^\perp$ . By exactly the same argument $\langle v, Pu\rangle = \langle Pv, Pu\rangle$ . The two agree, so $P^{*} = P$ .

For idempotence: $Pv$ already lies in $W$ , and the closest point of $W$ to a point of $W$ is itself, so $P(Pv) = Pv$ .□

Example 10.80 (Adjoint of a differentiation-type operator). On $\mathbb{R}^3$ with the standard inner product, let $T(x_1,x_2,x_3) = (x_2, x_3, 0)$ (the shift). Find $T^{*}$ .

Solution. The matrix of $T$ in the standard (orthonormal) basis is

A = \begin{pmatrix}0&1&0\\0&0&1\\0&0&0\end{pmatrix},

so $[T^{*}] = A^{\mathsf T}$ (real entries, so conjugation does nothing), giving $T^{*}(y_1,y_2,y_3) = (0, y_1, y_2)$ — the backward shift. Check directly: $\langle Tx, y\rangle = x_2y_1 + x_3y_2$ and $\langle x, T^{*}y\rangle = x_1\cdot 0 + x_2y_1 + x_3y_2$ . They agree.□

Example 10.81 (A self-adjoint operator on a matrix space). On $M_2(\mathbb{R})$ with the trace inner product, let $T(A) = A^{\mathsf T}$ . Show $T$ is self-adjoint.

Solution. We must check $\langle T(A), B\rangle = \langle A, T(B)\rangle$ , i.e. $\operatorname{tr}\!\big((A^{\mathsf T})^{\mathsf T}B\big) = \operatorname{tr}\!\big(A^{\mathsf T}B^{\mathsf T}\big)$ . Using the entrywise description $\langle X,Y\rangle = \sum_{i,j}x_{ij}y_{ij}$ ,

\langle A^{\mathsf T}, B\rangle = \sum_{i,j} a_{ji}b_{ij}, \qquad \langle A, B^{\mathsf T}\rangle = \sum_{i,j} a_{ij}b_{ji} .

The two sums contain the same terms, relabelled by swapping the names of $i$ and $j$ . Hence $T^{*} = T$ . (It also satisfies $T^{2} = I$ , so $T$ is both self-adjoint and an involution; its eigenvalues are $\pm 1$ , with eigenspaces the symmetric and antisymmetric matrices.)□

Proposition 10.82 (Kernel and image of the adjoint). For linear $T : V \to W$ between finite-dimensional inner product spaces,

\ker T^{*} = (\operatorname{im} T)^{\perp}, \qquad \operatorname{im} T^{*} = (\ker T)^{\perp}.

Proof. For the first: $w \in \ker T^*$ means $T^*w = 0$ , which implies $\langle Tv, w\rangle = \langle v, T^*w\rangle = 0$ for all $v$ , i.e. $w \perp \operatorname{im}T$ . Conversely if $w \perp \operatorname{im}T$ then $\langle v, T^*w\rangle = 0$ for all $v$ ; taking $v = T^*w$ gives $T^*w = 0$ .

The second follows by applying the first to $T^*$ (using $T^{**} = T$ ) and taking complements: $\ker T = (\operatorname{im}T^*)^\perp$ , so $(\ker T)^\perp = \operatorname{im}T^{*}$ by the double-complement corollary.∎

Taking $T$ to be multiplication by a real matrix $A$ , this is the four-fundamental-subspaces statement again, now proved without ever writing a row.

Remark. Self-adjointness is the hypothesis of the spectral theorem, the summit of the subject: every self-adjoint operator on a finite-dimensional inner product space has an orthonormal basis of eigenvectors, and all its eigenvalues are real. In matrix form: a real symmetric $A$ factors as $A = QDQ^{\mathsf T}$ with $Q$ orthogonal and $D$ diagonal; a Hermitian $A$ factors as $A = UDU^{*}$ with $U$ unitary. The two ingredients of the proof are already in this chapter — the reality of $\langle Az,z\rangle$ for Hermitian $A$ , and the fact that the orthogonal complement of an eigenvector is invariant under $A$ — and the chapter on eigenvalues and diagonalisation assembles them.

Summary. An inner product is a positive definite, symmetric (conjugate-symmetric) bilinear (sesquilinear) form. It induces a norm $\|v\| = \sqrt{\langle v,v\rangle}$ and a distance. Cauchy–Schwarz, $|\langle u,v\rangle| \le \|u\|\|v\|$ , gives the triangle inequality and defines angles. Orthogonal sets are independent and obey Pythagoras; Gram–Schmidt turns any independent set into an orthogonal one with the same spans, producing orthonormal bases, Legendre and Chebyshev polynomials. Every finite-dimensional subspace $W$ gives $V = W\oplus W^{\perp}$ , with $\operatorname{proj}_W v$ the unique nearest point of $W$ — the engine of least squares and of Fourier series. Bessel bounds the captured energy; Parseval says a full orthonormal basis captures all of it. Over $\mathbb{C}$ one conjugates the second slot, transposition becomes $A^{*}$ , and the operators worth studying are the self-adjoint and the unitary ones.

Exercises for §10.13 →

Common Mistakes

Assuming a symmetric bilinear form is an inner product. Positive definiteness is a separate requirement and the one that usually fails. Test $\langle v,v\rangle > 0$ on a suspicious vector before using any geometry.
Using weights that are not all positive. $\sum w_ix_iy_i$ is an inner product only when every $w_i > 0$ ; a single zero or negative weight destroys axiom 4.
Forgetting the denominators in the projection formula. For an orthogonal (not orthonormal) basis, $\operatorname{proj}_W v = \sum_i \frac{\langle v,v_i\rangle}{\|v_i\|^2}v_i$ . Dropping $\|v_i\|^2$ is the most common arithmetic error in the chapter.
Projecting onto a non-orthogonal spanning set. Gram–Schmidt first, always — otherwise the "projection" is some vector in $W$ , but not the nearest one, and the residual will not be orthogonal to $W$ .
**Writing $\|u+v\|^2 = \|u\|^2+\|v\|^2$ without checking orthogonality.** The cross term $2\langle u,v\rangle$ vanishes only when $u\perp v$ , and for three or more vectors *every* pair must be orthogonal.
Treating Gram–Schmidt output as canonical. Reordering the inputs gives a different orthonormal basis. Projections, distances and dimensions are order-independent; individual basis vectors are not.
**Assuming $(W^\perp)^\perp = W$ in infinite dimensions.** It needs $W$ finite-dimensional (or closed). The polynomials inside $C[0,1]$ are the standard counterexample.
Dropping the absolute value in Cauchy–Schwarz. The inequality bounds $|\langle u,v\rangle|$ ; the lower bound $-\|u\|\|v\|$ is the half you need when minimising.
**Forgetting to conjugate in $\mathbb{C}^n$ . ** $\langle z,w\rangle = \sum z_k\overline{w_k}$ , not $\sum z_kw_k$ ; the second slot is conjugate-linear, so $\langle u,cv\rangle = \bar c\langle u,v\rangle$ ; and $\|z\|^2 = \sum|z_k|^2$ .
**Expecting the converse of Pythagoras over $\mathbb{C}$ . ** There $\|u+v\|^2 = \|u\|^2+\|v\|^2$ only says $\operatorname{Re}\langle u,v\rangle = 0$ .
**Using $[T^*] = [T]^*$ in a non-orthonormal basis.** The identity is a theorem *about orthonormal bases*; in a skewed basis the Gram matrix intervenes.
**Mishandling $a_0$ in a Fourier series.** The constant term is $a_0/2$ , because $\|1\|^2 = 2\pi$ while $\|\cos mt\|^2 = \pi$ .

About the practice questions. They check that you can carry out this chapter's computations correctly, and each one is graded on a single answer. They are not proof exercises: working through them confirms the mechanics, not that you could prove the results yourself. For that, re-read the statements above and try to reconstruct their proofs with the page closed.

← Orthogonality and Least Squares Symmetric Matrices and Quadratic Forms →

Helpful?