Contents / Linear Algebra / Symmetric Matrices and Quadratic Forms

Chapter 11

Symmetric Matrices and Quadratic Forms

The Spectral Theorem, quadratic forms, and positive definite matrices.

Introduction

A matrix is symmetric when it equals its own transpose. That one equation, $A = A^T$ , is worth more than any other structural hypothesis in linear algebra. It forces the eigenvalues to be real, forces eigenvectors belonging to different eigenvalues to be perpendicular, and guarantees a full orthonormal basis of eigenvectors even when eigenvalues repeat. The general theory of diagonalisation has to worry about complex roots, defective matrices and skewed eigenbases; for symmetric matrices none of those worries survive.

The payoff is the Spectral Theorem:

A = Q\Lambda Q^T,

with $Q$ orthogonal and $\Lambda$ diagonal and real. Read from left to right it says that a symmetric matrix is a pure stretch along $n$ perpendicular axes, dressed up in the wrong coordinate system. Every other result in this chapter is a consequence. Quadratic forms $\mathbf{x}^TA\mathbf{x}$ lose their cross terms in the eigenvector coordinates, so conics and quadric surfaces are classified by the signs of the eigenvalues. The maximum of $\mathbf{x}^TA\mathbf{x}$ on the unit sphere is the largest eigenvalue, which is where constrained optimisation meets the spectrum. And positive definiteness — the matrix version of "positive number" — turns out to have five different faces, one of which (the Cholesky factorisation) is how a computer actually checks it.

Symmetric matrices appear wherever a quantity is measured symmetrically in two arguments: covariance, energy, curvature, conductance, distance. The first three sections build the spectral machinery, the next three apply it to quadratic forms and optimisation, and the last three develop positive definiteness and what it is for. The Singular Value Decomposition, which extends these ideas to matrices that are neither square nor symmetric, has a chapter of its own.

11.1Symmetric matrices and the transpose

Definition 11.1 (Symmetric and skew-symmetric matrices). A square real matrix $A$ is symmetric if

A^T = A,

equivalently $a_{ij} = a_{ji}$ for all $i,j$ . It is skew-symmetric if $A^T = -A$ , which forces every diagonal entry to be zero.

Symmetry is a statement about reflection across the main diagonal: the entry in row $i$ , column $j$ matches the entry in row $j$ , column $i$ . A symmetric $n \times n$ matrix therefore carries $\tfrac{n(n+1)}{2}$ independent numbers rather than $n^{2}$ , and only square matrices can be symmetric at all.

The reason symmetry is powerful is not the picture but the following identity, which converts a statement about entries into a statement about the dot product.

Proposition 11.2 (Symmetry is self-adjointness). A real $n\times n$ matrix $A$ is symmetric if and only if

(A\mathbf{x})\cdot\mathbf{y} = \mathbf{x}\cdot(A\mathbf{y}) \qquad \text{for all } \mathbf{x},\mathbf{y}\in\mathbb{R}^n.

Proof. Write the dot product as a matrix product: $\mathbf{u}\cdot\mathbf{v} = \mathbf{u}^T\mathbf{v}$ . Then

(A\mathbf{x})\cdot\mathbf{y} = (A\mathbf{x})^T\mathbf{y} = \mathbf{x}^TA^T\mathbf{y}, \qquad \mathbf{x}\cdot(A\mathbf{y}) = \mathbf{x}^TA\mathbf{y}.

If $A^T = A$ the two right-hand sides are identical, so the identity holds. Conversely, suppose the identity holds for all $\mathbf{x}$ and $\mathbf{y}$ . Then $\mathbf{x}^T(A^T - A)\mathbf{y} = 0$ for all $\mathbf{x},\mathbf{y}$ ; taking $\mathbf{x}=\mathbf{e}_i$ and $\mathbf{y}=\mathbf{e}_j$ picks out the $(i,j)$ entry of $A^T-A$ , so every entry of $A^T-A$ is zero.∎

This is the property the whole chapter runs on: a symmetric matrix can be moved from one side of a dot product to the other for free. Every proof about real eigenvalues, orthogonal eigenvectors and the Spectral Theorem is an application of it. In the language of the inner-product-spaces chapter, a symmetric matrix is exactly a self-adjoint operator on $\mathbb{R}^n$ with the standard inner product.

Symmetric matrices are not closed under multiplication, but they are closed under the operations that matter most.

Proposition 11.3 (Constructions that produce symmetric matrices). Let $A$ be any real $m\times n$ matrix and let $S, T$ be symmetric $n\times n$ matrices. Then:

$A^TA$ (size $n\times n$ ) and $AA^T$ (size $m\times m$ ) are symmetric;
$S + T$ and $cS$ are symmetric for every scalar $c$ ;
$S^k$ is symmetric for every $k\geq 0$ , and $S^{-1}$ is symmetric when $S$ is invertible;
$P^TSP$ is symmetric for every real matrix $P$ of compatible size;
every square matrix $M$ splits uniquely as $M = \tfrac12(M+M^T) + \tfrac12(M-M^T)$ , a symmetric plus a skew-symmetric matrix.

Proof. For (1), $(A^TA)^T = A^T(A^T)^T = A^TA$ . Parts (2) and (3) follow from $(S+T)^T = S^T+T^T$ and $(S^k)^T = (S^T)^k$ ; for the inverse, transpose $SS^{-1}=I$ to get $(S^{-1})^TS = I$ , so $(S^{-1})^T = S^{-1}$ . For (4), $(P^TSP)^T = P^TS^TP = P^TSP$ . For (5), the first summand is symmetric and the second is skew by inspection, and if $M = S' + K'$ with $S'$ symmetric and $K'$ skew, transposing gives $M^T = S' - K'$ , so $S' = \tfrac12(M+M^T)$ and $K' = \tfrac12(M-M^T)$ .∎

Part (1) is the single most-used fact in applied linear algebra: normal equations $A^TA\hat{\mathbf{x}} = A^T\mathbf{b}$ , Gram matrices, covariance matrices and kernel matrices are all of this form, so the Spectral Theorem applies to all of them. Part (4) is the transformation rule for quadratic forms under a change of variable, and it is why congruence $S\mapsto P^TSP$ rather than similarity $S \mapsto P^{-1}SP$ is the natural notion for forms.

Intuition. Think of $A$ as a table of pairwise measurements: $a_{ij}$ is how strongly station $i$ is connected to station $j$ . Symmetry says the connection is mutual — the road from $i$ to $j$ is the same road as the one from $j$ to $i$ . Distance matrices, spring networks, electrical conductances and correlation tables are all symmetric for this reason, and the theorems below are why those subjects have such clean theory. Traffic flow, in contrast, is genuinely directional: $a_{ij}$ cars go one way and $a_{ji}$ the other, the matrix is not symmetric, and none of this chapter applies.

Example 11.4 (Recognising and building symmetric matrices). Which of

M_1 = \begin{pmatrix}2&-3\\-3&7\end{pmatrix}, \quad M_2 = \begin{pmatrix}0&5\\-5&0\end{pmatrix}, \quad M_3 = \begin{pmatrix}1&2&3\\2&4&5\\3&5&6\end{pmatrix}

are symmetric? For $A = \begin{pmatrix}1&2\\0&1\\3&-1\end{pmatrix}$ , compute $A^TA$ and check that it is symmetric.

Solution.

$M_1$ : the $(1,2)$ and $(2,1)$ entries are both $-3$ , so $M_1^T = M_1$ . Symmetric.
$M_2$ : the $(1,2)$ entry is $5$ and the $(2,1)$ entry is $-5$ , so $M_2^T = -M_2$ . Skew-symmetric, not symmetric.
$M_3$ : reflect across the diagonal — $2\leftrightarrow 2$ , $3 \leftrightarrow 3$ , $5 \leftrightarrow 5$ . Symmetric.
$A^TA = \begin{pmatrix}1&0&3\\2&1&-1\end{pmatrix}\begin{pmatrix}1&2\\0&1\\3&-1\end{pmatrix} = \begin{pmatrix}1+0+9 & 2+0-3\\ 2+0-3 & 4+1+1\end{pmatrix} = \begin{pmatrix}10&-1\\-1&6\end{pmatrix}.$
Sanity check: the diagonal entries of $A^TA$ are the squared lengths of the columns of $A$ , namely $1+0+9=10$ and $4+1+1=6$ , and the off-diagonal entry is the dot product of the two columns, $2+0-3=-1$ . Both appear once above and once below the diagonal, as they must.

□

Example 11.5 (The symmetric part of a matrix). Split $M = \begin{pmatrix}4&10\\-2&6\end{pmatrix}$ into a symmetric and a skew-symmetric part.

Solution.

$M^T = \begin{pmatrix}4&-2\\10&6\end{pmatrix}$ .
Symmetric part: $\tfrac12(M+M^T) = \tfrac12\begin{pmatrix}8&8\\8&12\end{pmatrix} = \begin{pmatrix}4&4\\4&6\end{pmatrix}$ .
Skew part: $\tfrac12(M-M^T) = \tfrac12\begin{pmatrix}0&12\\-12&0\end{pmatrix} = \begin{pmatrix}0&6\\-6&0\end{pmatrix}$ .
Sanity check: the two parts add back to $M$ , the second has zero diagonal, and the skew part contributes nothing to $\mathbf{x}^TM\mathbf{x}$ — a fact used in the section on quadratic forms.

□

Pitfall. The product of two symmetric matrices is usually not symmetric. With $S = \begin{pmatrix}1&0\\0&2\end{pmatrix}$ and $T = \begin{pmatrix}0&1\\1&0\end{pmatrix}$ , both symmetric, $ST = \begin{pmatrix}0&1\\2&0\end{pmatrix}$ is not. In general $(ST)^T = T^TS^T = TS$ , so $ST$ is symmetric exactly when $S$ and $T$ commute.

Exercises for §11.1 →

11.2The Spectral Theorem

Diagonalising a matrix means finding a basis in which it acts by pure scaling. For a general matrix such a basis may not exist, and when it does the axes can be badly skewed. Symmetry removes both problems at once. We build the result out of two lemmas, each a two-line application of self-adjointness.

Lemma 11.6 (Real symmetric matrices have real eigenvalues). Every eigenvalue of a real symmetric matrix $A$ is real, and every eigenvalue has a real eigenvector.

Proof. The characteristic polynomial $\det(A-\lambda I)$ has real coefficients but might, a priori, have complex roots, so work in $\mathbb{C}^n$ . Suppose $A\mathbf{v} = \lambda\mathbf{v}$ with $\mathbf{v}\neq\mathbf{0}$ , and write $\bar{\mathbf{v}}$ for the entrywise complex conjugate. Consider the scalar

s = \bar{\mathbf{v}}^{\,T}A\mathbf{v}.

On one hand $s = \bar{\mathbf{v}}^{\,T}(\lambda\mathbf{v}) = \lambda\,\bar{\mathbf{v}}^{\,T}\mathbf{v} = \lambda\sum_{k}|v_k|^2$ . On the other hand, conjugating and transposing a $1\times 1$ matrix leaves it fixed up to conjugation, so

\bar{s} = \mathbf{v}^T\bar{A}\bar{\mathbf{v}} = \mathbf{v}^TA\bar{\mathbf{v}} = \left(\mathbf{v}^TA\bar{\mathbf{v}}\right)^T = \bar{\mathbf{v}}^{\,T}A^T\mathbf{v} = \bar{\mathbf{v}}^{\,T}A\mathbf{v} = s,

using $\bar{A}=A$ (the entries are real) and $A^T=A$ . So $s$ equals its own conjugate and is therefore real. Since $\sum_k|v_k|^2$ is a positive real number, $\lambda = s/\sum_k |v_k|^2$ is real.

Given that $\lambda$ is real, the matrix $A-\lambda I$ is a real singular matrix, so its null space contains a nonzero real vector; that is a real eigenvector.∎

Two hypotheses are load-bearing. Real entries: the complex matrix $\begin{pmatrix}i&0\\0&i\end{pmatrix}$ is equal to its transpose and has eigenvalue $i$ . (For complex matrices the right condition is $A^* = A$ , Hermitian, and the same proof then gives real eigenvalues.) Symmetry: the rotation matrix $\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ is real with eigenvalues $\pm i$ .

Lemma 11.7 (Eigenvectors for distinct eigenvalues are orthogonal). Let $A$ be real symmetric with $A\mathbf{u} = \lambda\mathbf{u}$ and $A\mathbf{w} = \mu\mathbf{w}$ , where $\lambda\neq\mu$ . Then $\mathbf{u}\cdot\mathbf{w} = 0$ .

Proof. Move $A$ across the dot product using the self-adjointness identity:

\lambda(\mathbf{u}\cdot\mathbf{w}) = (A\mathbf{u})\cdot\mathbf{w} = \mathbf{u}\cdot(A\mathbf{w}) = \mu(\mathbf{u}\cdot\mathbf{w}).

Hence $(\lambda-\mu)(\mathbf{u}\cdot\mathbf{w}) = 0$ . Since $\lambda\neq\mu$ , the factor $\lambda-\mu$ is nonzero, so $\mathbf{u}\cdot\mathbf{w} = 0$ .∎

Notice how little was used: the eigenvectors were never computed. The orthogonality is forced by the algebra of the transpose, which is why it holds for every symmetric matrix, however large. For a non-symmetric matrix the conclusion fails at once: $\begin{pmatrix}1&1\\0&2\end{pmatrix}$ has eigenvectors $\langle 1,0\rangle$ and $\langle 1,1\rangle$ for the distinct eigenvalues $1$ and $2$ , and their dot product is $1$ .

When the $n$ eigenvalues are distinct, the two lemmas already finish the job: normalise the $n$ mutually orthogonal eigenvectors and you have an orthonormal eigenbasis. The work is in the repeated case, and the following induction handles it without ever mentioning multiplicities.

Theorem 11.8 (Spectral Theorem). Let $A$ be a real symmetric $n\times n$ matrix. Then there is an orthogonal matrix $Q$ (so $Q^TQ = I$ ) and a real diagonal matrix $\Lambda = \operatorname{diag}(\lambda_1,\dots,\lambda_n)$ with

A = Q\Lambda Q^T.

The columns $\mathbf{q}_1,\dots,\mathbf{q}_n$ of $Q$ are an orthonormal basis of $\mathbb{R}^n$ consisting of eigenvectors of $A$ , with $A\mathbf{q}_k = \lambda_k\mathbf{q}_k$ .

Proof. Induct on $n$ . For $n=1$ take $Q = (1)$ and $\Lambda = A$ .

Let $n\geq 2$ and assume the theorem for symmetric matrices of size $n-1$ . The characteristic polynomial of $A$ has a root in $\mathbb{C}$ , and by the real-eigenvalue lemma that root $\lambda_1$ is real with a real eigenvector, which we may scale to a unit vector $\mathbf{q}_1$ .

Let $W = \{\mathbf{w} : \mathbf{w}\cdot\mathbf{q}_1 = 0\}$ , a subspace of dimension $n-1$ . The key point is that $A$ maps $W$ into itself: if $\mathbf{w}\in W$ then

(A\mathbf{w})\cdot\mathbf{q}_1 = \mathbf{w}\cdot(A\mathbf{q}_1) = \lambda_1(\mathbf{w}\cdot\mathbf{q}_1) = 0,

so $A\mathbf{w}\in W$ . This step is exactly where symmetry is used; without it the orthogonal complement of an eigenvector need not be invariant.

Choose an orthonormal basis $\mathbf{z}_2,\dots,\mathbf{z}_n$ of $W$ (Gram–Schmidt supplies one) and set $Q_1 = [\,\mathbf{q}_1\ \mathbf{z}_2\ \cdots\ \mathbf{z}_n\,]$ , an orthogonal matrix. Consider $B = Q_1^TAQ_1$ . Its first column is $Q_1^TA\mathbf{q}_1 = \lambda_1Q_1^T\mathbf{q}_1 = \lambda_1\mathbf{e}_1$ . Moreover $B^T = Q_1^TA^TQ_1 = B$ , so $B$ is symmetric and its first row is $\lambda_1\mathbf{e}_1^T$ as well. Therefore

B = \begin{pmatrix}\lambda_1 & \mathbf{0}^T\\ \mathbf{0} & A_1\end{pmatrix}

with $A_1$ symmetric of size $n-1$ . By the induction hypothesis $A_1 = P\Lambda_1P^T$ with $P$ orthogonal and $\Lambda_1$ diagonal. Put

R = \begin{pmatrix}1 & \mathbf{0}^T\\ \mathbf{0} & P\end{pmatrix},

which is orthogonal, and $Q = Q_1R$ , a product of orthogonal matrices and hence orthogonal. Then

Q^TAQ = R^T(Q_1^TAQ_1)R = R^TBR = \begin{pmatrix}\lambda_1&\mathbf{0}^T\\\mathbf{0}&P^TA_1P\end{pmatrix} = \begin{pmatrix}\lambda_1&\mathbf{0}^T\\\mathbf{0}&\Lambda_1\end{pmatrix} = \Lambda,

a diagonal matrix. Multiplying by $Q$ on the left and $Q^T$ on the right gives $A = Q\Lambda Q^T$ . Reading $AQ = Q\Lambda$ column by column gives $A\mathbf{q}_k = \lambda_k\mathbf{q}_k$ .∎

The converse is a one-liner and worth recording, because it says symmetry is not merely sufficient for orthogonal diagonalisability but necessary.

Proposition 11.9 (Orthogonally diagonalisable means symmetric). If $A = Q\Lambda Q^T$ with $Q$ orthogonal and $\Lambda$ real diagonal, then $A$ is symmetric.

Proof. $A^T = (Q\Lambda Q^T)^T = Q\Lambda^TQ^T = Q\Lambda Q^T = A$ , since a diagonal matrix is its own transpose.∎

Corollary 11.10 (What the Spectral Theorem buys). For a real symmetric $n\times n$ matrix $A$ :

all $n$ eigenvalues are real, counted with multiplicity;
for every eigenvalue, the dimension of the eigenspace equals its multiplicity as a root of the characteristic polynomial — a symmetric matrix is never defective;
eigenvectors for distinct eigenvalues are orthogonal, and eigenspaces for distinct eigenvalues are orthogonal subspaces;
$\operatorname{tr}A = \lambda_1+\cdots+\lambda_n$ and $\det A = \lambda_1\cdots\lambda_n$ ;
$\operatorname{rank}A$ is the number of nonzero eigenvalues, and $\operatorname{Nul}A$ is the eigenspace for $\lambda = 0$ .

Proof. Parts (1) and (3) are the two lemmas. For (2), $\Lambda$ is similar to $A$ , so they share a characteristic polynomial; the number of $\lambda$ 's on the diagonal of $\Lambda$ is therefore the algebraic multiplicity, and the corresponding columns of $Q$ are that many independent eigenvectors, so the geometric multiplicity is at least — hence equal to — the algebraic one. For (4), similarity preserves trace and determinant, and for a diagonal matrix both are read off directly. For (5), $\operatorname{rank}(Q\Lambda Q^T) = \operatorname{rank}\Lambda$ because multiplying by invertible matrices does not change rank, and the rank of a diagonal matrix is its number of nonzero entries.∎

Intuition. Picture the unit sphere in $\mathbb{R}^3$ and a symmetric $A$ acting on it. The image is an ellipsoid, and the Spectral Theorem says its three axes are perpendicular and are exactly the directions $\mathbf{q}_1,\mathbf{q}_2,\mathbf{q}_3$ , stretched by $\lambda_1,\lambda_2,\lambda_3$ . A non-symmetric matrix also turns the sphere into an ellipsoid, but its eigenvectors are no longer the axes of that ellipsoid; it shears as well as stretches. Symmetry is the statement "no shear in the eigenvector frame".

Method 11.11 (Orthogonally diagonalising a symmetric matrix).

Compute the characteristic polynomial $\det(A - \lambda I)$ and find its roots. They are guaranteed real.
For each eigenvalue, solve $(A-\lambda I)\mathbf{x} = \mathbf{0}$ for a basis of the eigenspace.
Apply Gram–Schmidt inside each eigenspace of dimension $\geq 2$ . Vectors from different eigenspaces are already orthogonal and need no work.
Normalise every vector to unit length and place them as the columns of $Q$ , with the matching eigenvalues in the same order on the diagonal of $\Lambda$ .
Check $Q^TQ = I$ and, on a small example, $Q\Lambda Q^T = A$ .

Example 11.12 (A $2\times 2$ orthogonal diagonalisation). Find $Q$ and $\Lambda$ with $A = Q\Lambda Q^T$ for $A = \begin{pmatrix}3&1\\1&3\end{pmatrix}$ .

Solution.

Characteristic equation: $(3-\lambda)^2 - 1 = 0$ , i.e. $\lambda^{2} - 6\lambda + 8 = 0$ , so $\lambda_1 = 4$ and $\lambda_2 = 2$ .
For $\lambda_1 = 4$ : $(A-4I)\mathbf{v} = \begin{pmatrix}-1&1\\1&-1\end{pmatrix}\mathbf{v} = \mathbf{0}$ gives $v_1 = v_2$ , so $\langle 1,1\rangle$ spans the eigenspace and $\mathbf{q}_1 = \langle 1/\sqrt2, 1/\sqrt2\rangle$ .
For $\lambda_2 = 2$ : $\begin{pmatrix}1&1\\1&1\end{pmatrix}\mathbf{v} = \mathbf{0}$ gives $v_1 = -v_2$ , so $\mathbf{q}_2 = \langle 1/\sqrt2, -1/\sqrt2\rangle$ . As the orthogonality lemma promised, $\mathbf{q}_1\cdot\mathbf{q}_2 = \tfrac12-\tfrac12 = 0$ .
Assemble $Q = \dfrac{1}{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$ and $\Lambda = \begin{pmatrix}4&0\\0&2\end{pmatrix}$ .
Sanity check: $Q\Lambda = \dfrac{1}{\sqrt2}\begin{pmatrix}4&2\\4&-2\end{pmatrix}$ , and multiplying by $Q^T = Q$ gives $\dfrac12\begin{pmatrix}4+2&4-2\\4-2&4+2\end{pmatrix} = \begin{pmatrix}3&1\\1&3\end{pmatrix} = A$ . Also $\operatorname{tr}A = 6 = 4+2$ and $\det A = 8 = 4\cdot 2$ .

□

Example 11.13 (A $3\times 3$ with a repeated eigenvalue). Orthogonally diagonalise

A = \begin{pmatrix}3&-2&4\\-2&6&2\\4&2&3\end{pmatrix}.

Solution.

Expanding $\det(A-\lambda I)$ gives $-(\lambda-7)^2(\lambda+2)$ , so the eigenvalues are $\lambda = 7$ (twice) and $\lambda = -2$ . Check against the corollary: $\operatorname{tr}A = 3+6+3 = 12 = 7+7-2$ , and $\det A = -98 = 7\cdot 7\cdot(-2)$ .
For $\lambda = -2$ : $A + 2I = \begin{pmatrix}5&-2&4\\-2&8&2\\4&2&5\end{pmatrix}$ . The second row gives $-x+4y+z = 0$ , so $z = x-4y$ ; substituting into the first row gives $9x - 18y = 0$ , i.e. $x = 2y$ , and then $z = -2y$ . The eigenspace is spanned by $\langle 2,1,-2\rangle$ , of length $3$ , so $\mathbf{q}_3 = \tfrac13\langle 2,1,-2\rangle$ .
For $\lambda = 7$ : $A - 7I = \begin{pmatrix}-4&-2&4\\-2&-1&2\\4&2&-4\end{pmatrix}$ has rank $1$ ; every row is a multiple of $2x+y-2z = 0$ . The eigenspace is the plane $2x+y-2z=0$ , dimension $2$ — matching the algebraic multiplicity, as a symmetric matrix must.
Pick a basis of that plane: $\mathbf{w}_1 = \langle 1,-2,0\rangle$ and $\mathbf{w}_2 = \langle 0,2,1\rangle$ . They are not orthogonal ( $\mathbf{w}_1\cdot\mathbf{w}_2 = -4$ ), so Gram–Schmidt within the eigenspace:

\mathbf{w}_2' = \mathbf{w}_2 - \frac{-4}{5}\mathbf{w}_1 = \left\langle \tfrac45, \tfrac25, 1\right\rangle,

and scaling by $5$ gives $\langle 4,2,5\rangle$ , of length $\sqrt{45} = 3\sqrt5$ . 5. Normalise: $\mathbf{q}_1 = \tfrac{1}{\sqrt5}\langle 1,-2,0\rangle$ , $\mathbf{q}_2 = \tfrac{1}{3\sqrt5}\langle 4,2,5\rangle$ , $\mathbf{q}_3 = \tfrac13\langle 2,1,-2\rangle$ , and $\Lambda = \operatorname{diag}(7,7,-2)$ . 6. Sanity check the cross-eigenspace orthogonality that the lemma guarantees: $\langle 1,-2,0\rangle\cdot\langle 2,1,-2\rangle = 2-2+0 = 0$ and $\langle 4,2,5\rangle\cdot\langle 2,1,-2\rangle = 8+2-10 = 0$ . Only the two vectors inside the $\lambda=7$ eigenspace needed adjusting.□

Pitfall. Gram–Schmidt is needed only within a repeated eigenspace. Orthogonalising a vector from one eigenspace against a vector from another is not merely unnecessary — it destroys the eigenvector property, because the result is a combination of eigenvectors for different eigenvalues and is no longer an eigenvector at all.

Exercises for §11.2 →

11.3The spectral decomposition

The factorisation $A = Q\Lambda Q^T$ can be expanded into a sum, and the summands turn out to be projections. This form is how symmetric matrices are used in practice, and it makes powers, inverses and square roots of $A$ immediate.

Theorem 11.14 (Spectral decomposition). Let $A$ be real symmetric with orthonormal eigenvectors $\mathbf{q}_1,\dots,\mathbf{q}_n$ and eigenvalues $\lambda_1,\dots,\lambda_n$ . Then

A = \lambda_1\mathbf{q}_1\mathbf{q}_1^T + \lambda_2\mathbf{q}_2\mathbf{q}_2^T + \cdots + \lambda_n\mathbf{q}_n\mathbf{q}_n^T = \sum_{k=1}^n \lambda_k P_k,

where $P_k = \mathbf{q}_k\mathbf{q}_k^T$ is the matrix of the orthogonal projection of $\mathbb{R}^n$ onto the line spanned by $\mathbf{q}_k$ . These projections satisfy

P_k^T = P_k, \qquad P_k^2 = P_k, \qquad P_jP_k = 0 \ (j\neq k), \qquad P_1+\cdots+P_n = I.

Proof. Write $Q\Lambda$ column by column: the $k$ -th column of $Q\Lambda$ is $\lambda_k\mathbf{q}_k$ . Multiplying a matrix by another on the right is the sum of column-times-row outer products, so

A = (Q\Lambda)Q^T = \sum_{k=1}^n(\lambda_k\mathbf{q}_k)\mathbf{q}_k^T = \sum_{k=1}^n \lambda_k\mathbf{q}_k\mathbf{q}_k^T.

For the projection properties, $P_k^T = (\mathbf{q}_k\mathbf{q}_k^T)^T = \mathbf{q}_k\mathbf{q}_k^T = P_k$ , and since $\mathbf{q}_k^T\mathbf{q}_k = 1$ ,

P_k^2 = \mathbf{q}_k(\mathbf{q}_k^T\mathbf{q}_k)\mathbf{q}_k^T = \mathbf{q}_k\mathbf{q}_k^T = P_k.

For $j \neq k$ , $\mathbf{q}_j^T\mathbf{q}_k = 0$ gives $P_jP_k = \mathbf{q}_j(\mathbf{q}_j^T\mathbf{q}_k)\mathbf{q}_k^T = 0$ . Finally $\sum_kP_k = QQ^T = I$ because $Q$ is orthogonal. That $P_k\mathbf{x} = (\mathbf{q}_k\cdot\mathbf{x})\mathbf{q}_k$ is precisely the orthogonal projection onto $\operatorname{span}\{\mathbf{q}_k\}$ .∎

The identity $\sum_kP_k = I$ is a resolution of the identity: every vector splits into its components along the eigen-directions, $\mathbf{x} = \sum_k(\mathbf{q}_k\cdot\mathbf{x})\mathbf{q}_k$ , and then

A\mathbf{x} = \sum_{k=1}^n\lambda_k(\mathbf{q}_k\cdot\mathbf{x})\mathbf{q}_k.

So $A$ does nothing but rescale each component by its own eigenvalue. Every $\lambda_k\mathbf{q}_k\mathbf{q}_k^T$ has rank $1$ (or rank $0$ when $\lambda_k = 0$ ), which is why the decomposition is called a sum of rank-one pieces. When eigenvalues repeat, the projections onto the individual lines can be grouped: for a distinct eigenvalue $\mu$ with eigenspace $E_\mu$ , summing the $P_k$ over the $\mathbf{q}_k$ spanning $E_\mu$ gives the orthogonal projection onto the whole eigenspace, and $A = \sum_{\mu}\mu\,P_{E_\mu}$ over the distinct eigenvalues.

Example 11.15 (Spectral decomposition of a $2\times 2$ matrix). Write $A = \begin{pmatrix}3&1\\1&3\end{pmatrix}$ as $\lambda_1P_1 + \lambda_2P_2$ .

Solution.

From the previous section, $\lambda_1 = 4$ with $\mathbf{q}_1 = \tfrac{1}{\sqrt2}\langle 1,1\rangle$ and $\lambda_2 = 2$ with $\mathbf{q}_2 = \tfrac{1}{\sqrt2}\langle 1,-1\rangle$ .
$P_1 = \mathbf{q}_1\mathbf{q}_1^T = \dfrac12\begin{pmatrix}1\\1\end{pmatrix}\begin{pmatrix}1&1\end{pmatrix} = \dfrac12\begin{pmatrix}1&1\\1&1\end{pmatrix}$ .
$P_2 = \dfrac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix}$ .
$4P_1 + 2P_2 = \begin{pmatrix}2&2\\2&2\end{pmatrix} + \begin{pmatrix}1&-1\\-1&1\end{pmatrix} = \begin{pmatrix}3&1\\1&3\end{pmatrix} = A$ .
Sanity check the projection identities: $P_1+P_2 = I$ , and $P_1P_2 = \tfrac14\begin{pmatrix}1&1\\1&1\end{pmatrix}\begin{pmatrix}1&-1\\-1&1\end{pmatrix} = 0$ .

□

Because the projections multiply so simply, any polynomial or power series in $A$ collapses onto the eigenvalues.

Proposition 11.16 (Functions of a symmetric matrix). With $A = \sum_k\lambda_kP_k$ as above, for every integer $m\geq 0$

A^m = \sum_{k=1}^n\lambda_k^mP_k = Q\Lambda^mQ^T,

and if $A$ is invertible, $A^{-1} = \sum_k\lambda_k^{-1}P_k$ . If every $\lambda_k\geq 0$ , the matrix $A^{1/2} = \sum_k\sqrt{\lambda_k}\,P_k$ is symmetric with nonnegative eigenvalues and satisfies $\left(A^{1/2}\right)^2 = A$ .

Proof. $A^m = (Q\Lambda Q^T)(Q\Lambda Q^T)\cdots(Q\Lambda Q^T)$ ; each interior $Q^TQ$ collapses to $I$ , leaving $Q\Lambda^mQ^T$ . Expanding as before gives $\sum_k\lambda_k^mP_k$ . For the inverse, $Q\Lambda^{-1}Q^T$ multiplied by $Q\Lambda Q^T$ is $Q\Lambda^{-1}\Lambda Q^T = I$ . For the square root, $\left(\sum_k\sqrt{\lambda_k}P_k\right)^2 = \sum_{j,k}\sqrt{\lambda_j\lambda_k}\,P_jP_k = \sum_k\lambda_kP_k = A$ , using $P_jP_k=0$ for $j\neq k$ and $P_k^{2} = P_k$ .∎

Example 11.17 (A matrix power and a matrix square root). Compute $A^{3}$ for $A = \begin{pmatrix}3&1\\1&3\end{pmatrix}$ , and find a symmetric square root of $B = \begin{pmatrix}5&4\\4&5\end{pmatrix}$ .

Solution.

$A^{3} = 4^3P_1 + 2^3P_2 = 64\cdot\tfrac12\begin{pmatrix}1&1\\1&1\end{pmatrix} + 8\cdot\tfrac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix} = \begin{pmatrix}32&32\\32&32\end{pmatrix}+\begin{pmatrix}4&-4\\-4&4\end{pmatrix} = \begin{pmatrix}36&28\\28&36\end{pmatrix}$ .
Check directly: $A^{2} = \begin{pmatrix}10&6\\6&10\end{pmatrix}$ and $A^{2}A = \begin{pmatrix}36&28\\28&36\end{pmatrix}$ .
For $B$ : $\operatorname{tr}B = 10$ , $\det B = 25-16 = 9$ , so $\lambda^{2}-10\lambda+9 = 0$ and the eigenvalues are $9$ and $1$ , with the same eigenvectors $\tfrac{1}{\sqrt2}\langle 1,1\rangle$ and $\tfrac{1}{\sqrt2}\langle 1,-1\rangle$ .
$B^{1/2} = 3P_1 + 1\cdot P_2 = \tfrac32\begin{pmatrix}1&1\\1&1\end{pmatrix} + \tfrac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix} = \begin{pmatrix}2&1\\1&2\end{pmatrix}$ .
Sanity check: $\begin{pmatrix}2&1\\1&2\end{pmatrix}^2 = \begin{pmatrix}5&4\\4&5\end{pmatrix} = B$ .

□

Remark. The square root above is the unique symmetric square root with nonnegative eigenvalues; other square roots exist (flip a sign on $P_2$ to get $\begin{pmatrix}1&2\\2&1\end{pmatrix}$ , whose square is also $B$ ), but only one of them is positive semidefinite. This canonical root is what lets a covariance matrix be "unsquared" and is the standard way to whiten data.

Pitfall. The formula $A^m = Q\Lambda^mQ^T$ needs an orthogonal $Q$ so that $Q^{-1} = Q^T$ . For a general diagonalisable matrix the correct statement is $A^m = P D^mP^{-1}$ , and writing $P^T$ in place of $P^{-1}$ is simply wrong. Likewise $\mathbf{q}\mathbf{q}^T$ is a projection only when $\mathbf{q}$ is a unit vector; in general the projection is $\dfrac{\mathbf{v}\mathbf{v}^T}{\mathbf{v}^T\mathbf{v}}$ .

Exercises for §11.3 →

11.4Quadratic forms

A quadratic form is a homogeneous polynomial of degree two. The point of this section is that every such polynomial is $\mathbf{x}^TA\mathbf{x}$ for exactly one symmetric $A$ , which puts the whole spectral machinery at its disposal.

Definition 11.18 (Quadratic form). A quadratic form on $\mathbb{R}^n$ is a function

Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x} = \sum_{i=1}^n\sum_{j=1}^n a_{ij}x_ix_j,

where $A$ is a real symmetric $n\times n$ matrix, called the matrix of the form.

Expanding the double sum and collecting the terms $a_{ij}x_ix_j$ and $a_{ji}x_jx_i$ gives the working formula

Q(\mathbf{x}) = \sum_{i=1}^n a_{ii}x_i^2 + 2\sum_{i<j}a_{ij}x_ix_j.

So the diagonal entries are the coefficients of the squares, and each off-diagonal entry is half the coefficient of the corresponding cross term. In two variables,

Q(x,y) = ax^2 + 2bxy + cy^2 \quad\longleftrightarrow\quad A = \begin{pmatrix}a&b\\b&c\end{pmatrix}.

Insisting that $A$ be symmetric is a normalisation, not a restriction: any square $M$ gives the same values, because the skew part contributes nothing.

Proposition 11.19 (The symmetric matrix of a form is unique). For any square $M$ , $\mathbf{x}^TM\mathbf{x} = \mathbf{x}^TS\mathbf{x}$ for all $\mathbf{x}$ , where $S = \tfrac12(M+M^T)$ . Moreover, if $A$ and $B$ are symmetric and $\mathbf{x}^TA\mathbf{x} = \mathbf{x}^TB\mathbf{x}$ for all $\mathbf{x}$ , then $A = B$ .

Proof. The number $\mathbf{x}^TM\mathbf{x}$ is $1\times 1$ , so it equals its own transpose $\mathbf{x}^TM^T\mathbf{x}$ ; averaging the two gives $\mathbf{x}^TM\mathbf{x} = \mathbf{x}^T\tfrac12(M+M^T)\mathbf{x}$ . For uniqueness let $C = A-B$ , symmetric with $\mathbf{x}^TC\mathbf{x} = 0$ for all $\mathbf{x}$ . Taking $\mathbf{x} = \mathbf{e}_i$ gives $c_{ii} = 0$ . Taking $\mathbf{x} = \mathbf{e}_i+\mathbf{e}_j$ gives $c_{ii}+c_{jj}+2c_{ij} = 0$ , hence $c_{ij} = 0$ . So $C = 0$ .∎

Example 11.20 (Passing between a polynomial and its matrix). (a) Write $Q(x_1,x_2,x_3) = 4x_1^2 + 3x_2^2 - x_3^2 + 6x_1x_2 - 8x_2x_3$ in the form $\mathbf{x}^TA\mathbf{x}$ . (b) Write out the polynomial for $A = \begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$ and evaluate it at $\mathbf{x} = \langle 1,2,1\rangle$ .

Solution.

(a) Diagonal entries are the coefficients of the squares: $a_{11}=4$ , $a_{22}=3$ , $a_{33}=-1$ .
Off-diagonal entries are half the cross-term coefficients: $a_{12}=a_{21}=6/2=3$ , $a_{23}=a_{32}=-8/2=-4$ , and $a_{13}=a_{31}=0$ since $x_1x_3$ is absent. Hence

A = \begin{pmatrix}4&3&0\\3&3&-4\\0&-4&-1\end{pmatrix}.

(b) Reading the matrix back: $Q = 2x_1^2+2x_2^2+2x_3^2 - 2x_1x_2 - 2x_2x_3$ (each off-diagonal $-1$ doubles to $-2$ ).
At $\langle 1,2,1\rangle$ : $2(1)+2(4)+2(1) - 2(1)(2) - 2(2)(1) = 2+8+2-4-4 = 4$ .
Sanity check by matrix multiplication: $A\mathbf{x} = \langle 2-2+0,\ -1+4-1,\ 0-2+2\rangle = \langle 0,2,0\rangle$ , and $\mathbf{x}^T\langle 0,2,0\rangle = 0+4+0 = 4$ .

□

A change of variable transforms the matrix by congruence, and this is the operation that will remove cross terms.

Proposition 11.21 (Change of variable in a quadratic form). If $\mathbf{x} = P\mathbf{y}$ with $P$ invertible, then

\mathbf{x}^TA\mathbf{x} = \mathbf{y}^T\!\left(P^TAP\right)\mathbf{y},

and $P^TAP$ is symmetric. The matrices $A$ and $P^TAP$ are said to be congruent.

Proof. Substitute: $(P\mathbf{y})^TA(P\mathbf{y}) = \mathbf{y}^TP^TAP\mathbf{y}$ . Symmetry of $P^TAP$ was proved among the constructions above.∎

Congruence is not similarity — $P^TAP$ and $A$ generally have different eigenvalues. What congruence does preserve is the number of positive, negative and zero eigenvalues.

Theorem 11.22 (Sylvester's law of inertia). Let $A$ be real symmetric and let $P$ be invertible. Then $A$ and $P^TAP$ have the same inertia: the same number $p$ of positive eigenvalues, the same number $q$ of negative eigenvalues, and the same number of zero eigenvalues. The pair $(p,q)$ is called the signature of the form.

Proof. Write $r = p+q$ for the rank, which is unchanged by multiplication by invertible matrices, so only $p$ needs proving. Let $V^+$ be a $p$ -dimensional subspace on which $Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x} > 0$ for $\mathbf{x}\neq\mathbf{0}$ — the span of the eigenvectors with positive eigenvalues works, since on it $Q = \sum_{\lambda_k>0}\lambda_ky_k^{2}$ . Let $V^{\leq}$ be the span of the remaining eigenvectors, of dimension $n-p$ , on which $Q\leq 0$ . Any subspace $U$ with $Q>0$ on $U\setminus\{\mathbf{0}\}$ must meet $V^{\leq}$ only in $\mathbf{0}$ , so $\dim U + (n-p)\leq n$ , i.e. $\dim U\leq p$ . Thus $p$ is the largest dimension of a subspace on which $Q$ is strictly positive — a description that mentions only the function $Q$ and the linear structure. Since $\mathbf{x} = P\mathbf{y}$ is a linear bijection carrying such subspaces for $P^TAP$ to such subspaces for $A$ and back, the two maxima agree. The same argument with $Q$ replaced by $-Q$ handles $q$ .∎

Completing the square is the hand method for finding the signature; it produces a congruence without any eigenvalue computation.

Example 11.23 (Diagonalising a form by completing the square). Write $Q(x,y) = 2x^{2} + 4xy + 5y^{2}$ as a sum of squares, and do the same for $R(x,y) = x^{2} - 4xy + 3y^{2}$ .

Solution.

Group the terms containing $x$ : $2x^{2}+4xy = 2(x^{2}+2xy) = 2(x+y)^2 - 2y^{2}$ .
Hence $Q = 2(x+y)^2 - 2y^{2} + 5y^{2} = 2(x+y)^2 + 3y^{2}$ . With $u = x+y$ , $v = y$ , $Q = 2u^{2}+3v^{2}$ : signature $(2,0)$ , and $Q>0$ unless $u=v=0$ , i.e. unless $x=y=0$ .
For $R$ : $x^{2}-4xy = (x-2y)^2 - 4y^{2}$ , so $R = (x-2y)^2 - 4y^{2} + 3y^{2} = (x-2y)^2 - y^{2}$ . Signature $(1,1)$ : $R$ takes both signs, e.g. $R(1,0) = 1$ and $R(2,1) = -1$ .
Sanity check against the matrices. $Q$ has $A = \begin{pmatrix}2&2\\2&5\end{pmatrix}$ with $\det A = 6 > 0$ and $\operatorname{tr}A = 7 > 0$ , so both eigenvalues are positive — consistent with signature $(2,0)$ . $R$ has $A = \begin{pmatrix}1&-2\\-2&3\end{pmatrix}$ with $\det A = -1 < 0$ , so the eigenvalues have opposite signs, consistent with $(1,1)$ .

□

Intuition. A quadratic form is a landscape over the plane with the origin pinned at height zero. $2u^{2}+3v^{2}$ is a bowl: every direction climbs, steeper along $v$ than along $u$ . $u^{2}-v^{2}$ is a mountain pass: climb along $u$ , descend along $v$ . The signature $(p,q)$ counts the climbing directions and the descending directions, and Sylvester's law says that count is a property of the landscape, not of the coordinates you happened to describe it in.

Pitfall. When reading off the matrix of a form, the off-diagonal entry is half the cross-term coefficient. For $Q = 5x^{2}+4xy+2y^{2}$ the matrix is $\begin{pmatrix}5&2\\2&2\end{pmatrix}$ , not $\begin{pmatrix}5&4\\4&2\end{pmatrix}$ . Putting the whole coefficient in changes the eigenvalues and can change the classification.

Exercises for §11.4 →

11.5Principal axes and the geometry of level sets

Completing the square uses a general invertible change of variable and distorts angles and lengths. If instead the change of variable is orthogonal, the picture is rotated rather than deformed, so geometric features — axes, lengths, angles — survive. That is the content of the principal axes theorem, which is the Spectral Theorem restated for forms.

Theorem 11.24 (Principal axes theorem). Let $Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x}$ with $A$ symmetric, and let $A = Q\Lambda Q^T$ be an orthogonal diagonalisation with eigenvalues $\lambda_1,\dots,\lambda_n$ . Under the orthogonal change of variable $\mathbf{x} = Q\mathbf{y}$ ,

Q(\mathbf{x}) = \lambda_1y_1^2 + \lambda_2y_2^2 + \cdots + \lambda_ny_n^2,

a form with no cross terms. The columns of $Q$ are the principal axes of the form.

Proof. By the change-of-variable proposition the new matrix is $Q^TAQ = Q^T(Q\Lambda Q^T)Q = \Lambda$ , using $Q^TQ = I$ twice. A diagonal matrix has no off-diagonal entries, so the form is $\sum_k\lambda_ky_k^{2}$ .∎

Since $Q$ is orthogonal, $\|\mathbf{x}\| = \|Q\mathbf{y}\| = \|\mathbf{y}\|$ : the change of variable is a rotation (or a rotation combined with a reflection, when $\det Q = -1$ ). The coordinate $y_k$ is the component of $\mathbf{x}$ along $\mathbf{q}_k$ , namely $y_k = \mathbf{q}_k\cdot\mathbf{x}$ . A level set $Q(\mathbf{x}) = c$ is therefore congruent, as a geometric figure, to $\sum_k\lambda_ky_k^{2} = c$ , and the eigenvalues classify it.

Corollary 11.25 (Classification of central conics). For $A$ symmetric $2\times 2$ with eigenvalues $\lambda_1,\lambda_2$ , the curve $\mathbf{x}^TA\mathbf{x} = c$ with $c>0$ is:

an ellipse if $\lambda_1,\lambda_2 > 0$ , with semi-axes $\sqrt{c/\lambda_1}$ and $\sqrt{c/\lambda_2}$ along $\mathbf{q}_1,\mathbf{q}_2$ ;
a hyperbola if $\lambda_1,\lambda_2$ have opposite signs, with axis along the eigenvector for the positive eigenvalue;
a pair of parallel lines if exactly one eigenvalue is zero and the other is positive, and the empty set if the nonzero eigenvalue is negative;
the empty set if $\lambda_1,\lambda_2<0$ , and the single point $\mathbf{0}$ if $c = 0$ with both eigenvalues of the same sign.

Proof. In principal axes the equation reads $\lambda_1y_1^2+\lambda_2y_2^2 = c$ . Dividing by $c>0$ gives $\frac{y_1^2}{c/\lambda_1} + \frac{y_2^2}{c/\lambda_2} = 1$ whenever both eigenvalues are nonzero; this is the standard equation of an ellipse when both denominators are positive and of a hyperbola when one is negative. If both $\lambda_k<0$ the left side is never positive. If $\lambda_2 = 0$ the equation is $\lambda_1y_1^2 = c$ , which is $y_1 = \pm\sqrt{c/\lambda_1}$ — two parallel lines — when $\lambda_1>0$ , and has no solution when $\lambda_1<0$ .∎

Because $\det A = \lambda_1\lambda_2$ , the familiar discriminant test falls out: for $ax^{2}+2bxy+cy^{2} = k$ , the curve is an ellipse when $ac - b^{2} > 0$ , a hyperbola when $ac-b^{2}<0$ , and degenerate when $ac-b^{2} = 0$ . The eigenvalue version tells you more, though: it also gives the directions and the lengths of the axes.

Example 11.26 (A full conic classification). Classify and sketch $5x^{2} - 4xy + 5y^{2} = 21$ .

Solution.

Matrix of the form: $a = 5$ , $c = 5$ , and the cross coefficient $-4$ halves to $b = -2$ , so $A = \begin{pmatrix}5&-2\\-2&5\end{pmatrix}$ .
Eigenvalues: $\operatorname{tr}A = 10$ , $\det A = 25-4 = 21$ , so $\lambda^{2}-10\lambda+21 = 0$ , giving $\lambda_1 = 3$ and $\lambda_2 = 7$ . Both positive, so the curve is an ellipse.
Principal axes: for $\lambda = 3$ , $(A-3I) = \begin{pmatrix}2&-2\\-2&2\end{pmatrix}$ forces $x=y$ , so $\mathbf{q}_1 = \tfrac{1}{\sqrt2}\langle 1,1\rangle$ ; for $\lambda = 7$ , $x = -y$ and $\mathbf{q}_2 = \tfrac{1}{\sqrt2}\langle 1,-1\rangle$ .
In principal axes the equation is $3y_1^2 + 7y_2^2 = 21$ , i.e.

\frac{y_1^2}{7} + \frac{y_2^2}{3} = 1.

The semi-major axis has length $\sqrt7$ and points along $\langle 1,1\rangle$ ; the semi-minor axis has length $\sqrt3$ along $\langle 1,-1\rangle$ . The ellipse is the standard one rotated by $45^\circ$ . 5. Sanity check: the point $\sqrt7\,\mathbf{q}_1 = \left\langle\sqrt{7/2},\sqrt{7/2}\right\rangle$ should lie on the curve. Substituting, $5\cdot\tfrac72 - 4\cdot\tfrac72 + 5\cdot\tfrac72 = \tfrac72(5-4+5) = \tfrac72\cdot 6 = 21$ . Correct.□

Example 11.27 (A hyperbola). Classify $x^{2} + 4xy + y^{2} = 3$ and find its vertices.

Solution.

$A = \begin{pmatrix}1&2\\2&1\end{pmatrix}$ , with $\operatorname{tr}A = 2$ and $\det A = 1-4 = -3$ , so $\lambda^{2}-2\lambda-3 = 0$ and the eigenvalues are $3$ and $-1$ . Opposite signs: a hyperbola.
Eigenvectors: $\lambda = 3$ gives $x = y$ , so $\mathbf{q}_1 = \tfrac{1}{\sqrt2}\langle 1,1\rangle$ ; $\lambda = -1$ gives $x = -y$ , so $\mathbf{q}_2 = \tfrac{1}{\sqrt2}\langle 1,-1\rangle$ .
Principal-axis form: $3y_1^2 - y_2^2 = 3$ , i.e. $y_1^2 - \dfrac{y_2^2}{3} = 1$ . The transverse axis is the $y_1$ -axis, in the direction $\langle 1,1\rangle$ , and the vertices are at $y_1 = \pm 1$ , $y_2 = 0$ .
Back in the original coordinates the vertices are $\pm\mathbf{q}_1 = \pm\left\langle \tfrac{1}{\sqrt2},\tfrac{1}{\sqrt2}\right\rangle$ .
Sanity check: at $\left\langle \tfrac{1}{\sqrt2},\tfrac{1}{\sqrt2}\right\rangle$ the form is $\tfrac12 + 4\cdot\tfrac12 + \tfrac12 = 3$ . On the curve, as required. The asymptotes are $y_2 = \pm\sqrt3\,y_1$ , at $60^\circ$ to the transverse axis.

□

Example 11.28 (A quadric surface). Classify the surface $2x^{2}+2y^{2}+2z^{2}+2xy+2xz+2yz = 4$ .

Solution.

Each cross coefficient $2$ halves to $1$ , so $A = \begin{pmatrix}2&1&1\\1&2&1\\1&1&2\end{pmatrix} = I + J$ , where $J$ is the all-ones matrix.
$J$ has rank $1$ : its eigenvalues are $3$ (eigenvector $\langle 1,1,1\rangle$ , since each row sums to $3$ ) and $0$ twice (the plane $x+y+z = 0$ ). Adding $I$ shifts every eigenvalue by $1$ , so $A$ has eigenvalues $4$ and $1,1$ .
All eigenvalues are positive, so the surface is an ellipsoid. In principal axes, $4y_1^2 + y_2^2 + y_3^2 = 4$ , i.e.

\frac{y_1^2}{1} + \frac{y_2^2}{4} + \frac{y_3^2}{4} = 1.

The short axis has length $1$ along $\tfrac{1}{\sqrt3}\langle 1,1,1\rangle$ ; in the perpendicular plane $x+y+z=0$ the eigenvalue is $1$ in every direction, so the cross-section is a circle of radius $2$ . The surface is a prolate spheroid flattened along the diagonal.
Sanity check: $\operatorname{tr}A = 6 = 4+1+1$ , and the point $\tfrac{1}{\sqrt3}\langle 1,1,1\rangle$ gives $2\cdot 3\cdot\tfrac13 + 2\cdot 3\cdot\tfrac13 = 2+2 = 4$ , on the surface as predicted.

□

Remark. For a $2\times 2$ form the rotation angle can be read off without finding eigenvectors: the principal axes make an angle $\theta$ with the coordinate axes where

\tan 2\theta = \frac{2b}{a-c}.

When $a = c$ the denominator vanishes and $\theta = 45^\circ$ , which is why both worked examples above rotated by exactly $45^\circ$ .

Pitfall. The principal axes theorem applies to central conics, those with no linear terms. A curve such as $x^{2}+4xy+y^{2}+6x-2y = 3$ needs a translation first — complete the square in the linear terms to move the centre to the origin — and only then a rotation. Parabolas arise precisely when one eigenvalue is zero and a linear term survives the translation.

Exercises for §11.5 →

11.6Constrained optimisation and the Rayleigh quotient

A quadratic form has no maximum on all of $\mathbb{R}^n$ : scaling $\mathbf{x}$ by $t$ scales $Q(\mathbf{x})$ by $t^{2}$ . The meaningful question is the constrained one — how large can $Q$ be on the unit sphere? The answer is the largest eigenvalue, and the proof is two lines once the form is in principal axes.

Definition 11.29 (Rayleigh quotient). For a symmetric $A$ and $\mathbf{x}\neq\mathbf{0}$ , the Rayleigh quotient is

R(\mathbf{x}) = \frac{\mathbf{x}^TA\mathbf{x}}{\mathbf{x}^T\mathbf{x}}.

It is unchanged by scaling, $R(t\mathbf{x}) = R(\mathbf{x})$ for $t\neq 0$ , so its values are exactly the values of $\mathbf{x}^TA\mathbf{x}$ on the unit sphere.

Theorem 11.30 (Extreme values of a quadratic form on the unit sphere). Let $A$ be real symmetric with eigenvalues ordered $\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_n$ and corresponding orthonormal eigenvectors $\mathbf{q}_1,\dots,\mathbf{q}_n$ . Then

\max_{\|\mathbf{x}\|=1}\mathbf{x}^TA\mathbf{x} = \lambda_1, \qquad \min_{\|\mathbf{x}\|=1}\mathbf{x}^TA\mathbf{x} = \lambda_n,

attained at $\mathbf{x} = \mathbf{q}_1$ and $\mathbf{x} = \mathbf{q}_n$ respectively. Consequently, for every $\mathbf{x}$ ,

\lambda_n\|\mathbf{x}\|^2 \leq \mathbf{x}^TA\mathbf{x} \leq \lambda_1\|\mathbf{x}\|^2.

Proof. Write $\mathbf{x} = Q\mathbf{y}$ , so that $\|\mathbf{y}\| = \|\mathbf{x}\| = 1$ and $\mathbf{x}^TA\mathbf{x} = \sum_k\lambda_ky_k^{2}$ by the principal axes theorem. Since $\sum_ky_k^{2} = 1$ with every $y_k^{2}\geq 0$ , the quantity $\sum_k\lambda_ky_k^{2}$ is a weighted average of the eigenvalues. An average of numbers lies between the smallest and the largest:

\lambda_n = \lambda_n\sum_ky_k^2 \leq \sum_k\lambda_ky_k^2 \leq \lambda_1\sum_ky_k^2 = \lambda_1.

Both bounds are attained: $\mathbf{x} = \mathbf{q}_1$ corresponds to $\mathbf{y} = \mathbf{e}_1$ , giving $\lambda_1$ , and $\mathbf{x} = \mathbf{q}_n$ gives $\lambda_n$ . The unconstrained inequality follows by applying the constrained one to $\mathbf{x}/\|\mathbf{x}\|$ and multiplying through by $\|\mathbf{x}\|^2$ ; it also holds trivially at $\mathbf{x} = \mathbf{0}$ .∎

The proof explains why the eigenvector is optimal in a way that Lagrange multipliers do not: in principal axes the constraint is "distribute a total weight of one among the eigenvalues", and the best you can do is put all the weight on the largest. Incidentally, Lagrange multipliers give the same answer and reveal where the name comes from — maximising $\mathbf{x}^TA\mathbf{x}$ subject to $\mathbf{x}^T\mathbf{x} = 1$ has stationarity condition $A\mathbf{x} = \lambda\mathbf{x}$ , so the multiplier is the eigenvalue.

Restricting further to directions orthogonal to the top eigenvector produces the next eigenvalue, and iterating gives all of them.

Corollary 11.31 (The next eigenvalue). With notation as above,

\max\left\{\mathbf{x}^TA\mathbf{x} : \|\mathbf{x}\| = 1,\ \mathbf{x}\cdot\mathbf{q}_1 = 0\right\} = \lambda_2,

attained at $\mathbf{x} = \mathbf{q}_2$ . More generally, maximising over unit vectors orthogonal to $\mathbf{q}_1,\dots,\mathbf{q}_{k-1}$ gives $\lambda_k$ .

Proof. The constraint $\mathbf{x}\cdot\mathbf{q}_1 = 0$ says $y_1 = 0$ , so the average $\sum_{k\geq 2}\lambda_ky_k^{2}$ now runs over $\lambda_2,\dots,\lambda_n$ only, and the same averaging argument bounds it by $\lambda_2$ , attained at $\mathbf{y} = \mathbf{e}_2$ . The general case is identical with the first $k-1$ coordinates set to zero.∎

Theorem 11.32 (Courant–Fischer min–max, statement). For real symmetric $A$ with $\lambda_1\geq\cdots\geq\lambda_n$ and each $k$ ,

\lambda_k = \max_{\dim U = k}\ \min_{\mathbf{x}\in U,\ \|\mathbf{x}\|=1}\mathbf{x}^TA\mathbf{x} = \min_{\dim V = n-k+1}\ \max_{\mathbf{x}\in V,\ \|\mathbf{x}\|=1}\mathbf{x}^TA\mathbf{x},

the extrema running over subspaces of the stated dimension.

The previous corollary needed the eigenvectors $\mathbf{q}_1,\dots,\mathbf{q}_{k-1}$ in order to describe $\lambda_k$ ; Courant–Fischer removes them, characterising every eigenvalue purely by the form. That is what makes it the tool for comparing spectra: it yields at once that the eigenvalues of $A$ increase when $A$ is replaced by $A+B$ with $B$ positive semidefinite, and the interlacing theorem for the eigenvalues of a principal submatrix. The proof is the same averaging argument applied to a dimension count: any $k$ -dimensional $U$ must intersect the $(n-k+1)$ -dimensional span of $\mathbf{q}_k,\dots,\mathbf{q}_n$ nontrivially, which caps the inner minimum at $\lambda_k$ , and $U = \operatorname{span}\{\mathbf{q}_1,\dots,\mathbf{q}_k\}$ attains it.

Intuition. Stand on the surface $z = Q(x,y)$ above the unit circle and walk once around. The height you trace is a weighted average of $\lambda_1$ and $\lambda_2$ , and you are highest exactly over the direction $\mathbf{q}_1$ and lowest over $\mathbf{q}_2$ . For $A = \begin{pmatrix}3&1\\1&3\end{pmatrix}$ the walk climbs to $4$ over the $45^\circ$ direction and drops to $2$ over the $135^\circ$ direction — never outside $[2,4]$ , because you can never be higher than the highest point or lower than the lowest.

Example 11.33 (Maximum and minimum on the unit sphere). Find the maximum and minimum of $Q(x,y) = 5x^{2} + 4xy + 5y^{2}$ subject to $x^{2}+y^{2} = 1$ , and the points where they occur.

Solution.

$A = \begin{pmatrix}5&2\\2&5\end{pmatrix}$ , with $\operatorname{tr}A = 10$ and $\det A = 25-4 = 21$ .
$\lambda^{2}-10\lambda+21 = 0$ gives $\lambda = 7$ and $\lambda = 3$ .
Maximum $= 7$ , attained at the unit eigenvector for $7$ : $(A-7I)\mathbf{v} = \begin{pmatrix}-2&2\\2&-2\end{pmatrix}\mathbf{v} = \mathbf{0}$ gives $x=y$ , so at $\pm\tfrac{1}{\sqrt2}\langle 1,1\rangle$ .
Minimum $= 3$ , attained at $\pm\tfrac{1}{\sqrt2}\langle 1,-1\rangle$ .
Sanity check: at $\tfrac{1}{\sqrt2}\langle 1,1\rangle$ , $Q = 5\cdot\tfrac12 + 4\cdot\tfrac12 + 5\cdot\tfrac12 = 7$ ; at $\langle 1,0\rangle$ , $Q = 5$ , comfortably between $3$ and $7$ .

□

Example 11.34 (A three-variable constrained maximum). Maximise $\mathbf{x}^TA\mathbf{x}$ on the unit sphere in $\mathbb{R}^3$ for $A = \begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$ , and find the maximum subject to the extra constraint $\mathbf{x}\cdot\mathbf{q}_1 = 0$ .

Solution.

Expand $\det(A-\lambda I)$ along the first row: $(2-\lambda)\left[(2-\lambda)^2-1\right] + 1\cdot\left[-(2-\lambda)\right] = (2-\lambda)\left[(2-\lambda)^2-2\right]$ .
The roots are $\lambda = 2$ and $(2-\lambda)^2 = 2$ , i.e. $\lambda = 2\pm\sqrt2$ . Ordered: $\lambda_1 = 2+\sqrt2$ , $\lambda_2 = 2$ , $\lambda_3 = 2-\sqrt2$ .
The maximum on the unit sphere is $\lambda_1 = 2+\sqrt2 \approx 3.414$ . Its eigenvector solves $(A-(2+\sqrt2)I)\mathbf{v} = \mathbf{0}$ : the first row gives $-\sqrt2\,v_1 - v_2 = 0$ , so $v_2 = -\sqrt2\,v_1$ , and by symmetry of the second and third rows $v_3 = v_1$ . Thus $\mathbf{q}_1 \parallel \langle 1,-\sqrt2,1\rangle$ , of length $2$ , so $\mathbf{q}_1 = \tfrac12\langle 1,-\sqrt2,1\rangle$ .
With the extra constraint the maximum drops to $\lambda_2 = 2$ , attained at the unit eigenvector for $2$ , which is $\tfrac{1}{\sqrt2}\langle 1,0,-1\rangle$ (check: $A\langle 1,0,-1\rangle = \langle 2,0,-2\rangle$ ).
Sanity check: $\operatorname{tr}A = 6$ and $(2+\sqrt2)+2+(2-\sqrt2) = 6$ . Also $\langle 1,-\sqrt2,1\rangle\cdot\langle 1,0,-1\rangle = 1+0-1 = 0$ , as the orthogonality lemma requires.

□

Remark. The same extremal problem for a non-symmetric or rectangular matrix asks for the maximum of $\|A\mathbf{x}\|$ on the unit sphere. Since $\|A\mathbf{x}\|^2 = \mathbf{x}^T(A^TA)\mathbf{x}$ and $A^TA$ is symmetric, this theorem answers it: the maximum is $\sqrt{\lambda_{\max}(A^TA)}$ , which is the largest singular value of $A$ . That observation is the doorway to the SVD chapter.

Pitfall. The maximum of $\mathbf{x}^TA\mathbf{x}$ on the unit sphere is $\lambda_{\max}$ , not $|\lambda|_{\max}$ . For $A = \operatorname{diag}(3,-5)$ the maximum is $3$ and the minimum is $-5$ ; the number $5$ is the spectral norm $\max_k|\lambda_k|$ , which answers a different question — the maximum of $\|A\mathbf{x}\|$ , not of $\mathbf{x}^TA\mathbf{x}$ .

Exercises for §11.6 →

11.7Positive definite matrices

Among symmetric matrices, the positive definite ones play the role that positive numbers play among the reals: they have square roots and inverses of the same kind, they define inner products and distances, and they mark the minima of smooth functions. This section gathers the five standard tests and proves them equivalent.

Definition 11.35 (Definiteness). Let $A$ be a real symmetric $n\times n$ matrix and $Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x}$ . Then $A$ (and $Q$ ) is

positive definite if $Q(\mathbf{x}) > 0$ for all $\mathbf{x}\neq\mathbf{0}$ ;
positive semidefinite if $Q(\mathbf{x})\geq 0$ for all $\mathbf{x}$ ;
negative definite if $Q(\mathbf{x}) < 0$ for all $\mathbf{x}\neq\mathbf{0}$ , and negative semidefinite if $Q(\mathbf{x})\leq 0$ for all $\mathbf{x}$ ;
indefinite if $Q$ takes both a positive and a negative value.

Negative definiteness needs no separate theory: $A$ is negative definite exactly when $-A$ is positive definite, so every test below applies after a sign flip. In signature language, positive definite means $(p,q) = (n,0)$ , positive semidefinite means $q = 0$ , and indefinite means $p>0$ and $q>0$ .

Theorem 11.36 (Five equivalent tests for positive definiteness). For a real symmetric $n\times n$ matrix $A$ , the following are equivalent.

$\mathbf{x}^TA\mathbf{x} > 0$ for every $\mathbf{x}\neq\mathbf{0}$ .
Every eigenvalue of $A$ is positive.
(Sylvester's criterion) Every leading principal minor is positive: $\det A_k > 0$ for $k = 1,\dots,n$ , where $A_k$ is the upper-left $k\times k$ submatrix.
Gaussian elimination on $A$ needs no row exchanges and produces $n$ positive pivots.
$A = B^TB$ for some matrix $B$ with linearly independent columns.

Proof. (1) $\Leftrightarrow$ (2). By the Rayleigh theorem, $\min_{\|\mathbf{x}\|=1}\mathbf{x}^TA\mathbf{x} = \lambda_{\min}$ . So the form is positive on all unit vectors — equivalently, by scaling, on all nonzero vectors — exactly when $\lambda_{\min}>0$ .

(2) $\Rightarrow$ (5). Write $A = Q\Lambda Q^T$ and put $B = \Lambda^{1/2}Q^T$ , where $\Lambda^{1/2} = \operatorname{diag}(\sqrt{\lambda_1},\dots,\sqrt{\lambda_n})$ is real because the eigenvalues are positive. Then $B^TB = Q\Lambda^{1/2}\Lambda^{1/2}Q^T = Q\Lambda Q^T = A$ , and $B$ is invertible (a product of invertible matrices), so its columns are independent.

(5) $\Rightarrow$ (1). $\mathbf{x}^TA\mathbf{x} = \mathbf{x}^TB^TB\mathbf{x} = \|B\mathbf{x}\|^2 \geq 0$ , with equality only if $B\mathbf{x} = \mathbf{0}$ ; independent columns force $\mathbf{x} = \mathbf{0}$ .

(1) $\Rightarrow$ (3). Fix $k$ and let $\mathbf{u}\in\mathbb{R}^k$ be nonzero. Pad it with zeros to $\mathbf{x} = \begin{pmatrix}\mathbf{u}\\\mathbf{0}\end{pmatrix}\in\mathbb{R}^n$ . Then $\mathbf{x}^TA\mathbf{x} = \mathbf{u}^TA_k\mathbf{u}$ , because the zero entries kill every term involving a row or column beyond the $k$ -th. So $A_k$ is itself positive definite, hence by (1) $\Leftrightarrow$ (2) all of its eigenvalues are positive, and $\det A_k$ — their product — is positive.

(3) $\Rightarrow$ (4). Elimination without row exchanges produces the factorisation $A = LDL^T$ with $L$ unit lower triangular and $D = \operatorname{diag}(d_1,\dots,d_n)$ the pivots, and comparing leading $k\times k$ blocks gives $A_k = L_kD_kL_k^T$ , hence $\det A_k = d_1d_2\cdots d_k$ . So

d_k = \frac{\det A_k}{\det A_{k-1}} \quad (\det A_0 := 1),

a ratio of positive numbers, hence positive. Positivity of $\det A_{k-1}$ also guarantees the $k$ -th pivot is nonzero at each stage, so no row exchange is ever needed.

(4) $\Rightarrow$ (1). With $A = LDL^T$ and all $d_k>0$ , substitute $\mathbf{z} = L^T\mathbf{x}$ :

\mathbf{x}^TA\mathbf{x} = \mathbf{x}^TLDL^T\mathbf{x} = \mathbf{z}^TD\mathbf{z} = \sum_{k=1}^nd_kz_k^2.

This is a sum of nonnegative terms, zero only when every $z_k = 0$ ; and $L^T$ is invertible, so $\mathbf{z} = \mathbf{0}$ forces $\mathbf{x} = \mathbf{0}$ .

The cycle (1) $\Rightarrow$ (3) $\Rightarrow$ (4) $\Rightarrow$ (1) together with (1) $\Leftrightarrow$ (2) and (2) $\Rightarrow$ (5) $\Rightarrow$ (1) proves all five equivalent.∎

Each test is best for something. Test (1) is the definition and the one that transfers to infinite dimensions. Test (2) gives the most information — the actual eigenvalues — but is the most work. Test (3) is fastest by hand for $n\leq 3$ : three determinants and no root-finding. Test (4) is what elimination already computes, so it is free if you were solving a system anyway. Test (5) is the structural one: it says positive definite matrices are exactly the Gram matrices of independent vectors, which is why $A^TA$ appears in least squares and why kernel matrices in machine learning are positive semidefinite.

Proposition 11.37 (Properties of positive definite matrices). Let $A,B$ be positive definite $n\times n$ matrices and $c>0$ . Then:

$A$ is invertible, and $A^{-1}$ is positive definite;
every diagonal entry $a_{ii}>0$ , and more generally every principal submatrix is positive definite;
$A+B$ and $cA$ are positive definite;
$P^TAP$ is positive definite for every invertible $P$ ;
$A$ has a unique positive definite square root $A^{1/2}$ ;
the largest entry of $A$ in absolute value sits on the diagonal.

Proof. (1) No eigenvalue is zero, so $A$ is invertible; the eigenvalues of $A^{-1}$ are the reciprocals $1/\lambda_k>0$ . (2) Take $\mathbf{x} = \mathbf{e}_i$ to get $a_{ii} = \mathbf{e}_i^TA\mathbf{e}_i>0$ ; the padding argument from the theorem, applied to an arbitrary index set rather than the first $k$ , handles principal submatrices. (3) $\mathbf{x}^T(A+B)\mathbf{x} = \mathbf{x}^TA\mathbf{x}+\mathbf{x}^TB\mathbf{x} > 0$ for $\mathbf{x}\neq\mathbf{0}$ . (4) $\mathbf{x}^TP^TAP\mathbf{x} = (P\mathbf{x})^TA(P\mathbf{x})>0$ since $P\mathbf{x}\neq\mathbf{0}$ when $\mathbf{x}\neq\mathbf{0}$ . (5) Existence is $\sum_k\sqrt{\lambda_k}P_k$ from the spectral decomposition; for uniqueness, a positive definite square root $S$ commutes with $A = S^{2}$ , hence preserves each eigenspace of $A$ , and on the eigenspace for $\lambda$ it must be a positive definite matrix squaring to $\lambda I$ , so it is $\sqrt{\lambda}I$ there. (6) Suppose $|a_{ij}|\geq a_{ii}, a_{jj}$ for some $i\neq j$ . The $2\times 2$ principal submatrix $\begin{pmatrix}a_{ii}&a_{ij}\\a_{ij}&a_{jj}\end{pmatrix}$ must be positive definite by (2), so $a_{ii}a_{jj} - a_{ij}^2 > 0$ , contradicting $a_{ij}^2 \geq a_{ii}a_{jj}$ .∎

Intuition. A positive definite $A$ defines a new way to measure length, $\|\mathbf{x}\|_A = \sqrt{\mathbf{x}^TA\mathbf{x}}$ , and a new inner product $\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^TA\mathbf{y}$ . Definiteness is exactly the axiom "the length of a nonzero vector is positive"; without it you would have nonzero vectors of length zero, and geometry would collapse. The unit "circle" in this geometry is the ellipse $\mathbf{x}^TA\mathbf{x} = 1$ , whose axes are the eigenvectors and whose semi-axes are $1/\sqrt{\lambda_k}$ — large eigenvalue, short axis, because that direction is expensive.

Method 11.38 (Classifying a symmetric matrix by hand).

If any diagonal entry is $\leq 0$ , it cannot be positive definite. If the diagonal has both signs, it is indefinite; stop.
Compute the leading principal minors $\det A_1,\dots,\det A_n$ .
All positive $\Rightarrow$ positive definite. Alternating $\det A_1<0$ , $\det A_2>0$ , $\det A_3<0,\dots$ $\Rightarrow$ negative definite (apply Sylvester to $-A$ ).
Some leading minor negative in the wrong place, or $\det A_n<0$ with $n$ even $\Rightarrow$ indefinite.
If a leading minor is zero, Sylvester is inconclusive: compute the eigenvalues, or check all principal minors (not just the leading ones) for the semidefinite question.

Example 11.39 (Sylvester's criterion in action). Classify $A = \begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$ and $B = \begin{pmatrix}1&2\\2&3\end{pmatrix}$ .

Solution.

$\det A_1 = 2 > 0$ .
$\det A_2 = \begin{vmatrix}2&-1\\-1&2\end{vmatrix} = 4-1 = 3 > 0$ .
$\det A_3 = 2(4-1) - (-1)(-2-0) + 0 = 6 - 2 = 4 > 0$ . All three positive, so $A$ is positive definite.
Cross-check with pivots: $d_1 = 2$ , $d_2 = \det A_2/\det A_1 = 3/2$ , $d_3 = \det A_3/\det A_2 = 4/3$ . All positive, agreeing with test (4). We computed the eigenvalues of this matrix earlier: $2\pm\sqrt2$ and $2$ , all positive, agreeing with test (2).
For $B$ : $\det B_1 = 1>0$ but $\det B_2 = 3-4 = -1 < 0$ . Sylvester fails, so $B$ is not positive definite; since $\det B<0$ the two eigenvalues have opposite signs, so $B$ is indefinite. Indeed $Q(1,0) = 1 > 0$ while $Q(1,-1) = 1-4+3\cdot 1 = 0$ and $Q(2,-1) = 4-8+3 = -1 < 0$ .

□

Example 11.40 (A definiteness threshold). For which real $k$ is $A = \begin{pmatrix}1&1&1\\1&2&2\\1&2&k\end{pmatrix}$ positive definite?

Solution.

$\det A_1 = 1 > 0$ always.
$\det A_2 = 2 - 1 = 1 > 0$ always.
$\det A_3$ : expanding along the first row, $1(2k-4) - 1(k-2) + 1(2-2) = 2k-4-k+2 = k-2$ .
Sylvester's criterion therefore holds exactly when $k - 2 > 0$ , i.e. $k > 2$ .
Sanity check at the threshold: for $k = 2$ the matrix is singular, and $\mathbf{x} = \langle 0,1,-1\rangle$ gives $A\mathbf{x} = \langle 0,0,0\rangle$ , so $Q(\mathbf{x}) = 0$ — positive semidefinite but not definite. For $k<2$ , $\det A_3<0$ and the same vector makes $Q$ negative: at $k = 1$ , $Q(0,1,-1) = 2 - 4 + 1 = -1$ .

□

Pitfall. Sylvester's criterion does not extend to the semidefinite case by relaxing " $>0$ " to " $\geq 0$ ". The matrix $\begin{pmatrix}0&0\\0&-1\end{pmatrix}$ has leading minors $0$ and $0$ , both nonnegative, yet $Q(0,1) = -1 < 0$ , so it is not positive semidefinite. The correct semidefinite test requires all $2^n-1$ principal minors — every symmetric choice of rows and columns, not just the leading ones — to be nonnegative. In practice, compute the eigenvalues instead.

Pitfall. Positive entries do not make a matrix positive definite, and positive definite matrices may have negative entries. $\begin{pmatrix}1&2\\2&1\end{pmatrix}$ has all entries positive and eigenvalues $3,-1$ ; $\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$ has negative entries and eigenvalues $1,3$ . Likewise $\det A > 0$ alone is not enough: $-I$ in $\mathbb{R}^2$ has determinant $1$ .

Exercises for §11.7 →

11.8The Cholesky factorisation

Test (5) above produces some $B$ with $A = B^TB$ . Demanding that $B$ be triangular pins it down completely and yields the algorithm that numerical software actually uses to test and exploit positive definiteness.

Theorem 11.41 (Cholesky factorisation). A real symmetric matrix $A$ is positive definite if and only if it can be written

A = LL^T

with $L$ lower triangular with strictly positive diagonal entries. This $L$ is unique.

Proof. ( $\Leftarrow$ ) If $A = LL^T$ with $L$ invertible (its diagonal entries are nonzero), then $A = B^TB$ with $B = L^T$ , whose columns are independent, so $A$ is positive definite by test (5).

( $\Rightarrow$ ) Induct on $n$ . For $n=1$ , $A = (a)$ with $a>0$ and $L = (\sqrt a)$ , uniquely. For $n\geq 2$ write

A = \begin{pmatrix}a & \mathbf{v}^T\\ \mathbf{v} & A'\end{pmatrix}

with $a = a_{11}>0$ . Set $\ell = \sqrt a$ and $\mathbf{w} = \mathbf{v}/\ell$ . A direct multiplication gives

\begin{pmatrix}\ell & \mathbf{0}^T\\ \mathbf{w} & I\end{pmatrix}\begin{pmatrix}1 & \mathbf{0}^T\\ \mathbf{0} & S\end{pmatrix}\begin{pmatrix}\ell & \mathbf{w}^T\\ \mathbf{0} & I\end{pmatrix} = \begin{pmatrix}a & \mathbf{v}^T\\ \mathbf{v} & \mathbf{w}\mathbf{w}^T + S\end{pmatrix},

so taking the Schur complement $S = A' - \mathbf{w}\mathbf{w}^T = A' - \mathbf{v}\mathbf{v}^T/a$ reproduces $A$ . The outer factors are invertible, so $\begin{pmatrix}1&\mathbf{0}^T\\\mathbf{0}&S\end{pmatrix}$ is congruent to $A$ and hence positive definite; therefore $S$ is a positive definite matrix of size $n-1$ . By induction $S = L'L'^T$ with $L'$ lower triangular with positive diagonal, and then

L = \begin{pmatrix}\ell & \mathbf{0}^T\\ \mathbf{w} & L'\end{pmatrix}

is lower triangular with positive diagonal and satisfies $LL^T = A$ . Uniqueness follows the same induction: comparing the $(1,1)$ entries of $A = LL^T$ forces $\ell_{11} = \sqrt{a}$ (the positive root), the first column then forces $\mathbf{w}$ , and the remaining block is the unique factorisation of $S$ .∎

Comparing $A = LL^T$ with $A = LDL^T$ from elimination shows $L_{\text{Cholesky}} = L_{\text{elim}}D^{1/2}$ , so the Cholesky diagonal entries are the square roots of the pivots: $\ell_{kk} = \sqrt{d_k}$ . Equating entries in $A = LL^T$ column by column turns the induction into a formula.

Method 11.42 (Computing the Cholesky factor). Process columns $j = 1,2,\dots,n$ in order. For each $j$ :

\ell_{jj} = \sqrt{a_{jj} - \sum_{k<j}\ell_{jk}^2}, \qquad \ell_{ij} = \frac{1}{\ell_{jj}}\left(a_{ij} - \sum_{k<j}\ell_{ik}\ell_{jk}\right) \ \ (i>j).

If at any step the quantity under the square root is zero or negative, stop: $A$ is not positive definite.

Example 11.43 (A $3\times 3$ Cholesky factorisation). Factor $A = \begin{pmatrix}4&2&-2\\2&10&2\\-2&2&6\end{pmatrix}$ as $LL^T$ .

Solution.

Column $1$ : $\ell_{11} = \sqrt{4} = 2$ , then $\ell_{21} = 2/2 = 1$ and $\ell_{31} = -2/2 = -1$ .
Column $2$ : $\ell_{22} = \sqrt{a_{22}-\ell_{21}^2} = \sqrt{10-1} = 3$ , then

\ell_{32} = \frac{a_{32}-\ell_{31}\ell_{21}}{\ell_{22}} = \frac{2-(-1)(1)}{3} = \frac{3}{3} = 1.

Column $3$ : $\ell_{33} = \sqrt{a_{33}-\ell_{31}^2-\ell_{32}^2} = \sqrt{6-1-1} = 2$ .
So $L = \begin{pmatrix}2&0&0\\1&3&0\\-1&1&2\end{pmatrix}$ . No square root of a non-positive number appeared, so $A$ is positive definite.
Sanity check: $LL^T = \begin{pmatrix}4&2&-2\\2&1+9&-1+3\\-2&-1+3&1+1+4\end{pmatrix} = \begin{pmatrix}4&2&-2\\2&10&2\\-2&2&6\end{pmatrix} = A$ . And $\det A = (\det L)^2 = (2\cdot3\cdot2)^2 = 144$ , while the pivots are $\ell_{kk}^2 = 4, 9, 4$ , whose product is also $144$ .

□

Example 11.44 (Cholesky as a definiteness test). Decide whether $A = \begin{pmatrix}1&2&3\\2&5&8\\3&8&14\end{pmatrix}$ is positive definite by attempting a Cholesky factorisation.

Solution.

Column $1$ : $\ell_{11} = 1$ , $\ell_{21} = 2$ , $\ell_{31} = 3$ .
Column $2$ : $\ell_{22} = \sqrt{5-4} = 1$ , and $\ell_{32} = (8 - 3\cdot 2)/1 = 2$ .
Column $3$ : $\ell_{33} = \sqrt{14 - 9 - 4} = \sqrt{1} = 1$ . The algorithm succeeds, so $A$ is positive definite with $L = \begin{pmatrix}1&0&0\\2&1&0\\3&2&1\end{pmatrix}$ .
Cross-check with Sylvester: $\det A_1 = 1$ , $\det A_2 = 5-4 = 1$ , and $\det A = (\det L)^2 = 1$ . All positive.
Contrast: changing the $(3,3)$ entry to $12$ would give $\ell_{33} = \sqrt{12-9-4} = \sqrt{-1}$ , and the algorithm would fail at that step, correctly reporting that the modified matrix is not positive definite.

□

Remark. Cholesky is the preferred numerical test for two reasons. It costs about $n^{3}/6$ multiplications, half of an ordinary $LU$ factorisation and far less than an eigenvalue computation, and it is backward stable without pivoting — positive definiteness itself keeps the entries of $L$ bounded, so no row exchanges are needed and no growth factor has to be monitored. Sylvester's criterion, by contrast, is a poor numerical test: computing $n$ determinants is both expensive and badly conditioned. Once $L$ is in hand, solving $A\mathbf{x} = \mathbf{b}$ costs two triangular solves, $L\mathbf{y} = \mathbf{b}$ then $L^T\mathbf{x} = \mathbf{y}$ .

Pitfall. $L$ is lower triangular but it is not a matrix of eigenvectors, and its diagonal holds the square roots of the pivots, not the square roots of the eigenvalues. Only the product $\prod\ell_{kk}^2 = \det A = \prod\lambda_k$ is shared. Also, $A = LL^T$ is not the same factorisation as $A = LDL^T$ ; the two differ by the scaling $D^{1/2}$ , and $LDL^T$ has the advantage of needing no square roots, which is why it is preferred for semidefinite or indefinite symmetric matrices.

Exercises for §11.8 →

11.9Where positive definiteness comes from

The tests above are worth having because positive definiteness is the hypothesis in a surprising number of theorems. Three sources account for most appearances: curvature, energy and covariance.

Theorem 11.45 (Second derivative test). Let $f:\mathbb{R}^n\to\mathbb{R}$ have continuous second partial derivatives, and let $\mathbf{a}$ be a critical point ( $\nabla f(\mathbf{a}) = \mathbf{0}$ ). Let $H$ be the Hessian at $\mathbf{a}$ , the symmetric matrix with entries $H_{ij} = \dfrac{\partial^{2}f}{\partial x_i\partial x_j}(\mathbf{a})$ . Then:

$H$ positive definite $\Rightarrow$ $f$ has a strict local minimum at $\mathbf{a}$ ;
$H$ negative definite $\Rightarrow$ a strict local maximum;
$H$ indefinite $\Rightarrow$ a saddle point;
$H$ singular $\Rightarrow$ the test is inconclusive.

The reason is Taylor's theorem. Near $\mathbf{a}$ ,

f(\mathbf{a}+\mathbf{h}) = f(\mathbf{a}) + \tfrac12\mathbf{h}^TH\mathbf{h} + o\!\left(\|\mathbf{h}\|^2\right),

the linear term having vanished. So the quadratic form $\mathbf{h}^TH\mathbf{h}$ is the leading behaviour, and by the Rayleigh theorem it is at least $\lambda_{\min}\|\mathbf{h}\|^2$ ; when $\lambda_{\min}>0$ that positive lower bound dominates the error term for small $\mathbf{h}$ , forcing $f(\mathbf{a}+\mathbf{h})>f(\mathbf{a})$ . The Hessian is symmetric precisely because mixed partials commute (Clairaut's theorem), which is what admits it to this chapter at all. When $H$ is singular the quadratic term vanishes in some direction and the cubic terms decide, as in $f = x^{2}+y^{4}$ versus $f = x^{2}-y^{4}$ .

Example 11.46 (Classifying critical points). Find and classify the critical points of $f(x,y) = x^{3}+y^{3}-3xy$ .

Solution.

$\nabla f = \langle 3x^{2}-3y,\ 3y^{2}-3x\rangle$ . Setting both to zero: $y = x^{2}$ and $x = y^{2} = x^{4}$ , so $x(x^{3}-1) = 0$ and $x = 0$ or $x = 1$ . The critical points are $(0,0)$ and $(1,1)$ .
$H = \begin{pmatrix}6x&-3\\-3&6y\end{pmatrix}$ .
At $(0,0)$ : $H = \begin{pmatrix}0&-3\\-3&0\end{pmatrix}$ with $\det H = -9 < 0$ , so the eigenvalues $\pm 3$ have opposite signs. Indefinite: a saddle point.
At $(1,1)$ : $H = \begin{pmatrix}6&-3\\-3&6\end{pmatrix}$ with leading minors $6>0$ and $36-9 = 27>0$ . Positive definite: a strict local minimum, of value $f(1,1) = 1+1-3 = -1$ .
Sanity check at the saddle: along $y = x$ , $f = 2x^{3}-3x^{2}$ has a local maximum at $x = 0$ ; along $y = -x$ , $f = 3x^{2} > 0$ for $x \neq 0$ , a local minimum. Two directions disagreeing is exactly what a saddle looks like.

□

Example 11.47 (A stiffness matrix). Two springs of stiffness $k_1, k_2 > 0$ are connected in series: the first joins a fixed wall to a mass at displacement $x_1$ , the second joins that mass to a second mass at displacement $x_2$ . The stored elastic energy is $E = \tfrac12k_1x_1^2 + \tfrac12k_2(x_2-x_1)^2$ . Write $E = \tfrac12\mathbf{x}^TK\mathbf{x}$ and show $K$ is positive definite.

Solution.

Expand: $2E = k_1x_1^2 + k_2x_2^2 - 2k_2x_1x_2 + k_2x_1^2 = (k_1+k_2)x_1^2 - 2k_2x_1x_2 + k_2x_2^2$ .
Halving the cross coefficient gives $K = \begin{pmatrix}k_1+k_2 & -k_2\\ -k_2 & k_2\end{pmatrix}$ .
Sylvester: $\det K_1 = k_1+k_2>0$ , and $\det K = (k_1+k_2)k_2 - k_2^2 = k_1k_2 > 0$ . Positive definite.
Structural reason: with $B = \begin{pmatrix}\sqrt{k_1}&0\\-\sqrt{k_2}&\sqrt{k_2}\end{pmatrix}$ we have $K = B^TB$ , and $B$ has independent columns because $k_1k_2\neq 0$ . This is test (5), and physically it says the energy is a sum of squares of the individual spring extensions — zero only when nothing is stretched.
Sanity check: the energy must be strictly positive whenever some spring is stretched, and indeed $\mathbf{x}^TK\mathbf{x} = k_1x_1^2 + k_2(x_2-x_1)^2 = 0$ forces $x_1 = 0$ and then $x_2 = 0$ .

□

Proposition 11.48 (Gram and covariance matrices). For any real $m\times n$ matrix $X$ , the matrix $X^TX$ is symmetric positive semidefinite, and it is positive definite exactly when the columns of $X$ are linearly independent.

Proof. Symmetry was proved among the constructions. For any $\mathbf{x}$ , $\mathbf{x}^T(X^TX)\mathbf{x} = \|X\mathbf{x}\|^2\geq 0$ , so $X^TX$ is positive semidefinite. Equality holds for some $\mathbf{x}\neq\mathbf{0}$ exactly when $X\mathbf{x} = \mathbf{0}$ has a nontrivial solution, i.e. exactly when the columns are dependent.∎

If the columns of $X$ are centred measurements of $n$ variables across $m$ observations, then

C = \frac{1}{m-1}X^TX

is the sample covariance matrix: positive semidefinite by the proposition, so all its eigenvalues are real and nonnegative, and by the Spectral Theorem it has an orthonormal eigenbasis. Those eigenvectors are the principal components and the eigenvalues are the variances along them — the maximisation $\max_{\|\mathbf{u}\|=1}\mathbf{u}^TC\mathbf{u} = \lambda_{\max}$ from the Rayleigh section is precisely the statement that the first principal component is the direction of greatest variance. Principal Component Analysis and the numerically stable route to it are developed in the Singular Value Decomposition chapter; everything it needs about $C$ is proved here.

The same proposition is the reason least squares works: the normal equations $X^TX\hat{\boldsymbol\beta} = X^T\mathbf{y}$ have a unique solution exactly when $X$ has independent columns, and then $X^TX$ is positive definite, so the critical point of the sum of squared residuals is a genuine minimum by the second derivative test — its Hessian is $2X^TX$ .

Remark. A positive definite $A$ also supplies a genuine inner product $\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^TA\mathbf{y}$ on $\mathbb{R}^n$ : bilinearity and symmetry are immediate from $A = A^T$ , and positivity is the definiteness. Every inner product on $\mathbb{R}^n$ arises this way, with $a_{ij} = \langle\mathbf{e}_i,\mathbf{e}_j\rangle$ . This is the bridge to the inner-product-spaces chapter, and it is why the Mahalanobis distance $\sqrt{(\mathbf{x}-\boldsymbol\mu)^TC^{-1}(\mathbf{x}-\boldsymbol\mu)}$ in statistics is a distance at all: $C^{-1}$ is positive definite whenever $C$ is.

Intuition. Three unrelated-looking questions — "is this critical point a minimum?", "is this structure stable?", "is this covariance matrix invertible?" — are the same question. Each asks whether a certain energy-like quantity is strictly positive in every direction. The matrix $A$ changes, the test does not, and the answer is always: check the eigenvalues, the leading minors, the pivots, or run Cholesky and see whether it finishes.

Summary (The Spectral Theorem and the definiteness tests). Everything here needs $A$ real, square and symmetric, $A^T=A$ — equivalently self-adjoint, $(A\mathbf{x})\cdot\mathbf{y}=\mathbf{x}\cdot(A\mathbf{y})$ , which is the identity every proof uses. Then all eigenvalues are real, eigenvectors for distinct eigenvalues are orthogonal, no eigenvalue is defective, and the Spectral Theorem gives $A=Q\Lambda Q^T$ with $Q$ orthogonal and $\Lambda$ real diagonal; conversely any such factorisation forces $A=A^T$ . Equivalently $A=\sum_k\lambda_kP_k$ with $P_k=\mathbf{q}_k\mathbf{q}_k^T$ orthogonal projections summing to $I$ , so $A^m=\sum_k\lambda_k^mP_k$ . For the form $Q(\mathbf{x})=\mathbf{x}^TA\mathbf{x}$ (whose symmetric matrix is unique, with off-diagonal entries half the cross-term coefficients) the orthogonal change $\mathbf{x}=Q\mathbf{y}$ removes all cross terms, leaving $\sum_k\lambda_ky_k^{2}$ : the signs of the eigenvalues classify the level sets (ellipse if both positive, hyperbola if mixed), and Sylvester's law of inertia says the signature $(p,q)$ survives any congruence $A\mapsto P^TAP$ with $P$ invertible. On the unit sphere $\max\mathbf{x}^TA\mathbf{x}=\lambda_1$ at $\mathbf{q}_1$ and $\min=\lambda_n$ at $\mathbf{q}_n$ , so $\lambda_n\|\mathbf{x}\|^2\le\mathbf{x}^TA\mathbf{x}\le\lambda_1\|\mathbf{x}\|^2$ . Finally, $A$ is positive definite iff any one of: $\mathbf{x}^TA\mathbf{x}>0$ for $\mathbf{x}\neq\mathbf{0}$ ; every eigenvalue is positive; every leading principal minor is positive (Sylvester's criterion — which does not relax to $\ge0$ for the semidefinite case); elimination needs no exchanges and gives $n$ positive pivots; $A=B^TB$ with $B$ of independent columns — equivalently $A=LL^T$ with $L$ lower triangular of positive diagonal, the unique Cholesky factor.

Exercises for §11.9 →

Common Mistakes

Assuming every matrix is orthogonally diagonalisable. Only symmetric matrices are, and the converse holds too: if $A = Q\Lambda Q^T$ with $Q$ orthogonal, then $A$ must be symmetric.
Applying Gram–Schmidt across different eigenspaces. Eigenvectors for distinct eigenvalues are already orthogonal; orthogonalising them against each other destroys the eigenvector property. Gram–Schmidt is needed only *within* a repeated eigenspace.
Writing $A^m = Q\Lambda^mQ^T$ for a non-symmetric matrix. The general formula is $PD^mP^{-1}$ ; $P^T$ equals $P^{-1}$ only when $P$ is orthogonal.
Putting the whole cross-term coefficient into the matrix of a quadratic form. For $ax^{2}+2bxy+cy^{2}$ the off-diagonal entry is $b$ , half of the coefficient of $xy$ .
Confusing congruence $P^TAP$ with similarity $P^{-1}AP$ . Congruence preserves the signature but changes the eigenvalues; only an orthogonal $P$ makes the two coincide.
Reporting $|\lambda|_{\max}$ as the maximum of $\mathbf{x}^TA\mathbf{x}$ on the unit sphere. The maximum is $\lambda_{\max}$ and the minimum is $\lambda_{\min}$ ; the largest absolute value answers a different question.
Forgetting that a quadratic form has no unconstrained maximum unless it is negative semidefinite. Optimisation of a form is only meaningful subject to a constraint such as $\|\mathbf{x}\| = 1$ .
Testing positive definiteness by looking at the entries. Positive entries prove nothing, and a positive definite matrix may have negative off-diagonal entries.
Using $\det A > 0$ alone as a test. In even dimensions a negative definite matrix also has positive determinant; all $n$ leading minors are needed.
Relaxing Sylvester's criterion to $\geq 0$ for the semidefinite case. That is false — $\operatorname{diag}(0,-1)$ is the standard counterexample. The semidefinite test needs *all* principal minors, not just the leading ones.
Applying the principal axes theorem to a conic with linear terms. Translate the centre to the origin first, then rotate.
Expecting the diagonal of the Cholesky factor to hold eigenvalues. It holds the square roots of the pivots; only the products agree.
Declaring a critical point a minimum because one second derivative is positive. The Hessian must be positive definite — every direction must curve upward, not just the coordinate directions.

About the practice questions. They check that you can carry out this chapter's computations correctly, and each one is graded on a single answer. They are not proof exercises: working through them confirms the mechanics, not that you could prove the results yourself. For that, re-read the statements above and try to reconstruct their proofs with the page closed.

← Inner Product Spaces The Singular Value Decomposition →

Helpful?