Linear Algebra: Matrices, Eigenvalues, and Quadratic Forms
Y. Eddie Lu, Summer 2026
ECON 8001 course index · Lecture 3 of 8
Orientation
What problem do matrices solve?
A linear map is an abstract rule. Once bases have been chosen, a matrix makes the rule computable: it records the image of every basis vector, composes maps, detects information loss, and identifies directions of pure scaling. Those last two jobs are the bridge to identification, covariance matrices, and curvature in optimization.
Where this sits
This page follows Linear algebra: objects. Its matrix calculations and quadratic forms are used in Differentiation and Optimization.
Let \(\mathbb F\) be \(\mathbb R\) or \(\mathbb C\). Vectors are columns. An \(m\times n\) matrix \(A=(a_{ij})\) has \(m\) rows, \(n\) columns, and entries in \(\mathbb F\); write \(A\in M_{m\times n}(\mathbb F)\). If \(x\in\mathbb F^n\), then \(Ax\in\mathbb F^m\). The \(j\)th column of \(A\) is \(A e_j\), where \(e_j\in\mathbb F^n\) is the \(j\)th standard basis vector.
Retrieval check. If \(A\) is \(3\times5\), what are the dimensions of its input and output vectors?
It maps \(\mathbb F^5\) to \(\mathbb F^3\): five input coordinates and three output coordinates.
1. Matrices are coordinate descriptions of maps
An \(m\times n\) matrix is a rectangular array \[ A=\begin{pmatrix} a_{11}&\cdots&a_{1n}\\ \vdots&\ddots&\vdots\\ a_{m1}&\cdots&a_{mn} \end{pmatrix},\qquad a_{ij}\in\mathbb F. \] For the linear map represented by \(A\), the column rule gives \[Ax=x_1Ae_1+\cdots+x_nAe_n\] for every \(x=(x_1,\ldots,x_n)^{\mathsf{T}}\in\mathbb F^n\).
For \(A=\begin{pmatrix}1&1\\1&1\\0&0\end{pmatrix}\), both \(e_1\) and \(e_2\) map to \((1,1,0)^{\mathsf{T}}\). Hence \(A(x_1,x_2)^{\mathsf{T}}=(x_1+x_2)(1,1,0)^{\mathsf{T}}\): two input directions are compressed to one output direction.
Operations and conformability
If \(A,B\in M_{m\times n}(\mathbb F)\), define \((A+B)_{ij}=a_{ij}+b_{ij}\). For \(c\in\mathbb F\), define \((cA)_{ij}=ca_{ij}\). If \(A\in M_{m\times n}(\mathbb F)\) and \(B\in M_{n\times k}(\mathbb F)\), their product is the \(m\times k\) matrix \[ (AB)_{ij}=\sum_{\ell=1}^{n}a_{i\ell}b_{\ell j}. \] The repeated inner dimension \(n\) is required. Addition requires identical dimensions; multiplication does not.
If \(A\) is \(m\times n\) and \(B\) is \(n\times k\), then \(AB\) is \(m\times k\). Read \(ABx\) from right to left: \(x\in\mathbb F^k\), then \(Bx\in\mathbb F^n\), then \(A(Bx)\in\mathbb F^m\). In general \(BA\) is not defined, and when it is defined it need not equal \(AB\).
Let \(A=\begin{pmatrix}1&2\\0&1\end{pmatrix}\) and \(B=\begin{pmatrix}3&0\\1&4\end{pmatrix}\). Then \[ \begin{aligned} AB&=\begin{pmatrix}5&8\\1&4\end{pmatrix},\\ BA&=\begin{pmatrix}3&6\\1&6\end{pmatrix}. \end{aligned} \] Both products exist, but they differ. The first applies \(B\) before \(A\).
Retrieval check. Why is \((AB)C=A(BC)\) dimensionally natural?
Both expressions apply \(C\), then \(B\), then \(A\) to an input vector. The entries also agree after expanding the two finite double sums.
Whenever the displayed operations are conformable, matrix addition is commutative and associative; multiplication is associative; and multiplication distributes over addition on either side: \[ \begin{aligned} (AB)C&=A(BC),\\ A(B+C)&=AB+AC,\\ (A+B)C&=AC+BC. \end{aligned} \] Matrix multiplication is not generally commutative.
Proof strategy. Check an arbitrary \((i,j)\) entry. Associativity is the only part that needs care: both sides become the same finite sum \(\sum_{r}\sum_s a_{ir}b_{rs}c_{sj}\), merely grouped differently.
Transpose
For \(A\in M_{m\times n}(\mathbb F)\), the transpose \(A^{\mathsf{T}}\in M_{n\times m}(\mathbb F)\) is defined by \((A^{\mathsf{T}})_{ij}=a_{ji}\). It exchanges rows and columns.
For conformable matrices, \[ \begin{aligned} (A+B)^{\mathsf{T}}&=A^{\mathsf{T}}+B^{\mathsf{T}},\\ (AB)^{\mathsf{T}}&=B^{\mathsf{T}}A^{\mathsf{T}}. \end{aligned} \]
Proof strategy. The \((i,j)\) entry of \((AB)^{\mathsf{T}}\) is \((AB)_{ji}\), the dot product of row \(j\) of \(A\) with column \(i\) of \(B\). Rewriting the same scalar product gives the \((i,j)\) entry of \(B^{\mathsf{T}}A^{\mathsf{T}}\). The reversal of order is not a mnemonic: it follows from exchanging rows and columns.
2. Rank measures surviving directions
For \(A\in M_{m\times n}(\mathbb F)\), the column rank is the dimension of the span of its \(n\) columns in \(\mathbb F^m\). The row rank is the dimension of the span of its \(m\) rows in \(\mathbb F^n\).
Let \(L:\mathbb F^n\to\mathbb F^m\) be represented by \(A\).
- \(\dim(\operatorname{im}L)\) equals the column rank of \(A\).
- The row rank and column rank of every matrix \(A\) are equal. Their common value is \(\operatorname{rank}(A)\).
Proof strategy. The first statement is immediate from the column rule: \(\operatorname{im}L\) is exactly the span of the columns. For row rank, reduce \(A\) by elementary row operations. These preserve the dimension of the row space and preserve the linear dependence relations among columns. In reduced row-echelon form, the nonzero rows and pivot columns are both counted by the
number of pivots.
The matrix \(A=\begin{pmatrix}1&1\\1&1\\0&0\end{pmatrix}\) has one independent column, so \(\operatorname{rank}(A)=1\). Its kernel is \[ \ker A=\{(t,-t)^{\mathsf{T}}:t\in\mathbb F\}, \] which has dimension \(1\). One input direction survives; one is lost.
If \(A\in M_{m\times n}(\mathbb F)\) represents \(L:\mathbb F^n\to\mathbb F^m\), then \[ n=\operatorname{rank}(A)+\dim(\ker A). \] The \(n\) on the left is the dimension of the domain, not the number of rows.
For a rectangular \(m\times n\) matrix, full column rank means \(\operatorname{rank}(A)=n\) and injectivity; full row rank means \(\operatorname{rank}(A)=m\) and surjectivity. Only when \(m=n\) can either condition give an ordinary two-sided inverse.
3. Square matrices: determinant and inverse
An \(n\times n\) matrix is square. It is symmetric when \(A=A^{\mathsf{T}}\); upper (lower) triangular when \(a_{ij}=0\) for \(i>j\) (\(i<j\)); diagonal when \(a_{ij}=0\) for \(i\ne j\); and the identity \(I_n\) when diagonal entries equal \(1\) and all off-diagonal entries equal \(0\). A triangular matrix is either upper or lower triangular.
For \(A=(a_{ij})\in M_{n\times n}(\mathbb F)\), \[\operatorname{tr}(A)=\sum_{i=1}^n a_{ii}.\] The trace is additive, satisfies \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\) for \(A\in M_{m\times n}\) and \(B\in M_{n\times m}\), and satisfies \(\operatorname{tr}(A)=\operatorname{tr}(A^{\mathsf{T}})\).
Proof strategy for \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\). Expand both sides. Each is \(\sum_{i=1}^m\sum_{j=1}^n a_{ij}b_{ji}\); only the order of the two finite sums changes. Notice that \(AB\) and \(BA\) can have different dimensions, but both traces are defined and equal.
For a \(1\times1\) matrix, \(\det(a_{11})=a_{11}\). For \(n\ge2\), let \(A_{ij}\) be the \((n-1)\times(n-1)\) matrix obtained by deleting row \(i\) and column \(j\). Define the determinant recursively by expansion along the first row: \[ \det A=\sum_{j=1}^n(-1)^{1+j}a_{1j}\det(A_{1j}). \] The minor is \(M_{ij}=\det(A_{ij})\) and the cofactor is \(C_{ij}=(-1)^{i+j}M_{ij}\). For \(A=\begin{pmatrix}a&b\\c&d\end{pmatrix}\), \(\det A=ad-bc\).
For every fixed row \(i\) of a square matrix, \[\det A=\sum_{j=1}^n(-1)^{i+j}a_{ij}\det(A_{ij}),\] and the analogous expansion holds along every fixed column. Thus the recursive first-row definition agrees with every cofactor expansion.
For a square matrix, swapping two rows or columns changes the determinant’s sign; multiplying one row or column by \(k\) multiplies it by \(k\); and adding a multiple of one row or column to another leaves it unchanged. Also, \[ \begin{aligned} \det(AB)&=\det(A)\det(B),\\ \det(A^{\mathsf{T}})&=\det(A). \end{aligned} \]
For a real square \(A\), \(|\det A|\) is its factor of volume scaling. Thus \(\det A=0\) means some positive-dimensional collection has been flattened, which is exactly the obstruction to inversion.
A square matrix \(A\in M_{n\times n}(\mathbb F)\) is invertible, or nonsingular, if there is \(A^{-1}\in M_{n\times n}(\mathbb F)\) such that \[AA^{-1}=A^{-1}A=I_n.\]
For \(A\in M_{n\times n}(\mathbb F)\), the following statements are equivalent:
- \(A\) is invertible.
- \(\operatorname{rank}(A)=n\).
- \(\det(A)\ne0\).
If \(A\) is triangular, it is invertible exactly when every diagonal entry is nonzero. If \(A^{-1}\) exists, it is unique.
Proof strategy. A zero determinant is preserved under the row operations that reveal a zero pivot, so it identifies a failure of full rank. Full rank for a square map means no kernel and every output is reached, hence a bijection with an inverse. For uniqueness, if \(B\) and \(C\) are both inverses, then \(B=BI=B(AC)=(BA)C=C\).
The adjugate is the transpose of the cofactor matrix, \(\operatorname{adj}(A)=(C_{ji})_{ij}\). If \(\det A\ne0\), then \[A^{-1}=\frac{1}{\det A}\operatorname{adj}(A).\] For invertible conformable square matrices, \[ \begin{aligned} (A^{-1})^{-1}&=A,\\ (A^{\mathsf{T}})^{-1}&=(A^{-1})^{\mathsf{T}},\\ (AB)^{-1}&=B^{-1}A^{-1},\\ \det(A^{-1})&=(\det A)^{-1}. \end{aligned} \] The inverse of an upper (lower) triangular invertible matrix is upper (lower) triangular.
\(\det(A)=0\) does not mean that every vector maps to zero. It means that at least one nonzero vector maps to zero, so distinct inputs can have the same output. The map in the earlier rank example has a nonzero image but determinant is not even defined because the matrix is rectangular.
4. Eigenvalues and changes of basis
Let \(A\in M_{n\times n}(\mathbb F)\). A scalar \(\lambda\in\mathbb F\) is an eigenvalue if there is a nonzero \(v\in\mathbb F^n\) such that \[Av=\lambda v.\] Such a \(v\) is an eigenvector corresponding to \(\lambda\). Equivalently, \(\lambda\) is an eigenvalue exactly when \[\det(A-\lambda I_n)=0.\]
For \(A=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), \[ \begin{aligned} A(1,1)^{\mathsf{T}}&=3(1,1)^{\mathsf{T}},\\ A(1,-1)^{\mathsf{T}}&=(1,-1)^{\mathsf{T}}. \end{aligned} \] The equal-coordinate direction is scaled by \(3\); the difference direction is unchanged. A negative eigenvalue would preserve the line but reverse its orientation.
Square matrices \(A\) and \(B\) are similar if \(B=P^{-1}AP\) for some invertible \(P\). A square matrix \(A\) is diagonalizable if it is similar to a diagonal matrix \(\Lambda\): \(P^{-1}AP=\Lambda\). Similar matrices represent the same linear map in different coordinate systems.
For \(A\in M_{n\times n}(\mathbb F)\), the following are equivalent:
- \(A\) is diagonalizable.
- \(A\) has \(n\) linearly independent eigenvectors.
If \(v_1,\ldots,v_n\) are independent eigenvectors and \(P=(v_1\ \cdots\ v_n)\), then \(P^{-1}AP\) is diagonal. Its diagonal entries are the corresponding eigenvalues; conversely, any diagonal matrix similar to \(A\) has eigenvalues of \(A\) on its diagonal.
Proof strategy. Put the eigenvector equations side by side: \(AP=P\Lambda\). Since the columns of \(P\) are independent, \(P\) is invertible; multiplying by \(P^{-1}\) gives \(P^{-1}AP=\Lambda\). Conversely, read that identity as \(AP=P\Lambda\): each column of \(P\) is an eigenvector.
Having repeated eigenvalues is neither sufficient nor fatal. \(I_n\) has one repeated eigenvalue and every nonzero vector is an eigenvector, so it is diagonalizable. In contrast, \(\begin{pmatrix}1&1\\0&1\end{pmatrix}\) has only one independent eigenvector, so it is not diagonalizable.
Orthogonality and the spectral theorem
Vectors \(v_1,\ldots,v_m\in\mathbb R^n\) are orthogonal when \(v_i^{\mathsf{T}}v_j=0\) for \(i\ne j\). A nonzero vector \(v\) is normalized when \(\|v\|=1\). An orthogonal list of normalized vectors is orthonormal. A real square matrix \(P\) is orthogonal, or orthonormal, when its columns are an orthonormal basis; equivalently \(P^{\mathsf{T}}P=I_n\).
If \(P\) is orthogonal, then \(P^{\mathsf{T}}=P^{-1}\). If \(A\in M_{n\times n}(\mathbb R)\) is symmetric, then there exist an orthogonal \(P\) and a diagonal \(\Lambda\) such that \[A=P\Lambda P^{\mathsf{T}},\] where the columns of \(P\) are eigenvectors of \(A\) and the diagonal entries of \(\Lambda\) are the associated real eigenvalues.
Proof. For the first statement, the \((i,j)\) entry of \(P^{\mathsf{T}}P\) is the inner product of columns \(i\) and \(j\). For the spectral theorem, first establish a real eigenpair. The Rayleigh quotient \(q(x)=x^{\mathsf{T}}Ax\) is continuous on the compact unit sphere \(S^{n-1}\), so it attains a maximum at some \(v_1\in S^{n-1}\). For every \(w\perp v_1\), the curve
\[ \gamma(t)=\frac{v_1+tw}{\|v_1+tw\|} \]
lies on \(S^{n-1}\) and has \(\gamma'(0)=w\). Since \(q\circ\gamma\) has a maximum at \(0\),
\[ 0=(q\circ\gamma)'(0)=2w^{\mathsf{T}}Av_1, \]
where symmetry of \(A\) gives the final equality. Thus \(Av_1\) is orthogonal to every \(w\in v_1^\perp\), so \(Av_1\in(v_1^\perp)^\perp =\operatorname{span}\{v_1\}\). Hence \(Av_1=\lambda_1v_1\) for the real scalar \(\lambda_1=v_1^{\mathsf{T}}Av_1\).
Now \(v_1^\perp\) is invariant under \(A\): if \(w\in v_1^\perp\), then \(v_1^{\mathsf{T}}Aw=(Av_1)^{\mathsf{T}}w=\lambda_1v_1^{\mathsf{T}}w=0\). The restriction of \(A\) to \(v_1^\perp\) is again real and symmetric. Induction on dimension, starting from \(n=1\), gives an orthonormal eigenbasis of \(v_1^\perp\); adjoining \(v_1\) gives an orthonormal eigenbasis of \(\mathbb R^n\) and hence \(A=P\Lambda P^{\mathsf{T}}\).
5. Quadratic forms and definiteness
Let \(A=(a_{ij})\in M_{n\times n}(\mathbb R)\) be symmetric. For \(x\in\mathbb R^n\), its quadratic form is the scalar-valued function \[ Q_A(x)=x^{\mathsf{T}}Ax =\sum_{i=1}^n\sum_{j=1}^n a_{ij}x_ix_j. \] The dimensions are \((1\times n)(n\times n)(n\times1)=1\times1\). Thus \(x^{\mathsf{T}}Ax\) is a scalar, not a vector or matrix-valued curvature object.
For real symmetric \(A\), quantify over every \(x\in\mathbb R^n\setminus\{0\}\):
| Type | Requirement on \(x^{\mathsf{T}}Ax\) |
|---|---|
| positive definite | \(>0\) |
| positive semidefinite | \(\ge0\) |
| negative definite | \(<0\) |
| negative semidefinite | \(\le0\) |
| indefinite | positive for some nonzero \(x\) and negative for another |
Let \(A\in M_{n\times n}(\mathbb R)\) be symmetric. It is positive definite, positive semidefinite, negative definite, negative semidefinite, or indefinite exactly when, respectively, all its eigenvalues are positive, nonnegative, negative, nonpositive, or include both signs.
Proof strategy. Use the spectral theorem. If \(A=P\Lambda P^{\mathsf{T}}\) and \(z=P^{\mathsf{T}}x\), then \(z\ne0\) whenever \(x\ne0\), and \[x^{\mathsf{T}}Ax=z^{\mathsf{T}}\Lambda z=\sum_{i=1}^n\lambda_i z_i^2.\]
The signs of the eigenvalues therefore control the sign in every direction.
If \(A=D^2f(x)\) for a \(C^2\) scalar function, then \(A\) is symmetric and \(v^{\mathsf{T}}Av\) is its second-order curvature in direction \(v\in\mathbb R^n\). For a nonsymmetric matrix, \(v^{\mathsf{T}}Av\) depends only on its symmetric part. Let \(X\in\mathbb R^n\) be a random vector with covariance matrix \(\Sigma\). For \(w\in\mathbb R^n\), \(w^{\mathsf{T}}\Sigma w=\operatorname{Var}(w^{\mathsf{T}}X)\) is the variance of a linear portfolio. Both expressions are scalars, and both require conformable column vectors.
The symmetric matrix \(A=\begin{pmatrix}1&2\\2&1\end{pmatrix}\) has positive diagonal entries, but its eigenvalues are \(3\) and \(-1\). Indeed, \((1,1)A(1,1)^{\mathsf{T}}=6\) while \((1,-1)A(1,-1)^{\mathsf{T}}=-2\). Hence \(A\) is indefinite. Checking only coordinate directions misses the saddle direction.
Assumption audit and proof blueprint
- Matrix multiplication needs matching inner dimensions. Rank–nullity uses the domain dimension \(n\).
- Determinants, inverses, eigenvalues, and diagonalization here concern square matrices. \(A^{-1}\) requires \(\det A\ne0\).
- Orthogonal diagonalization and the eigenvalue test for definiteness require a real symmetric matrix. A general real matrix need not have real eigenvalues or an orthonormal eigenbasis.
- Definiteness is a statement about every nonzero direction. A merely semidefinite, but not definite, matrix can have flat directions and does not give a strict curvature conclusion.
Blueprint: classify a symmetric quadratic form.
- Check that \(A\) is real, square, and symmetric.
- Solve \(\det(A-\lambda I)=0\) for its eigenvalues.
- Record their signs, including zeros.
- Apply Theorem 1.4.26. Do not infer a strict sign from a nonnegative one.
Exit tickets
- A \(4\times6\) matrix has rank \(4\). State its nullity and whether it is injective or surjective as a map \(\mathbb R^6\to\mathbb R^4\).
- Explain in one line why \((AB)^{-1}\) has the reverse order.
- State exactly which hypothesis lets eigenvalue signs classify a quadratic form.
- Nullity is \(6-4=2\); it is surjective but not injective. 2. Multiplication gives \((AB)(B^{-1}A^{-1})=A(BB^{-1})A^{-1}=I\), and similarly on the other side. 3. \(A\) must be a real symmetric matrix.