Linear Algebra for Data Science

Units

1. 1

   Vectors, dot products and norms

   5 lessons

   1. Vectors as arrows and dataWrite a data row as an ordered list of numbers, draw a two-feature vector as an arrow from the origin, and see why a third feature needs a third axis.
   2. Vector addition and scalingAdd and scale two-dimensional vectors componentwise, then interpret each result as a displacement.
   3. Linear combinations and spanForm linear combinations c₁v₁ + c₂v₂ with given weights, find the weights that reach a target, and describe the span of one or two vectors as a line or a plane through the origin.
   4. The dot productCompute dot products, angles and cosine similarity for nonzero vectors; recognize orthogonal vectors.
   5. Norms and distanceMeasure a vector's length with the L2, L1 and L∞ norms, draw their unit balls, find distances between data points, form unit vectors, and use the triangle and Cauchy–Schwarz inequalities.
2. 2

   Matrices as transformations

   3 lessons

   1. Matrix-vector multiplicationCompute Ax two ways, as a combination of the columns of A and as dot products with its rows, and use a data matrix times a weight vector.
   2. Linear transformationsTest whether a map is linear, build its matrix from the images of the basis vectors, and picture it as a transformation of the unit square and of a grid.
   3. Rotations, reflections and projectionsWrite the standard 2 × 2 matrices for rotation, reflection, scaling, shear and projection by reading the images of e₁ and e₂, and describe each by its effect on the unit square.
3. 3

   Matrix multiplication

   3 lessons

   1. Multiplication as compositionRead AB as "apply B, then A", compute the product with the row-column rule after checking the sizes fit, and show with a shear and a rotation that AB and BA differ.
   2. Four views of the productCompute one product AB four ways (entries as dot products, columns as combinations of the columns of A, rows as combinations of the rows of B, and a sum of column-times-row pieces) and know which view suits which question.
   3. Transpose and special matricesForm the transpose, use (AB)ᵀ = BᵀAᵀ, recognise symmetric, diagonal, identity and triangular matrices, and read the Gram matrix XᵀX of a data matrix as the table of dot products of its columns.
4. 4

   Linear systems and elimination

   4 lessons

   1. Row and column picturesDraw the system Ax = b two ways, as lines (or planes) that meet in the solution and as columns of A combined to reach b, and tell from either picture whether there are zero, one or infinitely many solutions.
   2. Gaussian eliminationReduce a system Ax = b to upper triangular form with row operations on the augmented matrix, read off the pivots, and finish by back substitution, including what happens when a pivot is zero.
   3. Reduced row echelon formReduce an augmented matrix to reduced row echelon form, name the pivot and free variables, and write the complete solution as a particular solution plus combinations of null space vectors.
   4. LU factorizationRecord the multipliers of elimination in a lower triangular matrix L so that A = LU with U the upper triangular result, and solve Ax = b by forward then back substitution, cheaply for many right sides.
5. 5

   Rank and inverses

   4 lessons

   1. RankCompute the rank of a matrix as its number of pivots, see it as the number of independent columns (and of independent rows), and spot a redundant feature in a data matrix.
   2. The inverse matrixDefine A⁻¹ by A A⁻¹ = I, find it for a 2 × 2 matrix by the formula and by Gauss-Jordan elimination on [A | I], and use (AB)⁻¹ = B⁻¹A⁻¹.
   3. When a matrix is invertibleDecide whether a square matrix is invertible from its determinant, its rank, its null space or its columns, and see why these tests all say the same thing.
   4. Numerical conditioningExplain why a nearly singular matrix is dangerous on a computer, measure the danger with the condition number, and solve Ax = b without forming A⁻¹.
6. 6

   Vector spaces and subspaces

   3 lessons

   1. Vector spacesState the vector space rules, recognise polynomials and functions as vectors, and test whether a subset of a space is a subspace.
   2. Column space and null spaceDescribe the column space C(A) and the null space N(A) of a matrix, and use them to decide whether Ax = b is solvable and whether its solution is unique.
   3. The four fundamental subspacesName the four subspaces of an m × n matrix of rank r, give their dimensions r, n − r, r and m − r, and say which pairs are perpendicular.
7. 7

   Bases and dimension

   4 lessons

   1. Linear independenceDecide whether a list of vectors is independent from the definition and by elimination, and recognise a dependent column in a data matrix.
   2. Basis and dimensionDefine a basis as an independent spanning set, define dimension as the number of vectors in a basis, and read bases for the column space and null space off the reduced row echelon form.
   3. Coordinates and change of basisWrite a vector in coordinates relative to a basis, convert between standard and skewed coordinates with the change of basis matrix, and see why a well-chosen basis makes coordinates easy.
   4. The rank-nullity theoremState and use rank(A) + dim N(A) = n, read as pivots plus free columns, and apply it to solution sets and to systems with more unknowns than equations.
8. 8

   Orthogonality and projections

   4 lessons

   1. Orthogonal vectors and subspacesTest orthogonality with the dot product, find the orthogonal complement of a subspace, see that the row space is perpendicular to the null space, and use orthonormal bases to read coordinates as dot products.
   2. Projection onto a lineProject a vector b onto the line through a with p = (a·b / a·a) a, check that the error b − p is orthogonal to a, build the projection matrix P = aaᵀ/(aᵀa), and see p as the closest point of the line to b.
   3. Projection onto a subspaceProject b onto the column space of A by solving AᵀA x̂ = Aᵀb, build P = A(AᵀA)⁻¹Aᵀ, check P² = P and Pᵀ = P, and split b = p + e with e orthogonal to the subspace.
   4. Gram-Schmidt and QRTurn independent columns into orthonormal ones with Gram-Schmidt, record the steps as A = QR with R upper triangular, and use orthogonal matrices (QᵀQ = I) that preserve lengths.
9. 9

   Least squares

   3 lessons

   1. The normal equationsWhen Ax = b has no solution, find the x̂ that minimizes |Ax − b|² by solving AᵀA x̂ = Aᵀb, and read the error e = b − Ax̂ as orthogonal to every column of A.
   2. Fitting lines and curvesBuild the design matrix A for a line and for a parabola, solve AᵀA x̂ = Aᵀb, and read the fit and its residuals; see why a model with as many columns as points fits exactly yet predicts badly.
   3. Least squares in practiceSolve least squares with QR (R x̂ = Qᵀb), explain why that is more accurate than the normal equations, and choose the minimum-norm solution with the pseudoinverse when the columns are dependent.
10. 10

    Eigenvalues and diagonalization

    5 lessons

    1. Eigenvalues and eigenvectorsDefine an eigenvector and eigenvalue by Av = λv, find them for simple 2 × 2 matrices by inspection and checking, and see which matrices have real eigen-directions at all.
    2. The characteristic polynomialFind all eigenvalues of a matrix from det(A − λI) = 0, find eigenvectors as the null space of A − λI, and use trace and determinant as the sum and product of the eigenvalues.
    3. Diagonalizing a matrixWrite a matrix with n independent eigenvectors as A = SΛS⁻¹, use it to compute powers Aᵏ = SΛᵏS⁻¹, and find the steady state of a Markov chain from the eigenvector for λ = 1.
    4. Symmetric matrices and the spectral theoremShow that a real symmetric matrix has real eigenvalues and perpendicular eigenvectors, write it as A = QΛQᵀ with Q orthogonal, and expand it as a sum λ₁q₁q₁ᵀ + λ₂q₂q₂ᵀ of projections.
    5. Positive definite matricesDecide whether a symmetric matrix is positive definite by its eigenvalues, its pivots or xᵀAx > 0, draw its quadratic form as nested ellipses, and see why Gram and covariance matrices are positive semidefinite.
11. 11

    Singular value decomposition

    3 lessons

    1. The singular value decompositionState A = UΣVᵀ for any matrix, find singular values as square roots of the eigenvalues of AᵀA, and read the factorization as rotate, stretch, rotate: the unit circle becomes an ellipse with half-axes σ₁ and σ₂.
    2. Computing the SVDCompute the SVD of a 2 × 2 matrix by finding V and the σs from the eigenvectors and eigenvalues of AᵀA and each uᵢ = Avᵢ/σᵢ, and read bases of the four fundamental subspaces off U and V.
    3. Low-rank approximationWrite a matrix as a sum of rank-one layers σᵢuᵢvᵢᵀ, use the Eckart-Young theorem to say the truncated SVD is the best rank-k approximation with error σₖ₊₁, and judge an image compression by energy kept and numbers stored.
12. 12

    Principal component analysis

    4 lessons

    1. The covariance matrixCenter a data matrix, form the sample covariance matrix C = XᵀX/(n − 1), and read its diagonal entries as variances and its off-diagonal entries as covariances of features.
    2. Principal componentsFind the direction of maximal variance of a data cloud as the top eigenvector of its covariance matrix, draw the principal components as rotated perpendicular axes, and compute each point's scores on them.
    3. PCA via SVD and explained varianceCompute PCA from the SVD of the centered data matrix (components V, variances σᵢ²/(n − 1), scores UΣ), report the explained variance ratio, read a scree plot and choose the number of components.
    4. Applying PCAStandardize features before PCA when their units differ, project the data onto the first components to plot it, and read the loadings to name what each component measures, on a small worked data set.