Mathematical Foundations for Machine Learning

The journey into machine learning begins not with algorithms or data, but with mathematics. Before we can understand how a neural network learns to recognize faces or how a recommendation system predicts your next favorite movie, we must first establish the mathematical language through which these systems communicate their logic. This chapter serves as that foundation—a carefully constructed scaffold of concepts drawn from linear algebra, probability theory, and information theory that will appear, time and again, as we progress from the simplest linear models to the most sophisticated ensemble methods.

The importance of this mathematical grounding cannot be overstated. When you read that a support vector machine finds an "optimal hyperplane" that "maximizes the margin" between classes, you are witnessing linear algebra (hyperplanes as high-dimensional surfaces), geometry (distances and margins), and optimization theory (finding maxima under constraints) working in concert. When you encounter claims that a Bayesian classifier "minimizes expected risk" or that entropy "measures uncertainty," you are dealing with probability and information theory at their core. Without these foundations, machine learning becomes a collection of black-box procedures to be memorized rather than understood. With them, algorithms reveal their inner elegance and limitations become transparent.

Let us begin with linear algebra, the workhorse of numerical computation in machine learning. Throughout this book, you will encounter vectors representing data points, matrices encoding transformations and relationships, and eigenstructures revealing the fundamental patterns within data. We will explore these concepts both analytically and geometrically, for visual intuition often illuminates algebraic manipulation.

Vectors and Vector Spaces

A vector, at its most fundamental level, is an ordered collection of numbers. In machine learning, we typically work with vectors in $\mathbb{R}^n$, the space of all n-dimensional real-valued vectors. A data point with $d$ features—say, a patient's blood pressure, age, and cholesterol level—is naturally represented as a vector $\mathbf{x} = (x_1, x_2, \ldots, x_d) \in \mathbb{R}^d$.

The geometric interpretation of vectors is essential for building intuition. A vector in $\mathbb{R}^2$ can be visualized as an arrow in the plane, originating from the origin and terminating at the point specified by its coordinates. This arrow has both a direction and a magnitude (or length). The magnitude of a vector $\mathbf{x}$ is given by the Euclidean norm:

$$\|\mathbf{x}\| = \sqrt{x_1^2 + x_2^2 + \cdots + x_d^2}$$

The norm satisfies three critical properties that we will rely upon throughout our study: non-negativity ($\|\mathbf{x}\| \geq 0$, with equality only for the zero vector), absolute homogeneity ($\|a\mathbf{x}\| = |a|\|\mathbf{x}\|$ for any scalar $a$), and the triangle inequality ($\|\mathbf{x} + \mathbf{y}\| \leq \|\mathbf{x}\| + \|\mathbf{y}\|$).

The dot product (or inner product) between two vectors $\mathbf{x}$ and $\mathbf{y}$ is defined as:

$$\mathbf{x} \cdot \mathbf{y} = \sum_{i=1}^{d} x_i y_i = \|\mathbf{x}\|\|\mathbf{y}\|\cos\theta$$

where $\theta$ is the angle between the vectors. The dot product encapsulates several important relationships: when $\mathbf{x} \cdot \mathbf{y} = 0$, the vectors are orthogonal (perpendicular); when $\mathbf{x} \cdot \mathbf{y} > 0$, they point in generally similar directions; and when $\mathbf{x} \cdot \mathbf{y} < 0$, they point in generally opposite directions. This geometric interpretation becomes crucial in understanding concepts like projection, which underlies algorithms from linear regression to principal component analysis.

Consider a concrete example in sentiment analysis. Suppose we represent words as vectors in a high-dimensional space using techniques like word2vec, where semantic similarity corresponds to vector similarity. The word "happy" might be represented as $\mathbf{v}_{\text{happy}}$, and "joyful" as $\mathbf{v}_{\text{joyful}}$. Because these words have similar meanings, their vectors would point in nearly the same direction, yielding a large dot product after normalization. Conversely, "happy" and "tragic" would have vectors pointing in nearly opposite directions, resulting in a negative dot product.

A vector space (or linear space) is a collection of vectors that is closed under addition and scalar multiplication. For machine learning, the space $\mathbb{R}^d$ serves as our primary arena, where each dimension corresponds to a feature or attribute. When we perform linear regression, we are finding a linear combination of feature vectors that best approximates our target variable. When we reduce dimensionality with PCA, we are identifying the subspace of $\mathbb{R}^d$ that captures the most variance in our data.

Key subspaces that appear throughout machine learning include the column space and null space of a matrix. The column space (or range) of a matrix $A$ consists of all linear combinations of its column vectors—it represents the space of all outputs achievable by the linear transformation defined by $A$. The null space (or kernel) consists of all vectors that $A$ maps to zero. Understanding these subspaces helps explain why certain linear systems have unique solutions, infinitely many solutions, or no solution at all.

Matrices: Transformations and Operations

If vectors are the nouns of linear algebra, matrices are its verbs—representing transformations, encoding relationships, and enabling the efficient computation that machine learning demands. A matrix $A \in \mathbb{R}^{m \times n}$ is a rectangular array with $m$ rows and $n$ columns, and it can represent a linear transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$.

When we multiply a matrix $A$ by a vector $\mathbf{x}$, we are applying the transformation encoded in $A$:

$$A\mathbf{x} = \mathbf{y}$$

This operation has a beautiful geometric interpretation. Every matrix transformation preserves two fundamental properties: lines remain lines (though they may change direction), and the origin remains fixed. Beyond these basics, matrices can perform rotations (preserving distances), scalings (changing distances), shearing (tilting shapes), and projections (collapsing dimensions).

The identity matrix $I_n$ plays the role of "do nothing" transformation:

$$I_n \mathbf{x} = \mathbf{x} \quad \text{for all } \mathbf{x} \in \mathbb{R}^n$$

The inverse of a matrix $A$, denoted $A^{-1}$, satisfies $A^{-1}A = I = AA^{-1}$. When $A$ has an inverse, the linear system $A\mathbf{x} = \mathbf{b}$ has the unique solution $\mathbf{x} = A^{-1}\mathbf{b}$. However, not all matrices are invertible. A matrix is singular (non-invertible) if its rows or columns are linearly dependent—if one row can be expressed as a combination of others. Geometrically, singular matrices "collapse" dimensions, mapping multiple different inputs to the same output.

The determinant of a matrix, $\det(A)$, provides crucial information about the transformation it represents. When $|\det(A)| = 1$, the transformation preserves volume (though it may rotate or reflect). When $\det(A) = 0$, the transformation collapses volume to zero—it is singular and non-invertible. When $|\det(A)| > 1$, the transformation expands volume; when $0 < |\det(A)| < 1$, it contracts volume. In machine learning, the determinant appears in contexts ranging from computing probabilities in multivariate Gaussian distributions to understanding the stability of numerical algorithms.

Matrix Decompositions: Breaking Down Complexity

Matrix decompositions express a matrix as a product of matrices with special structures, revealing hidden patterns and enabling efficient computation. We will encounter several decompositions repeatedly in our study of machine learning algorithms.

The LU decomposition factors a matrix $A$ as $A = L