Mathematical Foundations for Computer Vision

The ability to interpret and understand visual information lies at the heart of computer vision, yet the elegant algorithms and powerful neural networks that enable machines to "see" are fundamentally built upon a surprisingly small set of mathematical ideas. Before we can appreciate how a convolutional neural network extracts features from an image or how an autoencoder learns meaningful representations, we must first establish fluency in the language of mathematics that these systems speak. This chapter serves as that essential foundation, guiding you through the linear algebra, calculus, probability theory, optimization, and information theory that together form the mathematical bedrock of modern computer vision.

Understanding these prerequisites is not merely an academic exercise—it directly enables deeper insight into why existing methods work, where they fail, and how to adapt them to novel problems. When you read a paper proposing a new architecture, the mathematical notation becomes transparent rather than opaque. When debugging a model that isn't learning, the gradient flow through your network becomes traceable rather than mysterious. The investment in mastering these foundations pays dividends throughout every subsequent chapter of this book and throughout your career as a practitioner or researcher in this field.

Linear Algebra: The Language of Spatial Transformations

Linear algebra provides the vocabulary for describing how data moves through the layers of a neural network, how images are transformed and manipulated, and how high-dimensional visual information can be compressed, projected, and analyzed. In computer vision, we routinely work with images represented as matrices or higher-dimensional tensors, and understanding matrix operations is fundamental to understanding everything from simple image filters to complex neural network architectures.

Vectors and Vector Spaces

A vector represents both a direction and a magnitude in space. In computer vision, we frequently encounter vectors in many contexts: pixel intensity values at different color channels, feature representations extracted by a network, or the gradients that guide learning. A column vector of dimension $n$ can be written as:

$$\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}$$

The norm (or length) of a vector measures its magnitude and is computed as:

$$\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\mathbf{v}^\top \mathbf{v}}$$

Different norm types serve different purposes in computer vision. The L2 norm (Euclidean distance) is ubiquitous in loss functions, measuring the straight-line distance between two points. The L1 norm, being more robust to outliers, appears in certain regression losses and sparse coding formulations. The L∞ norm, measuring the maximum absolute component, finds applications in certain normalization schemes and adversarial robustness analysis.

The dot product (or inner product) between two vectors measures their alignment and is fundamental to understanding projections, similarities, and attention mechanisms:

$$\mathbf{v} \cdot \mathbf{w} = \mathbf{v}^\top \mathbf{w} = \sum_{i=1}^{n} v_i w_i = \|\mathbf{v}\| \|\mathbf{w}\| \cos\theta$$

This relationship reveals why the dot product serves as a natural similarity measure—when two vectors point in the same direction ($\theta = 0$), their dot product equals the product of their magnitudes; when they are orthogonal ($\theta = 90°$), the dot product is zero.

Matrices: Structure for Transformation

A matrix is a two-dimensional array of numbers that encodes a linear transformation—a function that maps vectors from one space to another while preserving the essential structure of vector addition and scalar multiplication. If $\mathbf{A}$ is an $m \times n$ matrix and $\mathbf{x}$ is an $n$-dimensional column vector, their product produces an $m$-dimensional vector:

$$\mathbf{y} = \mathbf{A}\mathbf{x}, \quad y_i = \sum_{j=1}^{n} a_{ij} x_j$$

Each element $a_{ij}$ in the matrix specifies how much the $j$-th component of the input contributes to the $i$-th component of the output. This compact notation hides enormous computational power: a single matrix multiplication simultaneously computes all outputs from all inputs.

In computer vision, matrices appear everywhere. An image is naturally represented as a matrix (for grayscale) or a three-dimensional tensor (for color). A convolution operation between an image and a filter corresponds to a matrix multiplication when the filter is "unrolled" into a Toeplitz matrix. The layer operations in a neural network—linear transformations followed by nonlinearities—are compositions of matrix multiplications and element-wise operations.

The transpose of a matrix, denoted $\mathbf{A}^\top$, swaps its rows and columns. The transpose appears frequently in computing loss functions (comparing predictions to targets), in the derivation of backpropagation, and in the relationship between covariance matrices and their inverses.

Matrix multiplication is not commutative: in general, $\mathbf{AB} \neq \mathbf{BA}$. This non-commutativity has profound implications. In a neural network, the order of layers matters—applying a convolution then a ReLU produces different results than applying ReLU then the convolution (though in this case, ReLU being applied element-wise does commute, but with batch normalization, the order can matter).

Eigenvalues and Eigenvectors: The Essence of Transformation

Among all the vectors in a space, eigenvectors hold a special relationship with their matrix: when a matrix acts on an eigenvector, it merely scales that vector without changing its direction:

$$\mathbf{A}\mathbf{v} = \lambda \mathbf{v}$$

Here, $\mathbf{v}$ is an eigenvector of $\mathbf{A}$ and $\lambda$ is its corresponding eigenvalue. This deceptively simple equation captures deep structural properties of linear transformations.

Consider what this means geometrically. Most vectors, when transformed by a matrix, get "tilted" into new directions. Eigenvectors are special—they map only to scalar multiples of themselves. The eigenvalue tells us the factor by which the eigenvector is stretched (positive values), compressed (negative values between -1 and 0), reflected (negative values less than -1), or left unchanged (λ = 1).

Eigenvalues and eigenvectors are fundamental to understanding principal component analysis (PCA), which we will encounter repeatedly in computer vision for dimensionality reduction, face recognition, and data visualization. PCA finds the eigenvectors of the data's covariance matrix; these principal components point in the directions of maximum variance in the data, and the corresponding eigenvalues indicate how much variance is captured along each direction.

In deep learning, the eigenvalues of the Hessian matrix (second derivatives of the loss function) reveal critical aspects of the loss landscape—their magnitude indicates curvature, and vanishing eigenvalues indicate flat regions that can slow learning. Eigenvalue analysis also appears in spectral normalization for controlling Lipschitz constants of neural networks, in understanding attention mechanisms, and in the theoretical analysis of neural network expressivity.

A matrix is symmetric ($\mathbf{A} = \mathbf{A}^\top$) if and only if it has real eigenvalues and a complete set of orthonormal eigenvectors. Symmetric matrices appear frequently because many important matrices in optimization and statistics are symmetric: covariance matrices, the Hessian of a loss function, and Laplacian matrices used in graph-based visual processing.

Singular Value Decomposition: The Cornerstone of Matrix Analysis

Singular value decomposition (SVD) provides the most generally applicable and numerically stable way to factor any matrix, including non-square and singular matrices. Any matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$ can be decomposed as:

$$\mathb