Mathematical Foundations for Computer Vision

Mathematical Foundations for Computer Vision

The remarkable progress in computer vision over the past decade has been fundamentally powered by deep learning, yet beneath the architecture innovations and training tricks lies an essential scaffold of mathematical knowledge that every practitioner must command. Understanding why convolutional neural networks learn hierarchical feature representations, how backpropagation efficiently computes gradients through millions of parameters, or why certain loss functions are preferred for specific visual recognition tasks requires a solid grasp of the underlying mathematics. This chapter establishes these foundational concepts, building from the linear algebra that enables efficient computation on high-dimensional data, through the calculus that powers learning algorithms, to the probability theory and information theory that frame visual understanding as statistical inference. These mathematical tools are not merely academic prerequisites but rather the conceptual lenses through which modern computer vision systems are designed, analyzed, and improved.

Linear Algebra Fundamentals

Linear algebra provides the language and computational framework for nearly all operations in computer vision systems. Images themselves are naturally represented as matrices of pixel intensities, and the feature representations learned by neural networks are high-dimensional vectors in learned spaces. A thorough understanding of linear transformations, matrix decompositions, and vector spaces is therefore essential for comprehending how visual information flows and transforms through deep networks.

Vectors, Matrices, and Tensor Operations

At the most fundamental level, an image can be represented as a matrix $\mathbf{I} \in \mathbb{R}^{H \times W}$, where $H$ and $W$ denote the height and width of the image respectively, and each entry $I_{ij}$ contains the intensity value at pixel position $(i, j)$. For color images, we typically employ a three-dimensional tensor $\mathbf{I} \in \mathbb{R}^{H \times W \times C}$, where $C = 3$ for RGB channels. This tensor representation extends naturally to batches of images, giving $\mathbf{I} \in \mathbb{R}^{B \times H \times W \times C}$, where $B$ denotes the batch size. Modern deep learning frameworks such as PyTorch and TensorFlow are built around efficient tensor operations that exploit the structure of these high-dimensional data structures.

Vectors in computer vision contexts rarely represent simple directed quantities as in physics; instead, they typically encode features or embeddings in high-dimensional spaces. A feature vector $\mathbf{f} \in \mathbb{R}^{D}$ extracted from an image might have dimensionality $D$ ranging from hundreds to thousands. The Euclidean distance between feature vectors, computed as $\|\mathbf{f}_1 - \mathbf{f}_2\|_2 = \sqrt{\sum_{i=1}^{D}(f_{1i} - f_{2i})^2}$, serves as a fundamental similarity measure for tasks ranging from image retrieval to face recognition. The cosine similarity, defined as $\frac{\mathbf{f}_1 \cdot \mathbf{f}_2}{\|\mathbf{f}_1\|\|\mathbf{f}_2\|}$, provides a scale-invariant comparison that is particularly useful when dealing with varying image conditions.

Matrix multiplication constitutes the computational backbone of neural network operations. Given input activations $\mathbf{X} \in \mathbb{R}^{N \times D_{in}}$ and weight matrix $\mathbf{W} \in \mathbb{R}^{D_{in} \times D_{out}}$, the output activations are computed as $\mathbf{Y} = \mathbf{X}\mathbf{W}$, where $\mathbf{Y} \in \mathbb{R}^{N \times D_{out}}$. The computational complexity of this operation is $O(N \cdot D_{in} \cdot D_{out})$, which becomes substantial when dealing with modern architectures containing millions or billions of parameters. Understanding this complexity is crucial for memory allocation and computational optimization.

The element-wise operations, including addition, multiplication, and activation functions, interact with matrix multiplication through the broadcasting rules of each framework. When adding a bias vector $\mathbf{b} \in \mathbb{R}^{D_{out}}$ to the matrix product $\mathbf{X}\mathbf{W}$, the framework automatically broadcasts the bias across all rows of the result, adding $\mathbf{b}$ to each of the $N$ output vectors in the batch.

Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors reveal fundamental properties of linear transformations that are directly relevant to understanding neural network behavior. For a square matrix $\mathbf{A} \in \mathbb{R}^{D \times D}$, an eigenvector $\mathbf{v} \in \mathbb{R}^{D}$ and its corresponding eigenvalue $\lambda \in \mathbb{R}$ satisfy the fundamental relationship:

$$\mathbf{A}\mathbf{v} = \lambda \mathbf{v}$$

This equation states that applying the linear transformation represented by $\mathbf{A}$ to the eigenvector $\mathbf{v}$ produces a result that is simply a scaled version of $\mathbf{v}$ itself, with the scaling factor given by $\lambda$. The eigenvalues thus characterize how the transformation stretches or compresses different directions in space.

In the context of neural networks, the spectral properties of weight matrices significantly influence learning dynamics. The spectral radius—the magnitude of the largest eigenvalue—determines whether repeated application of a weight matrix leads to explosion or decay of activations. Networks with eigenvalues exceeding 1 in magnitude may suffer from exploding gradients, while eigenvalues much smaller than 1 lead to vanishing gradients, both phenomena presenting challenges for training deep architectures.

Consider a simple recurrent neural network where the hidden state is updated as $\mathbf{h}_{t+1} = \mathbf{W}\mathbf{h}_t$. After $T$ time steps, the hidden state becomes $\mathbf{h}_T = \mathbf{W}^T \mathbf{h}_0$. If the largest eigenvalue of $\mathbf{W}$ exceeds 1, the hidden state will grow exponentially with $T$, leading to numerical overflow. Conversely, if all eigenvalues are smaller than 1, the hidden state will decay to zero. Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) were designed, in part, to mitigate these eigenvalue-related instabilities through gating mechanisms that control information flow.

The eigendecomposition $\mathbf{A} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{-1}$, where $\mathbf{V}$ contains the eigenvectors as columns and $\boldsymbol{\Lambda}$ is a diagonal matrix of eigenvalues, provides a canonical representation of a matrix. However, not all matrices admit eigendecomposition—only matrices that are diagonalizable, meaning they have a complete set of linearly independent eigenvectors. This limitation motivates the study of alternative decompositions that exist for any matrix.

Singular Value Decomposition

Singular Value Decomposition (SVD) extends eigendecomposition to any matrix, including non-square matrices, making it universally applicable in computer vision. For any matrix $\mathbf{A} \in \mathbb{R}^{M \times N}$, the SVD is expressed as:

$$\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T$$

where $\mathbf{U} \in \mathbb{R}^{M \times M}$ and $\mathbf{V} \in \mathbb{R}^{N \times N}$ are orthogonal matrices, and $\boldsymbol{\Sigma} \in \mathbb{R}^{M \times N}$ is a diagonal matrix containing the singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0$, with $r = \min(M, N)$ being the rank of the matrix.

The columns of $\mathbf{U}$ are called left singular vectors and form an orthonormal basis for the column space of $\mathbf{A}$. The columns of $\mathbf{V}$ are called right singular vectors and form an orthonormal basis for the row space of $\mathbf{A}$. The singular values encode the "energy" or importance of each component in this decomposition.

SVD provides the optimal low-rank approximation of a matrix in the Frobenius norm sense. The best rank-$k$ approximation $\mathbf{A}_k$ to $\mathbf{A}$ is obtained by keeping only the $k$ large