The Geometry of Meaning: Mathematical Foundations of Vector Spaces

In the architecture of modern semantic search systems, the vector space serves as the foundational substrate upon which meaning is mathematically encoded and computationally manipulated. The premise that semantic relationships can be captured as geometric structures—distances, angles, directions—in a high-dimensional space is not merely an analogy; it is the operational core of models from Word2Vec to modern large language models. This chapter establishes the rigorous mathematical framework that transforms this premise from intuitive metaphor into engineering practice. We will move beyond casual references to "vector embeddings" to confront the precise algebraic and topological properties of the spaces that contain them. Understanding these properties is not an academic exercise; it directly determines the choice of similarity metrics, the design of indexing structures, the interpretation of model outputs, and the very limits of what can be retrieved from a corpus. A system architect who treats all vectors as points in a generic Euclidean space, measured solely by Euclidean distance, will build a system that is mathematically brittle and semantically naive. Conversely, an architect who understands the normed vector space structure, the implications of an inner product, and the manifold hypothesis can make informed choices that align the mathematical machinery with the intrinsic geometry of meaning.

1.1 Vector Spaces: The Algebraic Canvas of Meaning

At its most abstract, a vector space (or linear space) is a collection of objects called vectors, which can be added together and multiplied ("scaled") by numbers, called scalars. The scalars are often real numbers, but can also be complex numbers or elements of more general fields. The operations of vector addition and scalar multiplication must satisfy a consistent set of axioms (closure, associativity, commutativity of addition, existence of additive identity and inverses, distributivity of scalar multiplication with respect to vector and field addition, and compatibility of scalar multiplication).

Definition (Vector Space): A set $V$ is a vector space over a field $\mathbb{F}$ (typically $\mathbb{R}$ or $\mathbb{C}$) if there exist two binary operations: 1. Vector addition: $+: V \times V \rightarrow V$, denoted $(u, v) \mapsto u + v$. 2. Scalar multiplication: $\cdot: \mathbb{F} \times V \rightarrow V$, denoted $(\alpha, v) \mapsto \alpha v$;

that satisfy the eight axioms for all $u, v, w \in V$ and $\alpha, \beta \in \mathbb{F}$: 1. $u + (v + w) = (u + v) + w$ (Associativity of addition) 2. $u + v = v + u$ (Commutativity of addition) 3. $\exists \mathbf{0} \in V$ such that $v + \mathbf{0} = v$ (Additive identity) 4. $\exists (-v) \in V$ such that $v + (-v) = \mathbf{0}$ (Additive inverse) 5. $\alpha \cdot (u + v) = \alpha \cdot u + \alpha \cdot v$ (Distributivity of scalar multiplication over vector addition) 6. $(\alpha + \beta) \cdot v = \alpha \cdot v + \beta \cdot v$ (Distributivity of scalar multiplication over field addition) 7. $(\alpha \beta) \cdot v = \alpha \cdot (\beta \cdot v)$ (Compatibility of scalar multiplication) 8. $1 \cdot v = v$, where $1$ is the multiplicative identity in $\mathbb{F}$.

Example 1.1.1 (The Euclidean Space $\mathbb{R}^n$): The most familiar vector space. Vectors are ordered $n$-tuples of real numbers: $v = (v_1, v_2, ..., v_n)$. Addition and scalar multiplication are performed component-wise: $$(v_1, ..., v_n) + (w_1, ..., w_n) = (v_1+w_1, ..., v_n+w_n)$$ $$\alpha (v_1, ..., v_n) = (\alpha v_1, ..., \alpha v_n)$$ This is the canonical space for word embeddings (e.g., GloVe, fastText) and many early vision models, where each dimension corresponds to a learned feature.

Example 1.1.2 (Function Spaces): The set of all real-valued functions defined on a domain $D$ forms a vector space under pointwise addition and scalar multiplication: $(f+g)(x) = f(x) + g(x)$, $(\alpha f)(x) = \alpha f(x)$. This abstract view is crucial for understanding kernel methods and certain neural network activations, where a layer's output can be seen as a point in a function space.

Example 1.1.3 (The Space of Probability Distributions): While not a vector space under standard addition (convex combinations are used instead), the set of all discrete probability distributions over a vocabulary of size $|V|$ lives on the probability simplex, a $(|V|-1)$-dimensional surface in $\mathbb{R}^{|V|}$. This geometric constraint is vital for interpreting softmax outputs and for applications requiring probabilistic semantics.

The power of the vector space abstraction lies in its generality. It allows us to reason about entities as diverse as words, images, graphs, and user profiles using the same geometric language. The specific *realization* of a vector space—its basis, its dimensionality, its metric—is what gives it semantic content.

1.1.1 Basis, Dimension, and Coordinate Systems

A basis of a vector space $V$ is a linearly independent set of vectors that spans $V$. This means every vector in $V$ can be written uniquely as a linear combination of the basis vectors. The number of vectors in a basis is the dimension of the space. The choice of basis is not unique and is often arbitrary; however, the *existence* of a basis and the dimension are fundamental invariants.

In $\mathbb{R}^n$, the standard basis is $e_1 = (1,0,...,0), e_2 = (0,1,...,0), ..., e_n = (0,0,...,1)$. A vector $v = (v_1, ..., v_n)$ has coordinates $(v_1, ..., v_n)$ relative to this basis. In machine learning, the basis is almost never explicitly constructed. Instead, we work with an *implicit* basis defined by the model's architecture. For instance, in a word embedding matrix $E \in \mathbb{R}^{d \times |V|}$, each column is a word vector in $\mathbb{R}^d$. The columns themselves are not necessarily orthogonal or independent. The "basis" here is effectively the set of latent dimensions the model has learned, but these are linear combinations of a hypothetical, unknown orthogonal basis.

Key Insight for Practitioners: The dimension $d$ of an embedding space is a hyperparameter with profound geometric consequences. A higher $d$ increases the capacity to represent intricate relationships (more degrees of freedom) but exacerbates the curse of dimensionality (sparsity, distance concentration). The choice of $d$ (e.g., 300 for Word2Vec, 768 for BERT base) is a trade-off between representational power and computational/statistical efficiency, rooted in the geometry of the space.

1.2 Normed Spaces and Metrics: Measuring "Distance" in Meaning

To discuss "similarity" or "dissimilarity" between semantic vectors, we need a way to measure the "size" of a vector (a norm) and the "distance" between two vectors (a metric). This moves us from a pure vector space to a normed vector space and then to a metric space.

1.2.1 Norms: The Length of a Vector

A norm on a vector space $V$ is a function $\| \cdot \|: V \rightarrow \mathbb{R}$ that assigns a non-negative length to each vector, satisfying: 1. Positive definiteness: $\|v\| \geq 0$, and $\|v\| = 0 \iff v = \mathbf{0}$. 2. Absolute homogeneity: $\|\alpha v\| = |\alpha| \|v\|$ for all $\alpha \in \mathbb{F}$. 3. Triangle inequality: $\|u + v\| \leq \|u\| + \|v\|$.

A vector space equipped with a norm is a normed vector space.

The most common norms on $\mathbb{R}^n$ are the $L^p$ norms: $$\|v\|_p = \left( \sum_{i=1}^{n} |v_i|^p \right)^{1/p} \quad \text{for } 1 \leq p < \infty$$ The $L^\infty$ norm (maximum norm) is: $$\|v\|_\infty = \max_{i} |v_i|$$

Special Cases:

• $p=1$: **Manhattan norm** ($L^1$). $\|v\|_1 = \sum |v_i|$. Measures total absolute deviation. Sensitive to all dimensions, robust to outliers in a specific sense.
• $p=2$: **Euclidean norm** ($L^2$). $\|v\|_2 = \sqrt{\sum v_i^2}$. The standard geometric length. Heavily penalizes large deviations in any single dime