Mathematical Foundations of Time Series Analysis

Mathematical Foundations of Time Series Analysis

Introduction: The Language of Temporal Processes

The analysis of time series data—observations collected sequentially over time—forms the backbone of forecasting across numerous scientific and engineering domains. Before one can effectively model, predict, or understand any temporal phenomenon, whether it be stock market fluctuations, weather patterns, neural activity, or manufacturing metrics, one must first establish a rigorous mathematical framework within which these observations can be properly characterized and analyzed. This chapter develops that framework with mathematical precision, introducing the conceptual machinery that enables rigorous reasoning about stochastic temporal processes.

The fundamental challenge in time series analysis arises from the inherent interdependence among observations. Unlike independent and identically distributed (i.i.d.) random variables, where each observation carries information independent of its predecessors, time series observations are typically correlated with past values in ways that encode the underlying dynamics of the system generating the data. Capturing and exploiting these temporal dependencies is the central objective of the field, and achieving this objective requires a careful development of probability theory, stochastic processes, and spectral analysis.

We begin with the foundational concept of a probability space and build toward the more sophisticated notion of a stochastic process, which provides the mathematical abstraction for a sequence of random variables indexed by time. From this foundation, we develop the critical concepts of stationarity—strict and weak—which characterize processes whose statistical properties remain invariant under time shifts. Stationarity serves as a organizing principle throughout time series analysis, as many powerful estimation and prediction techniques require this property or variants of it. We then examine ergodicity, which connects ensemble averages to time averages and justifies the common practice of estimating population parameters from single realizations of a process. The Wold decomposition theorem, one of the most profound results in the theory of stationary processes, establishes that any stationary process can be represented as a sum of a deterministic component and a moving average of white noise, thereby providing a canonical structure for understanding temporal dependencies. Following this theoretical development, we introduce the autocovariance and autocorrelation functions, which quantify linear temporal dependencies, and discuss their estimation from data. White noise processes, serving as the fundamental building blocks of more complex time series models, receive detailed treatment. The chapter concludes with an exploration of frequency-domain representations, including the spectral density function and the Wiener-Khinchin theorem, which establishes the fundamental duality between the autocovariance function and the power spectrum.

Probability Spaces and Stochastic Processes

The Mathematical Framework of Randomness

A probability space provides the rigorous foundation for any probabilistic reasoning. Formally, a probability space is a triple $(\Omega, \mathcal{F}, P)$ consisting of a sample space $\Omega$, a sigma-algebra $\mathcal{F}$ of subsets of $\Omega$, and a probability measure $P$ defined on $\mathcal{F}$. The sample space $\Omega$ represents the set of all possible outcomes of a random experiment; the sigma-algebra $\mathcal{F}$ contains the events to which we can assign probabilities; and the probability measure $P$ assigns probabilities to events in $\mathcal{F}$ satisfying the Kolmogorov axioms. This abstract formulation, developed by Andrey Kolmogorov in the 1930s, provides the logical foundation for probability theory and consequently for all of statistical inference.

Within this framework, a random variable $X$ is a measurable function $X: \Omega \to \mathbb{R}$ that maps outcomes in the sample space to real numbers. For a discrete random variable, we define the probability mass function $p_X(x) = P(X = x)$, while for a continuous random variable, we define the probability density function $f_X(x)$ such that $P(a \leq X \leq b) = \int_a^b f_X(x) dx$. The cumulative distribution function $F_X(x) = P(X \leq x)$ provides a unified representation applicable to both cases.

For time series analysis, we require the concept of a sigma-algebra generated by a collection of random variables. If $X_1, X_2, \ldots, X_n$ are random variables, the sigma-algebra they generate, denoted $\sigma(X_1, X_2, \ldots, X_n)$, contains all events that can be expressed in terms of these variables. This concept becomes essential when we discuss conditional expectations and information structures in temporal processes.

From Random Variables to Stochastic Processes

A stochastic process is a collection of random variables $\{X_t : t \in \mathcal{T}\}$ defined on a common probability space, where the index set $\mathcal{T}$ typically represents time. When $\mathcal{T} = \{1, 2, 3, \ldots\}$, we have a discrete-time process; when $\mathcal{T} = \mathbb{R}$ or $\mathcal{T} = [0, \infty)$, we have a continuous-time process. This book focuses primarily on discrete-time processes, as they arise most commonly in practical forecasting applications, though the continuous-time theory follows parallel principles.

For a given outcome $\omega \in \Omega$, the mapping $t \mapsto X_t(\omega)$ defines a sample path or realization of the process. Thus, a stochastic process can be viewed from two complementary perspectives: as a collection of random variables indexed by time, or as a collection of sample paths through the state space. This dual perspective is fundamental to understanding both the probabilistic structure of the process and the behavior of individual realizations.

The finite-dimensional distributions of a stochastic process provide a complete probabilistic characterization. For any collection of time indices $t_1, t_2, \ldots, t_k \in \mathcal{T}$, the joint distribution of $(X_{t_1}, X_{t_2}, \ldots, X_{t_k})$ is given by the cumulative distribution function

$$F_{t_1, \ldots, t_k}(x_1, \ldots, x_k) = P(X_{t_1} \leq x_1, \ldots, X_{t_k} \leq x_k)$$

Kolmogorov's extension theorem states that a stochastic process is uniquely determined (up to indistinguishability) by its finite-dimensional distributions, provided these distributions satisfy certain consistency conditions. This theorem justifies working with finite-dimensional distributions without explicitly constructing the infinite-dimensional probability space.

Information and Filtrations

The concept of a filtration proves essential for sequential learning and prediction. A filtration $\{\mathcal{F}_t\}$ is an increasing sequence of sigma-algebras such that $\mathcal{F}_s \subseteq \mathcal{F}_t$ for $s < t$. The sigma-algebra $\mathcal{F}_t$ represents all information available up to and including time $t$. A stochastic process $\{X_t\}$ is said to be adapted to the filtration $\{\mathcal{F}_t\}$ if $X_t$ is measurable with respect to $\mathcal{F}_t$ for each $t$.

This framework enables precise statements about prediction and learning. The conditional expectation $E[X_{t+h} | \mathcal{F}_t]$ represents the optimal prediction of $X_{t+h}$ given all information available at time $t$, where "optimal" means minimizing mean squared error. The conditional variance $\text{Var}(X_{t+h} | \mathcal{F}_t)$ represents the remaining uncertainty in this prediction.

Stationarity: The Foundation of Temporal Analysis

The concept of stationarity provides the most important structural assumption in time series analysis. A stationary process exhibits statistical properties that remain invariant under time shifts, meaning the process "looks the same" regardless of when we observe it. This property is crucial because it permits statistica