Foundations of Time Series Analysis: Concepts, Terminology, and Theoretical Framework

Foundations of Time Series Analysis: Concepts, Terminology, and Theoretical Framework

The analysis of time series data stands as one of the most consequential intellectual achievements of twentieth-century statistics and applied mathematics. From the early investigations of agricultural yields and生理学 rhythms by Yule and Walker in the 1920s, through the development of Box-Jenkins methodology in the 1960s and 1970s, to the contemporary revolution brought about by deep learning architectures, the field has consistently strived toward a single overarching objective: extracting meaningful patterns and generating accurate predictions from sequences of observations ordered in time. This book undertakes the ambitious task of tracing this intellectual journey from first principles to frontier architectures, with particular emphasis on the transformative potential of transformer-based approaches. Before we can appreciate the power of modern deep learning methods for time series forecasting, however, we must first establish the rigorous mathematical and conceptual foundations upon which the entire discipline rests.

This opening chapter serves as the essential groundwork for everything that follows. We begin with precise definitions of stochastic processes and their key properties—stationarity in both its strict and weak forms, ergodicity as a condition for learning from single observed paths, and the mathematical characterization of temporal dependencies. We then introduce the fundamental tools of time series analysis: the autocorrelation function and partial autocorrelation function, which characterize linear temporal structure, and spectral density analysis, which reveals the frequency-domain composition of a process. The representation theorems connecting time and frequency domains receive careful treatment, with particular attention to Wold's decomposition theorem, which demonstrates that any (non-deterministic) stationary process can be expressed as a linear combination of past innovation terms. Finally, we examine the forecasting paradigm in its full generality, distinguishing one-step from multi-step prediction, direct from recursive strategies, and establishing the theoretical limits of predictability that no method—however sophisticated—can overcome. The chapter concludes with a preview of the methodological landscape that this book will traverse.

1.1 Stochastic Processes: Mathematical Foundations

1.1.1 Definition and Probability Space Structure

A stochastic process $\{X_t : t \in \mathcal{T}\}$ is a collection of random variables indexed by a parameter $t$, which we interpret as time. The index set $\mathcal{T}$ may be discrete ($\mathcal{T} = \mathbb{Z}$ or $\mathcal{T} = \mathbb{N}$) or continuous ($\mathcal{T} = \mathbb{R}$). Each random variable $X_t$ is defined on a underlying probability space $(\Omega, \mathcal{F}, \mathbb{P})$, where $\Omega$ is the sample space of all possible outcomes, $\mathcal{F}$ is the sigma-algebra of measurable events, and $\mathbb{P}$ is the probability measure. For each fixed $\omega \in \Omega$, the mapping $t \mapsto X_t(\omega)$ traces a realization or sample path of the process—a single observed time series.

This dual perspective—viewing the process simultaneously as a collection of random variables at each time point and as a collection of functions of time parameterized by outcomes—lies at the heart of all time series analysis. When we observe a single historical sequence $\{x_1, x_2, \ldots, x_T\}$, we are observing a single sample path from an ensemble of infinitely many possible realizations. The fundamental challenge of time series analysis is to draw inferences about the underlying stochastic process from this single observed path.

The finite-dimensional distributions of a stochastic process are the joint distributions of all finite subsets of the process:

$$\mathbb{P}(X_{t_1} \leq x_1, X_{t_2} \leq x_2, \ldots, X_{t_n} \leq x_n)$$

for all $n \in \mathbb{N}$, all time indices $t_1, t_2, \ldots, t_n \in \mathcal{T}$, and all real values $x_1, x_2, \ldots, x_n$. Kolmogorov's existence theorem establishes that specifying a consistent family of finite-dimensional distributions determines a unique stochastic process (up to equivalence). This result, while abstract, provides the theoretical foundation for all parametric modeling in time series analysis.

1.1.2 Moments and Characteristic Properties

The mean function of a stochastic process is defined as

$$\mu(t) = \mathbb{E}[X_t] = \int_\Omega X_t(\omega) \, d\mathbb{P}(\omega)$$

whenever the expectation exists. The mean function characterizes the expected value of the process at each time point. The autocovariance function captures the linear relationship between values at different time points:

$$\gamma(s, t) = \operatorname{Cov}(X_s, X_t) = \mathbb{E}[(X_s - \mu(s))(X_t - \mu(t))]$$

The autocovariance function is fundamental to understanding temporal dependence. Note that $\gamma(t, t) = \operatorname{Var}(X_t)$, the variance at time $t$. The autocovariance function is symmetric: $\gamma(s, t) = \gamma(t, s)$, and it is positive semi-definite in the sense that for any finite collection of time points and any constants $a_1, \ldots, a_n$:

$$\sum_{i=1}^n \sum_{j=1}^n a_i a_j \gamma(t_i, t_j) \geq 0$$

The autocorrelation function (ACF), defined as

$$\rho(s, t) = \frac{\gamma(s, t)}{\sqrt{\gamma(s, s) \gamma(t, t)}}$$

normalizes the autocovariance by the standard deviations, yielding a measure that always lies in $[-1, 1]$. When $\rho(s, t) = 0$, the values $X_s$ and $X_t$ are linearly uncorrelated (though they may be dependent through nonlinear relationships).

1.1.3 Examples of Fundamental Stochastic Processes

Several stochastic processes serve as foundational building blocks and reference points throughout time series analysis. The white noise process $\{ \varepsilon_t \}$ is defined as a sequence of uncorrelated random variables with zero mean and constant variance:

$$\mathbb{E}[\varepsilon_t] = 0, \quad \operatorname{Var}(\varepsilon_t) = \sigma^2, \quad \operatorname{Cov}(\varepsilon_t, \varepsilon_s) = 0 \text{ for } t \neq s$$

White noise is the fundamental "atomic" unit of time series modeling. Every stationary linear process can be understood as a linear combination of white noise terms, as we shall see when we develop the Wold representation.

The random walk process is defined recursively by $X_t = X_{t-1} + \varepsilon_t$, with $X_0$ typically set to zero. This yields $X_t = \sum_{i=1}^t \varepsilon_i$. The random walk plays a distinguished role in both theory and applications—it is the simplest non-stationary process and the continuous-time analogue (Brownian motion) arises as the limit of aggregated random walks. Its variance grows linearly with time: $\operatorname{Var}(X_t) = t\sigma^2$, which means that unlike stationary processes, the uncertainty about future values grows without bound as we look further ahead.

The sine-wave process provides the archetype of purely deterministic temporal structure: $X_t = A \cos(2\pi f t) + B \sin(2\pi f t)$, where $A$ and $B$ are random variables and $f$ is the frequency. This process is mean square periodic with period $1/f$. The random amplitude and phase shift make each realization unique, yet all realizations share the same deterministic frequency structure.

1.2 Stationarity: The Cornerstone of Time Series Analysis

1.2.1 Strict Stationarity

Strict stationarity (also called strong stationarity or strong-sense stationarity) requires that the joint distribution of any collection of time points is invariant under shifts in time. Formally, a stochastic process $\{X_t\}$ is strictly stationary if for all $n \in \mathbb{N}$, all time points $t_1, t_2, \ldots, t_n \in \mathcal{T}$, all shifts $h \in \mathcal{T}$, and all Borel sets $B_1, \ldots, B_n \subset \mathbb{R}$:

$$\mathbb{P}(X_{t_1} \in B_1, \ldots, X_{t_n} \