Foundations of Stochastic Processes and Time Series Analysis

Time series data—sequences of observations recorded sequentially over time—permeate the natural and social sciences, engineering, finance, and virtually every domain of human inquiry. Unlike cross-sectional data, the temporal ordering imbues each observation with a causal and predictive relationship with its predecessors, rendering the analysis both uniquely challenging and profoundly consequential. The central objective of time series forecasting is to exploit this temporal dependence to estimate future values based on past and present information. However, to do so rigorously, one must first construct a solid probabilistic foundation that precisely characterizes the stochastic mechanisms generating the data. This chapter establishes that foundation, introducing the core concepts of stochastic process theory that underpin all subsequent modeling approaches, from classical autoregressive models to modern deep learning architectures. We will develop the mathematical language to describe dependence, stationarity, and spectral properties, culminating in the fundamental theorems that justify linear forecasting and a formal definition of the forecasting problem as a sequential decision-making process under uncertainty.

1.1 Stochastic Processes: The Mathematical Primitive

A stochastic process (or random process) is a collection of random variables indexed by time. Formally, let $(\Omega, \mathcal{F}, P)$ be a probability space. A stochastic process is a family $\{X_t : t \in T\}$, where each $X_t$ is a random variable mapping from $\Omega$ to some state space $\mathcal{S}$ (usually $\mathbb{R}$ or $\mathbb{R}^d$), and $T$ is the index set, which represents time. In this text, we primarily consider discrete-time processes where $T = \mathbb{Z}$ (all integers) or $T = \mathbb{N}$ (non-negative integers), and continuous-time processes where $T = \mathbb{R}$ or $\mathbb{R}^+$. The entire process is a function $X: \Omega \times T \to \mathcal{S}$; for a fixed outcome $\omega \in \Omega$, the function $t \mapsto X_t(\omega)$ is a sample path or realization of the process.

The probabilistic description of a stochastic process requires specifying the finite-dimensional distributions (FDDs): for any finite set of time points $t_1, \dots, t_n \in T$ and any Borel sets $B_1, \dots, B_n \subseteq \mathcal{S}$, the joint probability $$ P(X_{t_1} \in B_1, \dots, X_{t_n} \in B_n) $$ must be defined. A process is completely specified by its family of all such FDDs that are consistent (Kolmogorov’s extension theorem guarantees existence under consistency conditions). However, working with full FDDs is often intractable. Therefore, we introduce summary characteristics that capture essential features of the process with fewer parameters, often by focusing on low-order moments.

1.1.1 Moments and Dependence Measures

The mean function of a process $\{X_t\}$ is $\mu_t = E[X_t]$, provided it exists. The autocovariance function (ACVF) is defined for any two time points $s, t \in T$ as $$ \gamma(s, t) = \text{Cov}(X_s, X_t) = E[(X_s - \mu_s)(X_t - \mu_t)], $$ assuming second moments exist. The autocorrelation function (ACF) is the normalized version: $$ \rho(s, t) = \frac{\gamma(s, t)}{\sqrt{\gamma(s,s)\gamma(t,t)}}. $$ These functions quantify the linear dependence between the process at different times. For a stationary process (defined below), $\gamma(s,t)$ depends only on the lag $h = t - s$, and we write $\gamma(h)$.

A crucial concept is the partial autocorrelation function (PACF), which measures the correlation between $X_t$ and $X_{t-h}$ after removing the linear influence of the intermediate observations $X_{t-1}, \dots, X_{t-h+1}$. The PACF, denoted $\phi_{hh}$, is the last coefficient in the linear regression of $X_t$ on $X_{t-1}, \dots, X_{t-h}$. It is fundamental in identifying autoregressive models.

1.2 Stationarity: The Cornerstone of Time Series Analysis

Stationarity is a property that ensures the probabilistic structure of the process is invariant over time. It is the primary assumption that allows us to learn from historical data and project into the future. Two main notions exist: strict stationarity and weak (wide-sense) stationarity.

1.2.1 Strict-Sense Stationarity (SSS)

A process $\{X_t\}$ is strict-sense stationary if its finite-dimensional distributions are invariant under time shifts. That is, for any $n \in \mathbb{N}$, any $t_1, \dots, t_n \in T$, and any integer $k$ such that $t_1+k, \dots, t_n+k \in T$, the joint distribution of $(X_{t_1}, \dots, X_{t_n})$ is identical to that of $(X_{t_1+k}, \dots, X_{t_n+k})$. Equivalently, the process is statistically indistinguishable from its time-shifted version.

Implications: SSS implies that all moments (if they exist) are constant over time: $E[X_t] = \mu$ (constant), $\text{Var}(X_t) = \sigma^2$ (constant), and more generally, the joint distribution of $(X_t, X_{t+h})$ depends only on $h$, not on $t$. However, SSS is a very strong condition and often too restrictive for practical applications, especially for processes with trends or changing volatility.

1.2.2 Weak-Sense Stationarity (WSS) / Covariance Stationarity

A process $\{X_t\}$ is weak-sense stationary (or covariance stationary) if: 1. The mean function is constant: $E[X_t] = \mu$ for all $t \in T$. 2. The autocovariance function depends only on the lag: $\gamma(s,t) = \gamma(t-s)$ for all $s,t \in T$. That is, $\gamma(h)$ exists and is finite for all integer lags $h$. 3. The process is second-order (i.e., $E[X_t^2] < \infty$ for all $t$).

WSS is a much weaker condition than SSS. If a process is SSS and has finite second moments, then it is WSS. The converse is not true: a WSS process need not be SSS. For example, a Gaussian process (where all FDDs are multivariate normal) is SSS if and only if it is WSS, because the normal distribution is fully characterized by its mean and covariance. For non-Gaussian processes, WSS does not imply SSS.

Example 1.1 (Stationary AR(1) Process): Consider the autoregressive process of order 1: $$ X_t = \phi X_{t-1} + \varepsilon_t, \quad \varepsilon_t \sim \text{WN}(0, \sigma^2), $$ where $\text{WN}(0, \sigma^2)$ denotes white noise with zero mean and variance $\sigma^2$. If $|\phi| < 1$, the process is causal and stationary (both WSS and SSS if $\varepsilon_t$ are i.i.d.). The mean is zero, and the autocovariance is $\gamma(h) = \frac{\sigma^2}{1-\phi^2} \phi^{|h|}$. If $|\phi| \geq 1$, the process is non-stationary (explosive or unit root).

1.2.3 Ergodicity

Stationarity describes invariance over time, but for statistical inference, we need a stronger property: ergodicity. Intuitively, ergodicity means that time averages converge to ensemble (probabilistic) averages. Formally, a stationary process $\{X_t\}$ is ergodic in the mean if for any (measurable) function $g$ of a finite number of variables, $$ \frac{1}{T} \sum_{t=1}^T g(X_t, X_{t-1}, \dots, X_{t-k+1}) \xrightarrow{a.s.} E[g(X_0, X_{-1}, \dots, X_{-k+1})], $$ as $T \to \infty$. Ergodicity in the mean implies that a single, sufficiently long realization contains all the statistical information about the process. It justifies using sample averages (e.g., sample mean, sample autocorrelation) to estimate population parameters.

A sufficient condition for ergodicity in the mean for a WSS process is that the autocovariance function satisfies $$ \sum_{h=-\infty}^{\infty} |\gamma(h)| < \infty. $$ This condition ensures that the spectral density (defined later) is continuous and the process is mixing, a stronger form of asymptotic independence.

1.3 Autocovariance and Autocorrelation Functions

For a WSS process, the autocovariance function $\gamma(h)$ is even: $\gamma(-h) = \gamma(h)$. It is positive semi-definite: for any finite set of time points $t_1, \dots, t_n$ and real numbers $a_1, \dots, a_n$, $$ \sum_{i=1}^n \sum_{j=1}^n a_i a_j \gamma(