Foundations of Time Series Analysis

Foundations of Time Series Analysis

Introduction: The Nature of Temporal Data

Time series analysis represents one of the most consequential branches of statistical methodology, concerned with the collection, analysis, and interpretation of data points indexed chronologically. Unlike cross-sectional data, where observations are treated as independent draws from some population distribution, time series data embodies an intrinsic ordering that carries substantial information. The sequence in which observations arrive—their temporal structure—often encodes the very phenomena we seek to understand, predict, or control.

The importance of time series analysis extends across virtually every empirical discipline. In economics, we analyze GDP growth trajectories, stock price movements, and inflation dynamics. In finance, risk management depends critically on modeling volatility clustering and asset return dependencies. Epidemiology relies on surveillance data to track disease outbreaks and evaluate intervention effectiveness. Climate science depends on temperature records, precipitation patterns, and atmospheric CO₂ concentrations to understand planetary-scale phenomena. Engineering applications span control systems, signal processing, and network traffic analysis. The ubiquity of time series data in modern data science ensures that mastery of temporal analysis techniques remains an essential competency for any serious practitioner.

This opening chapter establishes the conceptual and mathematical foundations upon which the sophisticated methods developed throughout this book will build. We begin with the fundamental question of how to conceptualize time series data itself—understanding observations as realizations of underlying stochastic processes. From this perspective, we develop the critical concepts of stationarity and ergodicity, which provide the theoretical justification for drawing inferences from limited temporal data. We then explore the tools for characterizing temporal dependencies, including the autocorrelation and partial autocorrelation functions. The chapter proceeds to practical matters: statistical tests for stationarity, handling missing data and multiple seasonal patterns, and selecting appropriate temporal resolutions. We conclude with an overview of the forecasting pipeline and the essential concept of forecast horizons, preparing the ground for the systematic development of forecasting methodologies in subsequent chapters.

Time Series as Stochastic Processes

Conceptual Foundations

A stochastic process (also called a random process) is a family of random variables indexed by time. Formally, let $(\Omega, \mathcal{F}, \mathbb{P})$ denote a probability space, where $\Omega$ is the sample space of possible outcomes, $\mathcal{F}$ is the sigma-algebra of events, and $\mathbb{P}$ is the probability measure. A stochastic process is a collection $\{X_t : t \in \mathcal{T}\}$ of random variables defined on this probability space, where $\mathcal{T}$ is the index set representing time.

The index set $\mathcal{T}$ can be discrete (typically $\mathcal{T} = \{\ldots, -2, -1, 0, 1, 2, \ldots\}$ or $\mathcal{T} = \{1, 2, 3, \ldots\}$) or continuous (typically $\mathcal{T} = \mathbb{R}$ or $\mathcal{T} = [0, \infty)$). Most applications in forecasting work with discrete-time processes, where observations are recorded at regular intervals—daily, weekly, monthly, quarterly, or annually. Continuous-time processes arise more naturally in physics and engineering but can be approximated by discrete-time models for practical purposes.

When we observe a time series, we are not observing the entire stochastic process but rather a single realization (or sample path) of that process. If we denote a particular realization as $x_1, x_2, \ldots, x_T$, these are specific numbers—the outcomes of the random variables $X_1, X_2, \ldots, X_T$. This distinction between the underlying process (a family of random variables with associated probability distributions) and a particular realization (a sequence of observed values) is fundamental to understanding time series methodology.

Consider the following Python illustration of this concept:

```python import numpy as np import matplotlib.pyplot as plt

np.random.seed(42)

Define an AR(1) process: X_t = 0.7 * X_{t-1} + ε_t

def generate_ar1(n, phi=0.7, sigma=1.0): """Generate a single realization of AR(1) process.""" x = np.zeros(n) x[0] = np.random.normal(0, sigma) for t in range(1, n): x[t] = phi * x[t-1] + np.random.normal(0, sigma) return x

Generate multiple realizations (sample paths)

realizations = [generate_ar1(200) for _ in range(5)]

Plot all realizations

fig, axes = plt.subplots(3, 1, figsize=(12, 8))

Plot individual realizations

for i, ax in enumerate(axes[:2]): ax.plot(realizations[i], alpha=0.8) ax.set_title(f'Realization {i+1} of AR(1) Process ($X_t = 0.7X_{{t-1}} + \\epsilon_t$)') ax.set_xlabel('Time') ax.set_ylabel('Value') ax.grid(True, alpha=0.3)

Plot ensemble mean at each time point

ensemble_mean = np.mean(realizations, axis=0) axes[2].plot(ensemble_mean, 'k-', linewidth=2, label='Ensemble Mean') axes[2].set_title('Ensemble Mean Across 5 Realizations') axes[2].set_xlabel('Time') axes[2].set_ylabel('Mean Value') axes[2].legend() axes[2].grid(True, alpha=0.3)

plt.tight_layout() plt.savefig('stochastic_process_realizations.png', dpi=150) plt.close() ```

This code demonstrates how a single stochastic process generates multiple possible realizations, each with different specific values but sharing the same probabilistic structure. The ensemble mean—the average across realizations at each time point—reveals properties of the underlying process that individual realizations alone cannot show.

Common Stochastic Process Classes

Understanding time series requires familiarity with several fundamental classes of stochastic processes that serve as building blocks for more complex models.

The random walk process is perhaps the simplest non-stationary process and appears extensively in financial applications. Defined as:

$$X_t = X_{t-1} + \epsilon_t$$

where $\epsilon_t$ are independent innovations with mean zero and variance $\sigma^2$, the random walk accumulates random shocks over time. Starting from $X_0 = 0$, we have $X_t = \sum_{i=1}^{t} \epsilon_i$. The variance of $X_t$ grows linearly with time: $\text{Var}(X_t) = t\sigma^2$, which means the process has no finite variance in the limit as $t \to \infty$. This unbounded variance growth is a hallmark of non-stationarity.

The moving average (MA) process of order $q$, denoted MA($q$), expresses the current value as a linear combination of current and past white noise innovations:

$$X_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + \cdots + \theta_q \epsilon_{t-q}$$

where $\epsilon_t \sim WN(0, \sigma^2)$ denotes white noise—a sequence of uncorrelated random variables with zero mean and constant variance. The MA process has finite memory, as past shocks affect $X_t$ only through the $q$ most recent innovations.

The autoregressive (AR) process of order $p$, denoted AR($p$), models the current value as a linear combination of its own past values plus a current innovation:

$$X_t = \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \epsilon_t$$

The AR process generalizes the random walk by allowing the current value to depend on multiple lagged values with different weights. The AR(1) process with coefficient $\phi$ reduces to a random walk when $\phi = 1$ and to white noise when $\phi = 0$.

The autoregressive moving average (ARMA) process combines both autoregressive and moving average components:

$$X_t = \phi_1 X_{t-1} + \cdots + \phi_p X_{t-p} + \epsilon_t + \theta_1 \epsilon_{t-1} + \cdots + \theta_q \epsilon_{t-q}$$

More advanced classes include the ARIMA (integrated ARMA) process for non-stationary data, **S