Fundamentals of Time Series Analysis

Time series data—sequences of observations recorded sequentially over time—permeate nearly every domain of human inquiry, from economics and finance to climatology, engineering, and healthcare. Unlike cross-sectional data, time series exhibit inherent temporal dependence: the value at time $t$ is often influenced by past values, creating a structure that must be explicitly modeled to enable accurate forecasting and inference. This chapter establishes the rigorous mathematical and conceptual foundation upon which the entire edifice of time series analysis is built. We begin by formalizing what constitutes a time series, introducing the essential notation that will permeate this book. Central to all subsequent discussion is the concept of stationarity—a property that, while often violated in practice, provides the theoretical bedrock for many classical and modern forecasting methods. We will delineate between strict stationarity and weak (second-order) stationarity, examine their implications, and introduce formal statistical tests to assess them. Next, we turn to autocorrelation, the quantitative measure of linear dependence between observations at different time lags. The autocorrelation function (ACF) and partial autocorrelation function (PACF) are not merely diagnostic tools; they are the fingerprints of underlying data-generating processes, guiding model identification in both classical Box-Jenkins methodology and modern machine learning approaches. The pervasive presence of seasonality—regular, predictable patterns recurring at fixed intervals—requires specialized techniques for detection, modeling, and removal. We will explore both additive and multiplicative seasonal models, and introduce decomposition methods that separate a series into trend, seasonal, and residual components. Before any sophisticated modeling can commence, rigorous preprocessing is paramount. Real-world time series are rarely pristine; they suffer from missing observations, anomalous spikes, and structural breaks. We will survey principled methods for imputation, outlier detection, and transformation, emphasizing that poor preprocessing inevitably leads to poor forecasts. Finally, we touch upon foundational theorems, most notably the Wold decomposition theorem, which guarantees that any covariance-stationary process can be represented as a linear combination of its past innovations—a result that justifies the focus on linear models throughout much of classical time series analysis. Throughout, we interleave theory with concrete examples using real-world datasets: monthly airline passenger counts revealing strong seasonal patterns, daily temperature recordings exhibiting trends and autocorrelation, and financial return series demonstrating volatility clustering and near-zero autocorrelation. Practical code snippets in Python, utilizing libraries such as `pandas`, `statsmodels`, and `scikit-learn`, will demonstrate how these concepts are applied in practice, bridging the gap between abstract theory and actionable insight. By the end of this chapter, the reader will possess a precise, operational understanding of the core vocabulary and mechanics of time series, equipped to engage with the more advanced unified approaches that follow.

1.1 Formal Definition and Notation

A time series is formally defined as a sequence of random variables $\{Y_t\}$ indexed by time $t$, where $t$ is typically an integer representing discrete time points (e.g., days, months, quarters). The observed data is a single realization (or sample path) of this stochastic process: $\{y_1, y_2, \dots, y_T\}$, where $T$ denotes the sample size. This fundamental distinction—between the underlying probabilistic process and its single observed realization—is critical. All statistical inference about the process must be drawn from this one, potentially short, realization.

We denote the entire observed series as $\mathbf{y} = (y_1, y_2, \dots, y_T)^\top$. For analysis, we often work with vectors of lagged values. The lag operator $L$ is defined such that $L y_t = y_{t-1}$, and $L^k y_t = y_{t-k}$ for any integer $k \geq 0$. This operator provides a concise notation for expressing relationships. For example, a first difference is $\nabla y_t = y_t - y_{t-1} = (1 - L)y_t$, and a seasonal difference (with seasonal period $s$) is $\nabla_s y_t = y_t - y_{t-s} = (1 - L^s)y_t$.

The mean function of the process is $\mu_t = \mathbb{E}[Y_t]$, and the autocovariance function is $\gamma(k) = \text{Cov}(Y_t, Y_{t-k}) = \mathbb{E}[(Y_t - \mu_t)(Y_{t-k} - \mu_{t-k})]$, assuming it exists. The autocorrelation function (ACF) is $\rho(k) = \gamma(k) / \gamma(0)$, which standardizes the autocovariance to lie between $-1$ and $1$. These functions describe the second-order structure of the series.

In practice, we estimate these population quantities from the sample. The sample mean is $\bar{y} = \frac{1}{T}\sum_{t=1}^T y_t$. The sample autocovariance at lag $k$ is typically computed as: $$ \hat{\gamma}(k) = \frac{1}{T} \sum_{t=k+1}^T (y_t - \bar{y})(y_{t-k} - \bar{y}) $$ for $k = 0, 1, \dots, T-1$, and $\hat{\gamma}(k) = \hat{\gamma}(-k)$ by symmetry. Some estimators use $T-k$ in the denominator for unbiasedness under stronger assumptions, but the $1/T$ version is consistent and more common in modern software. The sample ACF is then $\hat{\rho}(k) = \hat{\gamma}(k) / \hat{\gamma}(0)$.

It is crucial to note that these are estimates of the theoretical ACF of the underlying process. With finite data, sampling variability means $\hat{\rho}(k)$ will not be exactly zero even if the true $\rho(k)=0$. Approximate confidence intervals, often $\pm 1.96/\sqrt{T}$ under white noise assumptions, help judge significance.

```python import pandas as pd import numpy as np from statsmodels.tsa.stattools import acf

Generate a sample time series: AR(1) with phi=0.6

np.random.seed(42) n = 500 y = np.zeros(n) y[0] = np.random.normal() for t in range(1, n): y[t] = 0.6 * y[t-1] + np.random.normal()

Compute sample ACF

lag_acf = acf(y, nlags=20, fft=False) print("Sample ACF at lags 0-5:", lag_acf[:6]) ```

This code generates an autoregressive process of order 1 (AR(1)) and computes its sample ACF, which should decay exponentially toward zero, reflecting the positive persistence introduced by $\phi=0.6$.

1.2 Stationarity: The Cornerstone of Time Series Analysis

Stationarity is a fundamental property that simplifies modeling and inference. A stochastic process $\{Y_t\}$ is said to be strictly stationary if the joint distribution of $(Y_{t_1}, Y_{t_2}, \dots, Y_{t_k})$ is identical to that of $(Y_{t_1+h}, Y_{t_2+h}, \dots, Y_{t_k+h})$ for all integers $h$ and all $k$. This implies that all statistical properties—mean, variance, and higher moments—are invariant to time shifts. Strict stationarity is a strong condition, rarely verifiable from a single finite realization.

In practice, we almost always work with weak stationarity (or covariance stationarity), which requires only: 1. A finite, constant mean: $\mathbb{E}[Y_t] = \mu < \infty$ for all $t$. 2. A finite, constant variance: $\text{Var}(Y_t) = \gamma(0) < \infty$ for all $t$. 3. An autocovariance function that depends only on the lag $k$: $\text{Cov}(Y_t, Y_{t-k}) = \gamma(k)$, not on $t$.

Weak stationarity is sufficient for many linear models (ARMA) and ensures that the ACF is well-defined and time-invariant. Note that strict stationarity implies weak stationarity if the second moments exist, but the converse is not true.

The importance of stationarity cannot be overstated. For a stationary series, past behavior is a reliable guide to future behavior. Many forecasting methods assume, either explicitly or implicitly, that the series is stationary or can be made stationary via transformations. Non-stationary series—those with trends, changing variance, or evolving seasonality—lead to spurious results if modeled a