Foundations of Time Series Analysis

Foundations of Time Series Analysis

Introduction to Time Series Analysis

Time series analysis represents one of the most practically significant branches of statistical inference, concerned with the analysis of data collected sequentially over time. Unlike cross-sectional data, where observations are assumed to be independent and identically distributed, time series data exhibits inherent temporal dependencies that must be carefully modeled to extract meaningful insights and generate accurate forecasts. The ubiquity of time series data in modern applications—ranging from financial market prediction and economic forecasting to climate modeling and healthcare monitoring—underscores the importance of developing a rigorous foundation in the theoretical and methodological principles that govern this field.

The fundamental challenge in time series analysis lies in recognizing that the past influences the future, but the reverse is not true. This temporal asymmetry creates both the opportunity for prediction and the complication that traditional statistical methods, which assume independence, cannot be directly applied. A time series $\{Y_t\}_{t \in \mathbb{T}}$ is not merely a collection of numbers indexed by time; it is a realization from an infinite-dimensional stochastic process, and our task is to understand the underlying data-generating mechanism well enough to make probabilistic statements about future values.

This chapter establishes the mathematical and conceptual foundations necessary for the advanced modeling techniques presented in subsequent chapters. We begin with precise definitions of stochastic processes and their key properties, particularly stationarity and ergodicity. We then examine the fundamental components that constitute most time series—trend, seasonality, cyclical patterns, and irregular fluctuations—and discuss decomposition methods that separate these components. The concepts of autocorrelation and partial autocorrelation are developed in detail, as these form the basis for model identification in the classical Box-Jenkins framework. We introduce the lag operator notation that provides elegant mathematical tools for expressing time series models, and we establish the conventions used throughout this book. Finally, we address practical considerations including data preprocessing, missing value handling, and the overall forecasting workflow, with code demonstrations using pandas and statsmodels that bridge theoretical concepts to implementation.

Stochastic Processes and Their Properties

Definition and Characterization

A stochastic process (or random process) is a collection of random variables indexed by time, typically defined on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$. Formally, we define a stochastic process as a family $\{Y_t : t \in \mathbb{T}\}$ where $\mathbb{T}$ is the index set, usually representing time. When $\mathbb{T} = \{1, 2, 3, \ldots\}$, we have a discrete-time process; when $\mathbb{T} = [0, \infty)$ or $\mathbb{R}$, we have a continuous-time process. This book focuses primarily on discrete-time processes, as most practical forecasting applications involve regularly-spaced observations.

Each random variable $Y_t$ takes values in some state space $\mathcal{Y}$, which is typically $\mathbb{R}$ for continuous-valued series or a countable set for discrete-valued series. For any fixed $\omega \in \Omega$, the mapping $t \mapsto Y_t(\omega)$ traces a realization or sample path of the process. Thus, a stochastic process can be viewed in two complementary ways: as a collection of random variables sharing a common probability distribution, or as a collection of infinitely many sample paths, one of which we observe.

The complete probabilistic description of a stochastic process requires specifying the joint distribution of all finite collections of random variables $\{Y_{t_1}, Y_{t_2}, \ldots, Y_{t_n}\}$ for all choices of $n$ and all time points. This is generally intractable for practical applications, motivating the search for parsimonious characterizations that capture essential features through lower-dimensional summaries. The moments of a stochastic process—particularly the mean function, variance function, and autocovariance function—provide such a characterization under appropriate regularity conditions.

The mean function of a stochastic process is defined as:

$$\mu_t = \mathbb{E}[Y_t] = \int_\Omega Y_t(\omega) \, d\mathbb{P}(\omega)$$

The autocovariance function (ACVF) measures the linear relationship between pairs of observations at different time points:

$$\gamma(s, t) = \text{Cov}(Y_s, Y_t) = \mathbb{E}[(Y_s - \mu_s)(Y_t - \mu_t)]$$

From the autocovariance function, we derive the autocorrelation function (ACF), which normalizes the autocovariance by the standard deviations to produce a unit-free measure:

$$\rho(s, t) = \frac{\gamma(s, t)}{\sqrt{\gamma(s, s) \gamma(t, t)}} = \frac{\text{Cov}(Y_s, Y_t)}{\sigma_s \sigma_t}$$

These functions are fundamental to understanding the temporal structure of a process. In particular, if the autocovariance depends only on the time lag between observations and not on their absolute positions, the process exhibits an important property called stationarity.

Weak Stationarity

Weak stationarity (also called second-order stationarity, wide-sense stationarity, or covariance stationarity) is a central concept in time series analysis. A stochastic process $\{Y_t\}$ is weakly stationary if it satisfies three conditions:

1. Constant mean: $\mathbb{E}[Y_t] = \mu$ for all $t \in \mathbb{T}$ 2. Constant variance: $\text{Var}(Y_t) = \mathbb{E}[(Y_t - \mu)^2] = \sigma^2$ for all $t \in \mathbb{T}$ 3. Time-invariant autocovariance: $\gamma(t, t-h) = \gamma(h)$ depends only on the lag $h$, not on $t$

The third condition implies that the autocovariance between $Y_t$ and $Y_{t+h}$ depends only on the separation $h$ between the two time points. For $h = 0$, we have $\gamma(0) = \text{Var}(Y_t) = \sigma^2$. For $h > 0$, $\gamma(h)$ measures the strength of linear dependence at lag $h$.

Weak stationarity is an extremely useful assumption because it allows us to:

• Estimate population moments from a single realization (under ergodicity)
• Develop tractable model classes
• Make predictions based on the assumed temporal structure

It is important to note that weak stationarity does not imply that individual realizations appear "stable" over time in any visual sense. A weakly stationary process can exhibit substantial fluctuations, as long as the statistical properties remain constant. Conversely, a non-stationary process may sometimes appear visually stable but violate the mathematical conditions.

Strong Stationarity

Strong stationarity (or strict stationarity) requires that the entire joint distribution of the process be invariant to shifts in time. Formally, $\{Y_t\}$ is strictly stationary if for any $n$, any set of time points $\{t_1, t_2, \ldots, t_n\}$, and any integer $k$, we have:

$$(Y_{t_1}, Y_{t_2}, \ldots, Y_{t_n}) \overset{d}{=} (Y_{t_1+k}, Y_{t_2+k}, \ldots, Y_{t_n+k})$$

That is, shifting all time indices by a constant amount does not change the joint distribution. Strong stationarity implies weak stationarity provided the first two moments exist, but the converse is generally false. Strong stationarity is a stronger condition because it requires agreement of all moments and even non-moments (skewness, kurtosis, etc.).

In practice, strong stationarity is difficult to verify and rarely assumed in its full generality. Most time series models assume weak stationarity and further specify parametric forms for the moments. However, there are cases where weak stationarity holds but strong stationarity does not. For example, a process with $Y_t = \sigma_t Z$, where $Z \sim N(0, 1)$ and $\sigma_t$ is deterministic but varying over time, has constant variance but non-normal distributions th