Foundations of Time Series Analysis

Foundations of Time Series Analysis

The systematic study of temporally ordered data constitutes one of the most consequential intellectual enterprises in modern statistics and applied mathematics. Time series analysis, as a discipline, addresses the fundamental challenge of extracting meaningful patterns, relationships, and predictive structure from observations collected sequentially over time. Unlike independent and identically distributed (i.i.d.) samples that populate classical statistical theory, temporal data exhibit intrinsic dependencies that violate standard assumptions and demand specialized methodological frameworks. The recognition that "time matters"—that the sequential ordering of observations carries informational content that cannot be ignored—represents a conceptual watershed in statistical thinking, one that has profound implications for how we model uncertainty, formulate predictions, and understand dynamic phenomena across virtually every domain of scientific inquiry.

The relevance of time series analysis has never been more pronounced than in the contemporary era. Financial markets generate continuous streams of price, volume, and volatility data that drive multi-trillion dollar decisions daily. Environmental scientists monitor atmospheric carbon concentrations, sea levels, and temperature anomalies to understand and predict climate change. Healthcare systems collect vital signs, electronic health records, and epidemiological surveillance data that inform public health interventions. Industrial processes produce sensor readings enabling predictive maintenance and quality control. The digital economy generates user behavior logs, clickstreams, and engagement metrics that power recommendation systems and inform business strategy. In each of these domains and countless others, the ability to analyze, model, and forecast temporal phenomena provides a decisive competitive advantage and, frequently, a matter of substantial societal consequence.

This opening chapter establishes the conceptual and mathematical foundations upon which the entire subsequent treatment rests. We begin by developing the stochastic process perspective that provides the fundamental framework for understanding time series data. This perspective situates any particular observed sequence within a broader ensemble of possible realizations that might have emerged under different conditions, thereby connecting the finite sample of actual observations to the infinite-dimensional probabilistic structures that generate them. We examine the distinction between deterministic and stochastic components of temporal phenomena, recognizing that even ostensibly deterministic systems often require stochastic treatment due to measurement error, unobserved influences, or intrinsic chaos. The concepts of dependence, stationarity, and ergodicity receive thorough development, as these properties fundamentally determine what statistical inferences are possible and what methods are appropriate. We then explore the three principal representational domains—time domain, frequency domain, and state-space domain—that provide complementary perspectives on temporal structure. The autocorrelation function and partial autocorrelation function receive detailed treatment, including both their mathematical definitions and practical interpretation guidelines. Spectral density analysis introduces the frequency-domain viewpoint, revealing cyclical patterns and periodicities that may remain obscure in the time domain. State-space representation provides a unifying framework that connects many classical and modern approaches. We conclude with a historical overview tracing the evolution of time series forecasting from early exponential smoothing methods through the ARIMA paradigm to the contemporary integration of statistical and machine learning approaches, situating the detailed treatments that follow within this broader intellectual trajectory.

The Stochastic Process Perspective

From Observations to Random Processes

A foundational shift in perspective occurs when we move from viewing a time series as merely a sequence of numbers to understanding it as a single realization from a stochastic process. Consider a sequence of observations $\{x_1, x_2, \ldots, x_T\}$ collected at times $t = 1, 2, \ldots, T$. Classical statistical inference typically assumes that these observations are independent draws from some common probability distribution. In time series analysis, this assumption is explicitly abandoned. Instead, we conceptualize these observations as the realized values of random variables $X_1, X_2, \ldots, X_T$, where the joint distribution of these random variables encodes the temporal dependencies that connect observations at different times.

Formally, a stochastic process is a collection of random variables $\{X_t : t \in \mathcal{T}\}$ defined on a common probability space and indexed by a parameter set $\mathcal{T}$. For discrete-time processes, which constitute the primary focus of this text, $\mathcal{T}$ is typically the set of integers $\mathbb{Z}$ or a contiguous subset such as $\{1, 2, \ldots, T\}$. The observed sequence represents a single sample path or realization from this process. This distinction between the process (the generating mechanism) and the realization (the observed data) is absolutely fundamental. All of statistical inference in time series analysis concerns using the observed realization to make statements about the underlying process that generated it.

This perspective immediately raises profound questions about statistical inference under dependence. When observations are independent, the law of large numbers and central limit theorem provide the theoretical foundation for asymptotic inference. Under dependence, these classical results require modification or replacement. The ergodic theorem, martingale convergence theorems, and mixingale theory provide alternative foundations, but the conditions required and the rates of convergence obtained differ substantially from the i.i.d. case. Understanding these theoretical foundations is essential for proper application of time series methods and for appreciating the limitations of various inferential procedures.

The Joint Distribution and Its Implications

The complete probabilistic description of a stochastic process is given by its finite-dimensional distributions: for any collection of times $t_1 < t_2 < \cdots < t_k$, the joint distribution of $(X_{t_1}, X_{t_2}, \ldots, X_{t_k})$. By the Kolmogorov extension theorem, specifying these finite-dimensional distributions for all finite collections of times uniquely determines the stochastic process (assuming certain measurability conditions are satisfied). In practice, however, working with the full joint distribution is generally infeasible. Instead, we focus on lower-order moments and specific features of the distribution that capture the aspects of temporal structure most relevant for forecasting and inference.

The first-order properties of a stochastic process are described by the mean function:

$$\mu_t = E[X_t]$$

which gives the expected value of the process at each time $t$. The mean function may be constant, trend through time, or exhibit more complex patterns. The second-order properties are captured by the covariance function:

$$\gamma(t, s) = \operatorname{Cov}(X_t, X_s) = E[(X_t - \mu_t)(X_s - \mu_s)]$$

which measures the linear relationship between observations at times $t$ and $s$. The covariance function is symmetric: $\gamma(t, s) = \gamma(s, t)$. When $t = s$, the covariance reduces to the variance: $\gamma(t, t) = \operatorname{Var}(X_t)$. These second-order moments—the mean function and covariance function—play a central role in time series analysis because they determine the optimal linear predictor and because Gaussian processes are completely characterized by their second-order properties.

Higher-order moments and the full d