Foundations of Speech Signal Processing and Acoustic Feature Extraction

Speech, as the primary conduit of human interaction, represents an intricately structured acoustic signal generated by the physiological mechanisms of the vocal tract and shaped by linguistic intent. Its computational analysis requires a rigorous transformation from a continuous physical pressure wave into a discrete, information-rich representation suitable for statistical and, subsequently, neural modeling. This chapter serves as the indispensable foundation for the entire discourse, establishing the physical principles of speech production, the mathematical tools of digital signal processing, and the canonical feature extraction pipelines that have defined the field for decades. We will systematically build the bridge from the analog world of sound to the digital vectors that feed into Hidden Markov Models (HMMs) and, eventually, end-to-end deep neural networks (DNNs). The journey begins with the source-filter model of speech production, proceeds through the analog-to-digital conversion and essential preprocessing stages—pre-emphasis, windowing, and frame blocking—and culminates in a detailed exposition of the short-time Fourier transform (STFT). The core of the chapter is a meticulous derivation and computational walkthrough of Mel-Frequency Cepstral Coefficients (MFCCs) and filter bank (FBank) energies, including their mathematical justifications and connections to human auditory perception. We will then explore advanced normalization techniques like Cepstral Mean and Variance Normalization (CMVN), voice activity detection (VAD) methods, and the incorporation of temporal dynamics via delta and delta-delta coefficients. The chapter concludes by formally positioning these hand-crafted features within the statistical modeling frameworks—particularly the Gaussian Mixture Model-HMM (GMM-HMM) paradigm—that dominated speech recognition for thirty years, thereby setting the stage for the transition to deep learning explored in subsequent chapters.

1. The Physics of Speech Production: The Source-Filter Model

The human speech production system can be elegantly modeled as a time-varying linear filter excited by a sound source. This source-filter theory, pioneered by Gunnar Fant and formalized in the acoustic theory of speech production, provides the fundamental physical basis for almost all modern feature extraction techniques. The model separates the vocal tract into two primary components: the glottal source and the vocal tract filter.

The glottal source generates a periodic airflow pulse train for voiced sounds (e.g., vowels, nasals) through the quasi-periodic opening and closing of the vocal folds. The fundamental frequency ($F_0$) of this pulse train, typically ranging from 80 Hz for adult males to 300 Hz for adult females and children, determines the perceived pitch. The glottal flow waveform itself has a characteristic shape, often modeled by the LF model (Liljencrants-Fant model) or simpler derivatives like the Rosenberg model. For unvoiced sounds (e.g., /s/, /f/, /t/), the glottis is held open, and the sound source becomes a noise-like aperiodic flow generated by turbulent airflow at a constriction in the vocal tract (e.g., at the teeth for /s/ or the lips for /f/).

This source signal then passes through the vocal tract, a complex, time-varying acoustic tube with a cross-sectional area function $A(x,t)$ that changes with the positions of the tongue, jaw, lips, and velum. Under the lossless tube model assumption, the vocal tract can be approximated as a series of concatenated cylindrical sections of varying diameters. The acoustic properties of this tube are governed by the one-dimensional wave equation. The transfer function of this filter, for a given configuration, is determined by its resonant frequencies, known as formants ($F_1, F_2, F_3, ...$). These formants are the peaks in the frequency response of the vocal tract and are the primary acoustic correlates of vowel identity. For instance, the vowel /i/ (as in "heed") has a low $F_1$ and high $F_2$, while /ɑ/ (as in "hod") has a high $F_1$ and low $F_2$.

The acoustic output at the lips, $s(t)$, is the convolution of the glottal source excitation, $e(t)$, with the impulse response of the vocal tract filter, $h(t)$: $s(t) = e(t) * h(t)$. In the frequency domain, this becomes $S(\omega) = E(\omega) \cdot H(\omega)$. This convolution model is critical: it implies that the spectral envelope of the speech signal (shaped by $H(\omega)$) carries information about the vocal tract configuration (i.e., the phoneme), while the fine spectral structure (the harmonics of $F_0$ for voiced sounds) carries information about the source (i.e., pitch and voicing). The central challenge of feature extraction is to separate these two pieces of information, extracting a representation that is primarily invariant to the speaker-dependent source characteristics ($F_0$, glottal flow shape) and sensitive only to the filter-dependent articulatory features (phoneme identity). This is the primary motivation for the cepstral analysis that leads to MFCCs.

2. Analog-to-Digital Conversion: Sampling and Quantization

Before digital processing can commence, the continuous acoustic pressure wave must be converted into a discrete-time sequence via analog-to-digital (A/D) conversion. This process involves two key steps: sampling and quantization.

Sampling is governed by the Nyquist-Shannon sampling theorem, which states that a bandlimited signal with no frequency components above $B$ Hz can be perfectly reconstructed from its samples if the sampling rate $f_s$ is greater than $2B$. Human speech, for telephony, is traditionally bandlimited to 300–3400 Hz, leading to the standard $f_s = 8$ kHz sampling rate (Nyquist rate = 4 kHz). For high-fidelity speech and audio processing, wider bandwidths are used; $f_s = 16$ kHz covers the range up to 8 kHz, which includes most fricative and plosive energy, while $f_s = 44.1$ kHz (CD quality) or $48$ kHz is common for general audio. In practice, an anti-aliasing filter (a low-pass filter with a sharp cutoff just below $f_s/2$) is applied before sampling to prevent aliasing of higher-frequency components.

Quantization maps each continuous-amplitude sample to a discrete digital value from a finite set, typically using uniform linear quantization with $B$ bits per sample, yielding $2^B$ possible levels. The signal-to-quantization-noise ratio (SQNR) improves by approximately 6 dB per additional bit. For speech, 16-bit linear quantization (yielding a 96 dB theoretical SQNR) is standard in research and applications like VoIP (e.g., G.711 µ-law or A-law companding uses 8-bit nonlinear quantization to optimize for the dynamic range of speech). The resulting digital signal is a sequence $x[n]$, where $n$ is the discrete sample index.

3. Fundamental Preprocessing: Pre-emphasis, Windowing, and Frame Blocking

The raw sampled speech signal $x[n]$ is not directly suitable for spectral analysis due to two inherent characteristics: (1) a typical $-20$ dB/decade spectral roll-off due to the glottal source spectrum (the $1/f$ characteristic), and (2) the non-stationary nature of speech, where its statistical properties change rapidly over time (on the order of 10–30 ms). Therefore, three canonical preprocessing steps are applied.

3.1 Pre-emphasis

To compensate for the natural high-frequency attenuation and to balance the spectrum, a pre-emphasis filter is applied. This is a first-order high-pass filter with transfer function $H(z) = 1 - \alpha z^{-1}$, where $\alpha$ is the pre-emphasis coefficient, typically chosen in the range $0.95 \leq \alpha \leq 0.97$. In the time domain, this implements the difference equation: $y[n] = x[n] - \alpha x[n-1]$. This operation boosts the high-frequency energy, making the overall spectrum flatter and more suitable for subsequent spectral analysis (e.g., LPC analysis). The inve