Foundations of Digital Speech Processing: Signals, Features, and Evaluation

Speech, as the most natural medium of human communication, stands at the precipice of a technological revolution driven by artificial intelligence. Yet, beneath the sophisticated architectures of modern speech recognizers and synthesizers lies a foundation built upon decades of signal processing research. This chapter serves as that essential foundation, establishing the universal "data language" and "evaluation compass" upon which all subsequent discussions of Hidden Markov Models, Connectionist Temporal Classification, and end-to-end deep learning are built. We will embark on a rigorous journey from the physical vibration of a speaker's vocal cords to the abstract numerical vectors that machine learning models consume, and finally to the metrics that quantify success. This is not merely a review of prerequisite material; it is a deliberate construction of a shared mental model for the entire field. We will define terms with mathematical precision, justify engineering choices with physical intuition, and critically examine the assumptions embedded in every standard pipeline. By the end, you will not only know *how* to compute a Mel-frequency cepstral coefficient (MFCC) or a Word Error Rate (WER), but more importantly, *why* we do so, and what the limitations of these foundational tools portend for the advanced models that follow.

The Physical and Digital Nature of Speech Signals

At its core, speech is a physical phenomenon: a pressure wave generated by the human vocal apparatus propagating through a medium, typically air. To be processed by a computer, this analog continuum must be captured, digitized, and represented as a discrete-time, discrete-amplitude sequence. This transformation is governed by the fundamental theorems of signal processing and introduces inherent trade-offs that every practitioner must understand.

Analog-to-Digital Conversion: Sampling and Quantization

The first critical step is sampling, the process of measuring the continuous acoustic pressure wave at uniform intervals in time. The Nyquist-Shannon sampling theorem provides the bedrock rule: to perfectly reconstruct a bandlimited signal from its samples, the sampling frequency $f_s$ must be strictly greater than twice the highest frequency component $f_{max}$ present in the signal. $$f_s > 2f_{max}$$ For human speech, the perceptually relevant frequency range is generally considered to be 300 Hz to 3400 Hz for telephony, giving a $f_{max}$ of 3400 Hz and a Nyquist rate of 6800 Hz. The standard 8 kHz sampling rate for telephone audio provides a comfortable margin above this minimum. For high-fidelity applications like music or wideband speech (e.g., 50–7000 Hz), rates of 16 kHz, 44.1 kHz (CD quality), or even 48 kHz are common. A critical engineering consequence is the necessity of an anti-aliasing filter—a low-pass analog filter applied *before* sampling—to strictly band-limit the input signal and prevent higher-frequency components from "aliasing" into the passband as false, lower-frequency signals.

The second step is quantization, mapping each sampled amplitude value to a discrete level from a finite set. This is characterized by the bit depth (e.g., 16-bit, 24-bit). Quantization introduces quantization noise, a deterministic rounding error modeled as a uniform random variable. The theoretical Signal-to-Quantization-Noise Ratio (SQNR) for a uniform quantizer with $B$ bits is approximately: $$\text{SQNR} \approx 6.02B + 1.76 \text{ dB}$$ A 16-bit system, standard for PCM audio in WAV files, yields a theoretical maximum SQNR of about 98 dB, which is more than sufficient for capturing the dynamic range of speech in most acoustic environments. However, real-world ADCs (Analog-to-Digital Converters) also introduce thermal noise and distortion, so the effective number of bits (ENOB) is often lower.

> 💡 Electronic Edition Sidebar: The 44.1 kHz CD Mystery > Why does a CD use 44.1 kHz? This seemingly arbitrary number is a historical artifact. Early digital audio experiments used video cassette recorders to store digital data. The NTSC video standard used 30 frames per second, with 245 lines of active video per frame. Storing 16-bit stereo samples required 3 samples per line, leading to: $30 \times 245 \times 3 = 22,050$ samples per second *per channel*. For stereo, this doubled to 44,100 Hz. This standard, born from engineering compromise, became the global benchmark for consumer audio.

Pre-emphasis: Balancing the Spectrum

Human speech exhibits a spectral envelope that roughly follows a 1/f roll-off; that is, energy decreases as frequency increases. This is a natural consequence of the source-filter model of speech production, where the glottal source has a low-pass characteristic and the vocal tract filter introduces formants but does not fully compensate. Consequently, the spectral energy in higher frequencies (above 1 kHz) is often an order of magnitude lower than in the lower frequencies. Many analysis techniques, such as cepstral analysis, are sensitive to such dynamic range disparities.

Pre-emphasis is a first-order high-pass filtering applied to the discrete-time signal $x[n]$ to compensate for this spectral tilt: $$y[n] = x[n] - \alpha x[n-1]$$ where $\alpha$ is the pre-emphasis coefficient, typically in the range [0.95, 0.97]. In the frequency domain, this filter has a response $H(\omega) = 1 - \alpha e^{-j\omega}$, which boosts high frequencies relative to low ones. The effect is to flatten the signal's spectrum, making the subsequent feature extraction (like MFCC) more robust and placing more perceptual emphasis on the higher-frequency consonant information that is crucial for intelligibility. The process is inverted during synthesis with a de-emphasis filter to reconstruct a signal with a natural spectral slope.

Framing and Windowing: The Short-Time Assumption

Speech is a non-stationary signal; its statistical properties (mean, variance) change over time, primarily because the vocal tract configuration changes to produce different phonemes. However, over a sufficiently short interval (typically 20-40 milliseconds), the signal can be considered stationary. This is the short-time assumption, the cornerstone of nearly all classical speech processing.

To apply stationary signal processing techniques (like the Fast Fourier Transform), we segment the continuous signal into overlapping frames: 1. Framing: The pre-emphasized signal is divided into overlapping frames of length $N$ samples. Common frame lengths are 20 ms (320 samples at 16 kHz) or 25 ms (400 samples). A frame shift (or hop size) of 10 ms (160 samples) is standard, providing a good trade-off between time resolution and computational redundancy. 2. Windowing: Each frame is multiplied element-wise by a window function $w[n]$ to minimize the spectral leakage caused by the abrupt frame boundaries. The abrupt start and end of a rectangular window create discontinuities, which in the frequency domain manifest as spreading of energy from a pure sinusoid into adjacent bins (leakage).

The most common window is the Hamming window: $$w[n] = 0.54 - 0.46 \cos\left(\frac{2\pi n}{N-1}\right), \quad 0 \le n \le N-1$$ Other choices include the Hann window and the Blackman window. The Hamming window offers a good compromise: its main lobe is narrow (preserving frequency resolution) while its side lobes are attenuated by about 41 dB (reducing leakage). The choice of window subtly affects the resulting spectral estimate and, consequently, the derived features.

```mermaid graph TD A[Raw Speech Waveform<br/>x(t), continuous] --> B[Sampling & Quantization<br/>fs=16kHz, 16-bit PCM]; B --> C[Pre-emphasis Filter<br/>y[n] = x[n] - αx[n-1]]; C --> D[Framing<br/>25ms frames, 10ms shift]; D --> E[Windowing<br/>Apply Hamming window]; E --> F[Short-Time Stationary Frames<br/>Ready for Spectral Analysis]; ```

This frami