The Speech Signal: Physics, Perception, and Digital Representation

Speech, as the primary medium of human communication, exhibits a rich structure that bridges physics, physiology, and cognition. Understanding the fundamental properties of the speech signal is not merely an academic exercise but a prerequisite for developing effective Speech AI systems. This chapter lays the groundwork by examining the physical mechanisms of speech production, the resulting acoustic characteristics, and the digital signal processing techniques that transform analog sound waves into discrete representations suitable for machine learning. We will also explore how human auditory perception has shaped the design of hand-crafted features that have dominated speech processing for decades, setting the stage for the statistical and end-to-end models discussed in later chapters.

The Source-Filter Model of Speech Production

The human vocal system can be elegantly described by the source-filter model, a theoretical framework that separates speech generation into two distinct components: a sound source and a acoustic filter. This model, pioneered by Gunnar Fant in the 1960s, provides a powerful abstraction for understanding how the physical structures of the vocal tract shape the acoustic signal.

Anatomical Subsystems

Speech production involves the coordinated operation of three primary subsystems:

1. Respiratory system: The lungs provide the airflow and air pressure necessary for phonation. The diaphragm and intercostal muscles control the pressure, which typically ranges from 5 to 30 cm H₂O for normal speech. 2. Phonatory system: The vocal folds (or vocal cords) in the larynx vibrate to produce a quasi-periodic sound source when air passes through them. The fundamental frequency (F₀) of this vibration, perceived as pitch, varies between 80–120 Hz for adult males and 180–240 Hz for adult females. 3. Articulatory system: The vocal tract—comprising the pharynx, oral cavity, and nasal cavity—acts as a filter that shapes the spectrum of the sound source. The configuration of the tongue, lips, jaw, and soft palate determines the resonant frequencies (formants) that characterize different speech sounds.

```mermaid graph TD A[ lungs: airflow & pressure ] --> B[ larynx: vocal fold vibration ] B --> C[ sound source: glottal pulse ] C --> D[ vocal tract: acoustic filter ] D --> E[ articulation: consonants/vowels ] E --> F[ radiation at lips: speech signal ] ```

Mathematical Formulation

In the source-filter model, the speech signal \( s(t) \) is modeled as the output of a linear, time-varying filter excited by an input source \( e(t) \):

$$ s(t) = e(t) * h(t) $$

where \( * \) denotes convolution, \( e(t) \) is the excitation signal, and \( h(t) \) is the impulse response of the vocal tract filter. For voiced sounds (e.g., vowels), \( e(t) \) is a periodic train of glottal pulses; for unvoiced sounds (e.g., /s/, /f/), \( e(t) \) is a noise-like signal generated by turbulent airflow at a constriction.

The vocal tract filter is typically modeled as an all-pole system with transfer function:

$$ H(z) = \frac{1}{1 - \sum_{k=1}^{p} a_k z^{-k}} $$

where \( p \) is the filter order (usually 8–16) and \( a_k \) are the linear predictive coding (LPC) coefficients. This all-pole approximation effectively captures the resonant characteristics of the vocal tract.

Acoustic Consequences

The filtering operation imposes a characteristic spectral envelope on the source signal. For voiced sounds, the spectrum consists of a series of peaks—formants—corresponding to the resonant frequencies of the vocal tract. The first formant (F₁) is primarily determined by tongue height, the second formant (F₂) by tongue advancement, and the third formant (F₃) by lip rounding. These relationships form the basis of the vowel quadrilateral in articulatory phonetics.

Acoustic Properties of Speech

Speech signals exhibit distinctive temporal and spectral properties that arise from the physics of production and the constraints of human articulation.

Temporal Characteristics

• **Duration**: Phoneme durations vary widely; vowels in stressed syllables may last 150–300 ms, while consonants can be as short as 50 ms. Prosodic features like stress and intentation modulate timing over longer timescales.
• **Voicing**: The presence or absence of vocal fold vibration divides speech sounds into voiced and unvoiced categories. The voice onset time (VOT) is a crucial cue for distinguishing stop consonants (e.g., /b/ vs /p/).
• **Stationarity**: Speech is approximately stationary over short intervals (20–40 ms), justifying the short-time analysis techniques described later. Over longer intervals, it is highly non-stationary due to articulatory changes.

Spectral Characteristics

• **Spectral envelope**: The overall shape of the magnitude spectrum, determined by the vocal tract filter, contains information about phonetic identity. It is typically characterized by a spectral tilt (more energy at low frequencies) and formant peaks.
• **Fine structure**: For voiced sounds, the spectrum exhibits harmonic structure at multiples of F₀. The relative amplitudes of these harmonics are shaped by the spectral envelope and the glottal pulse shape.
• **Formants**: The first three formants (F₁, F₂, F₃) are most important for vowel discrimination. Typical ranges: F₁: 300–900 Hz, F₂: 900–2500 Hz, F₃: 2500–3500 Hz. Formant transitions are critical cues for consonants.

Digital Representation of Speech

To process speech with digital computers, the continuous acoustic waveform must be converted into a discrete sequence of numbers via sampling and quantization. The resulting digital signal is then analyzed using time-frequency transforms to reveal its frequency content.

Sampling and Quantization

The analog speech signal \( x_a(t) \) is continuous in both time and amplitude. According to the Nyquist-Shannon sampling theorem, to avoid aliasing, the sampling frequency \( f_s \) must satisfy:

$$ f_s \ge 2B $$

where \( B \) is the bandwidth of the signal. For telephony, \( B \) is typically 4 kHz, hence \( f_s = 8 \) kHz. For wideband speech, \( B = 8 \) kHz yields \( f_s = 16 \) kHz. Modern speech processing often uses \( f_s = 16 \) kHz or 44.1 kHz to capture higher frequencies.

Quantization maps each sampled amplitude to a discrete level. With \( b \) bits, the quantization step size is \( \Delta = \frac{V_{max} - V_{min}}{2^b} \). The resulting quantization noise power is \( \frac{\Delta^2}{12} \), leading to a theoretical signal-to-noise ratio (SNR) of:

$$ \text{SNR}_{\text{dB}} \approx 6.02b + 1.76 \text{ dB} $$

In practice, 16-bit quantization (used in CD-quality audio) yields an SNR > 90 dB, which is more than adequate for speech.

The Short-Time Fourier Transform (STFT)

Because speech is non-stationary, a global Fourier transform is insufficient to capture its time-varying spectrum. The STFT addresses this by applying the Fourier transform to short, overlapping windows of the signal.

Given a window function \( w(t) \) (typically 20–40 ms long), the STFT is defined as:

$$ X(t, f) = \int_{-\infty}^{\infty} x(\tau) w(\tau - t) e^{-j2\pi f \tau} d\tau $$

In discrete form, for a signal \( x[n] \) and window \( w[n] \) of length \( N \), the STFT at frame \( m \) and frequency bin \( k \) is:

$$ X[m, k] = \sum_{n=0}^{N-1} x[n + mR] w[n] e^{-j2\pi kn/N} $$

where \( R \) is the hop size (typically 10 ms). Common window functions include Hamming and Hann windows, which reduce spectral leakage:

$$ w_{\text{ham}}[n] = 0.54 - 0.46 \cos\left(\frac{2\pi n}{N-1}\right), \quad n = 0, \dots, N-1 $$

The STFT yields a time-frequency representation where each column corresponds to a magnitude spectrum (or power spectrum) at a given time frame. The magnitude spectrum \( |X[m,k]| \) is often used for feature extraction, while the phase spectrum is typically discarded (though recent work has shown its utility for reconstruction).

```mermaid gr