Foundations of Speech Processing

Foundations of Speech Processing

1.1 The Nature of Spoken Language

Human speech represents one of the most remarkable achievements of biological information processing. Every second, a speaker transforms abstract intentions—semantic content, emotional states, pragmatic goals—into precise neuromuscular commands that generate acoustic waveforms capable of traveling through the air, arriving at a listener's ears, and being decoded back into meaning. This remarkable feat occurs effortlessly for healthy individuals who have acquired language, typically by age five or six, long before any formal education begins. Understanding how this process works, both in its biological instantiation and its mathematical abstraction, forms the foundation upon which the entire field of speech processing is built.

Spoken language differs fundamentally from written language in several critical dimensions. First, speech is temporal—it unfolds sequentially over time, with no spatial layout allowing simultaneous examination of the entire message. Second, speech is continuous—rather than a discrete sequence of well-separated symbols, natural speech flows as a continuous stream with boundaries between words, phrases, and sentences often signaled by subtle prosodic cues rather than explicit markers. Third, speech is redundant—acoustic signals contain multiple overlapping cues to linguistic structure, with information distributed across frequency bands and time scales. Fourth, speech is highly variable—the same utterance spoken by the same speaker on different occasions produces different acoustic realizations, while the same message expressed by different speakers creates acoustic signals that may differ more dramatically than those representing different messages.

These characteristics present both challenges and opportunities for engineers and researchers attempting to build automatic systems for processing speech. The temporal nature of speech makes it amenable to signal processing techniques developed for other time-series domains. The continuity and redundancy provide multiple paths to recovering linguistic content, offering robustness against noise and processing errors. The variability, however, demands sophisticated statistical modeling to capture the regularities underlying the surface diversity.

1.2 Anatomy of Speech Production

Understanding how humans produce speech requires examining the anatomy of the human vocal apparatus, often called the speech chain. This chain consists of three major subsystems: the respiratory system, the phonatory system, and the articulatory system.

The respiratory system provides the aerodynamic energy that drives speech production. During normal breathing at rest, inhalation and exhalation are roughly balanced, each lasting approximately two seconds. For speech production, however, the pattern changes dramatically. Inhalation becomes brief and rapid, while exhalation, during which speech occurs, extends considerably. The diaphragm and external intercostal muscles contract to draw air into the lungs, which can hold approximately five to six liters of air in an adult male. The pressure generated in the lungs, called subglottal pressure, typically ranges from 5 to 10 cm H₂O during quiet speech but can exceed 20 cm H₂O during loud speech or emphatic utterance. This pressure differential between the lungs and the atmosphere provides the power source for speech.

The phonatory system, centered on the larynx, converts aerodynamic energy into acoustic energy through the vibration of the vocal folds. The larynx, commonly known as the voice box, sits at the top of the trachea and houses the vocal folds—two bands of tissue stretched across the airway. The space between the vocal folds is called the glottis. When the vocal folds are held together and air is forced through them, they vibrate, producing a quasi-periodic source signal. The frequency of this vibration, called the fundamental frequency or F₀, determines the perceived pitch of the voice. Adult males typically have fundamental frequencies between 85 and 180 Hz, while adult females range from 165 to 255 Hz. Children generally have higher fundamental frequencies, and individual variation exists within and across populations.

The vocal folds can adopt several configurations beyond the vibrating mode:

• **Modal voice**: The normal vibrating pattern described above, used for voiced sounds like vowels and voiced consonants.
• **Falsetto**: The vocal folds vibrate with only the edges, producing a higher fundamental frequency with a different harmonic structure.
• **Vocal fry** (or creak): The vocal folds vibrate very slowly, sometimes with irregular closure patterns, producing a low-pitched, rattling quality.
• **Whisper**: The vocal folds are held apart, allowing air to pass through without vibration, producing unvoiced sounds.

The articulatory system modifies the acoustic signal produced by the phonatory system to create the full range of speech sounds. This system includes the oral cavity (mouth), nasal cavity, pharynx (throat), and various articulators:

• **Tongue**: Perhaps the most versatile articulator, capable of assuming an enormous range of shapes and positions. The tongue consists of several parts with independent movement capabilities: the tip (apex), blade (front), back (dorsum), and root.
• **Lips** (labial articulators): Can assume various shapes—rounded, spread, open, closed—to modify sounds.
• **Teeth** (dental articulators): Work with the tongue and lips to produce certain consonants.
• **Hard palate** (palatal articulator): The roof of the mouth, against which the tongue articulates for certain sounds.
• **Velum** (velar articulator): The soft palate at the back of the mouth, which can raise to close off the nasal cavity or lower to allow air to flow through the nose.

The combination of articulator positions determines the resulting speech sound. Vowels are primarily distinguished by the position of the tongue (high/mid/low, front/central/back) and lip rounding. Consonants are distinguished by several parameters:

• **Place of articulation**: Where in the vocal tract the constriction occurs (bilabial, labiodental, dental, alveolar, postalveolar, palatal, velar, glottal).
• **Manner of articulation**: How much constriction occurs and whether airflow is blocked, restricted, or allowed to escape (plosive/stop, fricative, affricate, nasal, approximant, tap/flap, lateral).
• **Voicing**: Whether the vocal folds vibrate during the production of the consonant.

1.3 Acoustic Theory of Speech Production

The acoustic signal that emerges from the speech production process can be analyzed using the principles of acoustics and linear systems theory. While the actual speech production mechanism is highly nonlinear and time-varying, the source-filter theory of speech production, developed by Gunnar Fant in the 1960s, provides an enormously useful approximation that forms the foundation of most classical speech processing approaches.

According to source-filter theory, speech production can be modeled as a linear system in which a sound source is modified by the filtering effects of the vocal tract. The primary sources of sound in speech are:

1. Glottal source: Vibration of the vocal folds produces a quasi-periodic waveform. The shape of this waveform, particularly its glottal flow derivative, affects the spectral characteristics of voiced speech. The glottal source has a fundamental frequency (F₀) and associated harmonics.

2. Turbulent noise source: Air flowing through a narrow constriction in the vocal tract becomes turbulent, producing noise with a broad spectrum. This occurs for unvoiced fricatives like /s/, /ʃ/, and /f/, as well as for aspiration.

3. Transient sources: Sudden releases of pressure, such as when the lips open after a stop closure, produce transient acoustic events with distinctive spectra