Foundations of Speech Processing

The human capacity for speech represents one of the most remarkable achievements of biological information processing. Every second, a speaker generates a complex acoustic signal containing thousands of bits of linguistic information—the meanings, intentions, and emotions that constitute meaningful communication. Understanding how this remarkable feat is accomplished, both by the human biological system and by the artificial systems we have constructed to process and synthesize speech, forms the foundational knowledge upon which all modern speech AI is built. This chapter establishes the conceptual and historical groundwork necessary for comprehending the journey from early mechanical speech devices to contemporary neural network-based systems that achieve human-like performance across a wide range of speech processing tasks.

The Human Speech Production Mechanism

Anatomy of the Speech Production System

Human speech production emerges from the coordinated action of three primary anatomical subsystems: the respiratory system, the phonatory system, and the articulatory system. The respiratory system, comprising the lungs, trachea, and associated musculature, provides the aerodynamic energy that drives speech production. During normal tidal breathing, approximately 500 milliliters of air moves in and out of the lungs with each breath. However, during speech, the respiratory system operates under significantly different conditions. Speakers typically exhale approximately 1500 to 2000 milliliters of air during a single utterance, and this exhalation is carefully controlled to provide a relatively constant flow rate that supports stable voice production.

The phonatory system, centered on the larynx and specifically the vocal folds (commonly called vocal cords), serves as the primary sound source for voiced speech sounds. The larynx is a complex structure of cartilages, muscles, and membranes located at the top of the trachea. The vocal folds are two bands of muscular tissue that can be positioned through the action of various intrinsic and extrinsic laryngeal muscles. When the vocal folds are brought together and air from the lungs is forced through the narrow opening between them, the folds vibrate in a periodic manner, producing a quasi-periodic acoustic signal. The fundamental frequency of this vibration, which corresponds to the perceived pitch of the voice, is determined by the mass, tension, and length of the vocal folds, as well as the subglottal pressure provided by the respiratory system.

The fundamental frequency of vocal fold vibration during speech typically ranges from approximately 100 Hz for adult male voices to 250 Hz for adult female voices, with children's voices often reaching 300 Hz or higher. However, this fundamental frequency varies considerably both within and across speakers. During normal conversation, the fundamental frequency may vary by as much as an octave as speakers express different emotional states, emphasize different words, or signal different pragmatic functions. Professional singers can extend this range even further, with trained sopranos capable of producing fundamental frequencies exceeding 1000 Hz.

The articulatory system shapes the acoustic signal produced by the vocal folds into the full range of speech sounds we recognize as language. This system includes the oral cavity (mouth), the nasal cavity, the pharynx (throat), and the various articulators: the tongue, lips, teeth, hard palate (roof of the mouth), velum (soft palate), and uvula. The tongue is particularly important for speech production, as its extreme flexibility allows it to assume many different positions that dramatically alter the shape of the vocal tract. The lips can be rounded, spread, or neutral, and can close completely to produce stop consonants. The velum can be raised to close off the nasal cavity for oral sounds or lowered to allow nasal airflow for nasal consonants like /m/, /n/, and /ŋ/.

The Acoustic Theory of Speech Production

The mathematical description of speech production begins with the acoustic theory developed by Fant and others in the mid-twentieth century. At its core, this theory describes speech production as a linear system in which an acoustic source is filtered by the time-varying resonances of the vocal tract. This source-filter model, which forms the foundation of virtually all approaches to speech processing, holds that the complex acoustic signal produced during speech can be decomposed into a source component and a filter component that operates independently.

For voiced speech, the source is the acoustic signal produced by vocal fold vibration. This glottal source signal is approximately periodic, with a fundamental frequency determined by the rate of vocal fold vibration. However, the raw glottal waveform is not a pure sinusoid but rather a complex shape that contains energy at the fundamental frequency and numerous harmonics at integer multiples of this fundamental. The relative amplitudes of these harmonics depend on the precise shape of the glottal pulse, which in turn depends on the physiological characteristics of the larynx and the manner in which the vocal folds vibrate.

The filter component of the source-filter model is provided by the vocal tract, which acts as an acoustic resonator. The vocal tract, extending from the vocal folds to the lips (and including the nasal cavity when the velum is lowered), is approximately 17 centimeters long in an average adult male. Because this tube is closed at one end (the vocal folds) and open at the other (the lips), it behaves as a quarter-wave resonator with a series of resonant frequencies, called formants, that depend on the length of the vocal tract and the positions of the various articulators.

The formant frequencies are among the most important acoustic parameters in speech processing. The first formant (F1) is primarily determined by the degree of mouth opening and the height of the tongue. A high F1 corresponds to an open vowel like /a/ (as in "father"), while a low F1 corresponds to a close vowel like /i/ (as in "beat"). The second formant (F2) is primarily determined by the front-back position of the tongue. A high F2 corresponds to a front vowel like /i/, while a low F2 corresponds to a back vowel like /u/ (as in "boot"). The third formant (F3) is also influenced by lip rounding and the position of the tongue tip, and plays an important role in distinguishing certain consonants, particularly liquids like /r/ and /l/.

The relationship between vocal tract length and formant frequencies can be derived from acoustic theory. For a uniform tube closed at one end and open at the other, the resonant frequencies occur at odd quarter-wavelengths of the acoustic wave. The fundamental resonant frequency of such a tube is given by:

$$f_1 = \frac{c}{4L}$$

where $c$ is the speed of sound in air (approximately 35,000 cm/s at room temperature) and $L$ is the length of the vocal tract. For an adult male with a vocal tract length of approximately 17 cm, this yields a first formant of approximately 500 Hz. The higher formants occur at frequencies of $3c/4L$, $5c/4L$, and so on, corresponding to approximately 1500 Hz, 2500 Hz, and higher frequencies for a typical male speaker.

The acoustic impedance of the vocal tract walls also affects formant frequencies and bandwidths. The walls of the vocal tract, particularly in the pharynx and oral cavity, have some compliance (elasticity) that allows them to vibrate in response to the acoustic pressure inside the tract. This wall compliance causes a slight lowering of the formant frequencies and an increase in formant bandwidths compared to what would be predicted for a rigid-walled tube. These wall vibration effects are more pronounced for higher formants and can be significant for formant frequencies above approximately 3000 Hz.

The Source-Filter Model in Detail

The source-filter model provides a powerful