The One-Sided AI: Why Early Machines Could Only Read, or Only See, or Only Hear

The screen of Sarah’s iPhone 5s glowed in the dark of her son’s bedroom, the only light source as she sat cross-legged on the carpet, surrounded by scattered teething toys and half-empty sippy cups. It was 2 a.m., the baby finally asleep after an hour of fussing, and she’d just remembered she’d taken a video of his first wobbly steps that afternoon—she wanted to send it to her mom who lived across the country. She opened the Photos app, scrolled past 3,247 unorganized shots and 17 videos from the past month, and tapped the search bar at the top. She typed in *baby first steps*, hit enter, and waited.

The results that popped up made her snort so loud she almost woke the baby. First was a blurry photo of the cat batting at a dangling toy. Then a screenshot of a Pinterest post titled “5 Tips for Baby’s First Steps.” Then a selfie she’d taken three weeks prior, grinning next to a “World’s Okayest Mom” mug, with the auto-generated tag “baby” because the algorithm had detected a tiny onesie peeking out from the couch behind her. Not a single photo or video of her son taking a single step.

“Useless,” she muttered, and started scrolling manually through the month’s worth of footage, thumb cramping within ten minutes.

Sarah didn’t know it at the time, but she was running headfirst into one of the biggest limitations of early artificial intelligence: the *modality silo* problem. For the first 60 years of AI research, machines were built to handle only one type of information at a time—either text, or images, or sound—and the systems that processed each type operated in completely separate, disconnected worlds. A machine that could read and understand a novel had no way to process a photo of the novel’s movie adaptation. A system that could identify a cat in a photo had no ability to understand a caption describing the cat’s silly antics. A speech recognition tool that could transcribe a podcast had no way to connect the spoken words to the sound of the host’s laughing in the background. These silos weren’t an intentional design choice—they were the inevitable product of technical limitations, academic silos, and a fundamental misunderstanding of how human understanding actually works.

The Three Islands of Early AI: Text, Vision, and Hearing

To understand why early AI was so one-sided, you first have to understand what researchers mean by a *modality*. If you think about how you experience the world, you don’t just process words, or just see pictures, or just hear sounds. Every moment is a mashup of all three: you see your baby’s wobbly legs, hear his giggles, and hear your partner cheering “There he goes!” all at the same time. Each of these separate streams of information is a modality—a distinct format for representing and communicating information. Text is a modality. Photographs are a modality. Spoken audio is a modality. Even the background hum of the refrigerator in your kitchen is a separate audio modality, distinct from your baby’s laugh.

From the 1950s through the early 2010s, AI research split almost entirely into three separate fields, each focused on a single modality, with almost no overlap between them.

The Text Island: Natural Language Processing

The first modality to be seriously studied was text, born out of early work in machine translation and computational linguistics. In 1954, the Georgetown-IBM experiment demonstrated the first public machine translation system, which could translate 60 Russian sentences into English using a simple rule-based system and a dictionary of 250 words. By the 1960s, researchers had built ELIZA, a simple chatbot that could mimic a psychotherapist by matching user input to pre-written response rules. These early systems were primitive, but they proved that machines could at least *manipulate* text symbols, even if they didn’t understand what the symbols meant.

For the next 50 years, natural language processing (NLP) evolved along its own track, focused almost entirely on text-based tasks: classifying the sentiment of a movie review, translating text between languages, answering questions based on a written passage. The core techniques of NLP for decades were built around discrete symbol manipulation: n-gram models that counted how often words appeared next to each other, one-hot encodings that represented each word as a vector of zeros with a single 1 in the position corresponding to that word in a fixed vocabulary, and rule-based systems that matched text patterns to pre-defined responses.

For context, a one-hot encoding for a vocabulary of 100,000 words looks like this: for the word “苹果” (apple), the vector $\mathbf{x}_{\text{apple}}$ is a 100,000-dimensional vector where only the 1,203rd position is 1, and all others are 0: $$\mathbf{x}_{\text{apple}} = [0, 0, ..., 1, ..., 0]$$ This representation is simple to compute, but it captures zero semantic meaning: the vector for “apple” is just as different from the vector for “pear” as it is from the vector for “airplane,” even though apples and pears are both fruits. For decades, NLP researchers had no way to bridge this gap, and no reason to look outside the text modality to do so.

The Vision Island: Computer Vision

While NLP researchers were playing with text, a separate group of researchers was working on getting machines to “see.” The earliest computer vision work dates back to the 1960s, when researchers built simple systems that could detect edges and basic shapes in images. But progress was slow for decades, because vision is fundamentally harder than text: text is a discrete, structured sequence of symbols, while an image is a continuous, unstructured grid of pixel values. A 1-megapixel photo is a grid of 1 million numbers, each representing the brightness of a single pixel, and figuring out how to turn those numbers into meaningful concepts like “cat” or “baby taking first steps” is an exponentially harder problem than matching text patterns.

For most of the 1980s and 1990s, computer vision (CV) relied on *hand-crafted features*: researchers would manually design mathematical formulas to extract specific patterns from images, like edges (SIFT features), textures (HOG features), or colors. These features worked for simple tasks like detecting faces or recognizing handwritten digits, but they failed completely at more complex tasks. There was no hand-crafted feature that could detect “a baby taking its first wobbly steps”—that concept requires understanding the context of the scene, the pose of the baby, the reaction of the people around it, and the emotional weight of the moment, none of which could be captured by a formula for edge detection.

By the early 2000s, the best image classification systems on the ImageNet dataset (a collection of 1.2 million labeled images across 1,000 categories) had a top-5 error rate of 25%—meaning they got the wrong answer one out of every four times. Image search was almost entirely reliant on user-added tags: if you uploaded a photo of your baby’s first steps and didn’t tag it with “baby” and “first steps,” no search engine would ever find it. The machines could see pixels, but they couldn’t understand what those pixels meant.

The Hearing Island: Automatic Speech Recognition

The third separate island of early AI was automatic speech recognition (ASR), the field focused on getting machines to transcribe and understand spoken audio. The first speech recognition system was built by Bell Labs in 1952, and could recognize only 10 digits spoken by a single speaker in a quiet room. Progress was slow for decades, because audio is a continuous, noisy signal: a single second of speech is a wave of 16,000 or more data points, and separating the relevant speech from background noise, accents, and variations in speaking style is an incredibly hard problem.

For most of the 1990s and 2000s, ASR systems were limited to narrow, controlled use cases: dictation software for office workers, automated phone menus t