The Senses of Machines: An Introduction to Multimodal AI

Think about the last time you experienced something really memorable. Maybe it was a concert where you could feel the bass vibrating through your chest, see the colorful lights dancing across the stage, and hear your favorite song performed live. Or perhaps it was a delicious meal where the aroma hit you before you even took a bite, the flavors exploded on your tongue, and the textures made every bite interesting. Maybe it was a moment with friends where you saw their smiles, heard their laughter, heard their stories, and felt the warmth of being together.

What do all these experiences have in common? They weren't just one thing. They weren't just what you saw, or just what you heard, or just what you felt. They were all of these senses working together, combining into a single rich experience that you remember and that means something to you. When you think back to that concert, it's not just the music you remember—it's the whole feeling, the whole experience. The same is true for that meal or that moment with friends.

This is something so natural to us as humans that we rarely stop to think about it. Our five senses—sight, hearing, touch, taste, and smell—don't operate in isolation. When you walk into a room, you don't just see the furniture and then separately hear the sound of a fan running and then separately feel the temperature of the air. Your brain takes all of this information and weaves it together into a unified understanding of the space you're in. You know immediately that it's a cozy living room with a running fan and afternoon sunlight streaming through the window. This combination of sensory information is so automatic for us that we don't even notice the magic happening in our brains.

Now here's a question that might seem strange at first: What would happen if you could only use one sense at a time? Imagine trying to understand a movie with your eyes closed, only hearing the audio. Or trying to read a book while wearing a blindfold. Or trying to navigate a new city using only your sense of touch. Each of these would give you some information, but you'd be missing so much context. You'd struggle to understand the full picture.

This is exactly the situation we've been in with artificial intelligence for most of its history. For decades, AI systems were like people who could only use one sense at a time. An AI might be brilliant at recognizing cats in photos, but completely unable to understand what a sentence about cats means. Another AI might be excellent at translating text from French to English, but couldn't tell you what a French street sign looks like. Each system was an expert in its own narrow domain but had no way of connecting different types of information.

This is beginning to change in one of the most exciting ways imaginable. Scientists and engineers have been working on something called multimodal AI—systems that can understand and work with multiple types of data at the same time, much like humans use multiple senses together. Instead of just recognizing cats in images, these new systems can look at an image of a cat, read the caption that describes it, listen to someone say "that's a cute cat," and understand that all of these different inputs are about the same thing: a cat.

This might sound like a small technical improvement, but it's actually a fundamental shift in what artificial intelligence can do. It's like the difference between having a friend who can only talk about one topic and a friend who can have a real conversation about anything. The multimodal AI systems of today and tomorrow can connect information across different formats—text, images, audio, video, and more—in ways that create understanding far deeper than any single-sense system could achieve.

In this chapter, we're going to explore this fascinating world of multimodal AI. We'll start by understanding what makes traditional AI systems "unimodal" (meaning they work with just one type of data), and why this limitation held them back for so long. Then we'll dive into what multimodal AI is and how it overcomes these limitations. We'll look at real examples of multimodal AI that you might already be using without even realizing it—filters on social media, voice assistants that can see what you're showing them, video games that respond to both your words and your actions. And we'll preview the three groundbreaking technologies that we'll be exploring in depth throughout this book: CLIP, GPT-4V, and Gemini.

By the time you finish this chapter, you'll have a solid foundation for understanding why multimodal AI is one of the most important technological developments of our time—and why it matters for your future in ways you might not have imagined.

The One-Sense World: Understanding Unimodal AI

To really appreciate what multimodal AI can do, it helps to first understand what came before it. For most of AI's history, systems were designed to handle one specific type of data. Computer scientists call these "unimodal" systems, with "uni" meaning one and "modal" referring to the mode or type of data.

Think of unimodal AI as having different specialized experts, each locked in their own room and unable to communicate with anyone else. One expert is brilliant at looking at images but can't read a single word. Another expert can understand text perfectly but is completely blind to images. A third expert might be able to transcribe speech into text but has no idea what the words mean or what emotion is behind them.

This isn't because the AI researchers weren't smart enough to build better systems. For a long time, the technology just didn't exist to create AI that could handle multiple types of data. The mathematical techniques, the computer power, and the vast amounts of training data needed for such systems simply weren't available.

Let's look at some examples of unimodal AI that you might have encountered. Perhaps you've used a facial recognition system to unlock your phone. That system takes in visual data—an image of a face—and outputs a decision about whether that face matches the person who owns the device. It's doing one thing with one type of data: analyzing images. It doesn't know anything about language, about sound, about touch. If you tried to ask it a question, it would have no idea what you were talking about.

Or maybe you've used a voice-to-text feature to transcribe a lecture or a meeting. This system takes in audio data—sound waves—and converts them into text. It's doing one thing with one type of data: turning spoken words into written words. It doesn't know what the words mean, what the speaker looked like, or what images were shown during the lecture.

Or perhaps you've used a language translation app. This system takes in text in one language and outputs text in another language. It's doing one thing with one type of data: converting words from one language to another. It doesn't know anything about the context in which those words were spoken, about the speaker's tone of voice, or about any images that might accompany the text.

Each of these systems is impressive in its own right. They represent years of research and development, and they can perform their specific tasks quite well. But they're fundamentally limited by their single-mindedness.

Here's a simple thought experiment that illustrates this limitation. Imagine showing an unimodal image recognition AI a photograph of a dog wearing a superhero cape, with the text "SUPERPUP SAVES THE DAY" written on the cape. A sophisticated image recognition system might be able to identify that there's a dog in the image, maybe even identify the specific breed of dog. But it would have no idea about the cape, the text, or the joke that the image is trying to convey.

Now imagine showing this same image to an unimodal text understanding AI. It could read the words "SUPERPUP SAVES THE DAY" and understand that this is a phrase suggesting heroic action. But it would have no idea wha