开篇：当机器睁开'眼睛'的那一刻
第 1 章 开篇：当机器睁开"眼睛"的那一刻
> Chapter 1 The Moment When the Machine Opened Its "Eyes" > > *Tag: #multimodal #foundation #CLIP #perception*
---
1.1 一个清晨的"魔法"，以及它背后藏着的革命
A morning of "magic" — and the revolution hiding behind it
清晨七点十五分，你迷迷糊糊地伸手摸到手机，习惯性地往上一划。 *At 7:15 in the morning, you fumble for your phone and swipe up out of habit.*
屏幕亮起的那一瞬间，你可能没意识到，自己已经和一种全新形态的智能打了四次照面： *The instant the screen lights up, you may not realize that you've already encountered a brand-new form of intelligence four times:*
第一次，锁屏界面上那张照片——昨天傍晚你在江边随手拍的逆光剪影——被手机自动归类到"风景 / 黄昏 / 人物剪影"文件夹里。它甚至自动加了个滤镜推荐：暖色调、HDR 加强。 *First, the photo on your lock screen — the silhouette you casually shot against the backlight by the river yesterday evening — has been automatically filed into a folder called "Scenery / Dusk / Silhouettes." It even auto-recommends a filter: warm tone, HDR enhancement.*
第二次，你打开微博，看到一段你关注的博主刚发的视频。视频下方赫然出现一行字："本视频主要内容：一只橘猫试图从纸箱里跳出来，三次失败，最终成功，但纸箱被它压扁了。" *Second, you open Weibo and see a new video from a creator you follow. Below the video is a line: "Main content: an orange cat tries to jump out of a cardboard box, fails three times, finally succeeds, but crushes the box."*
第三次，你点开了一段英文公开课的音频。一段略带机械感、但发音相当流利的女声从耳机里流出来，把那个语速飞快、夹杂学术术语的教授讲的话实时翻译成中文。 *Third, you tap into an English lecture audio. A slightly mechanical yet quite fluent female voice streams from your earphones, translating the fast-talking professor's jargon-filled speech into Chinese in real time.*
第四次，你对着手机说了一句："嘿，Siri，把明早七点的闹钟改成六点半。"Siri 不仅听懂了，还反问一句："您是要改明天的那一个，还是所有工作日的闹钟？" *Fourth, you tell your phone: "Hey Siri, change tomorrow's 7 a.m. alarm to 6:30." Siri not only understands but asks: "Do you want to change just tomorrow's, or all weekday alarms?"*
把手机放下，你心里可能浮现一个模糊的疑问：*Putting down the phone, a vague question surfaces:*
> 这些看起来已经"理所当然"的功能，到底是什么时候变得这么聪明的？它们之间，有什么共同的秘密？
> *When exactly did these "taken-for-granted" features get so smart? What common secret lies behind them?*
如果你愿意把这个问题再向前推一步，你会撞上 2023 年以来科技界最热闹、也最让人困惑的一组词：多模态大模型 (Multimodal Large Models)。 *If you push that question one step further, you'll bump into the hottest yet most confusing set of buzzwords in tech since 2023: Multimodal Large Models.*
GPT-4V、Gemini、Claude 3、文心一言、Qwen-VL、智谱 GLM-4V、InternVL……这些名字像爆米花一样在新闻里噼里啪啦地跳出来。它们既能看图、又能听话、还能写字；它们既能帮你读论文里的公式，又能帮你给朋友圈照片配文案，甚至能根据你画的一张草图直接生成一个能跑的网站。 *GPT-4V, Gemini, Claude 3, Ernie Bot, Qwen-VL, Zhipu GLM-4V, InternVL… these names pop like popcorn in the news. They can see images, hear speech, and write text; they can help you read formulas in a paper, draft a caption for a WeChat Moments photo, or even generate a working website from a sketch you draw.*
而这一切的起点，要从一个更朴素的问题讲起——"模态"，到底是什么？ *And the starting point of all this is a more fundamental question — what on earth is a "modality"?*
在接下来的这一章里，我们不会急着抛术语，而是先和你一起，回到人类最熟悉的场景：你的感官。 *In this chapter, we won't rush to throw jargon at you. Instead, we'll start with the most familiar scene: your senses.*
我们将从三个问题展开： *We will unfold the discussion through three questions:*
1. 什么是"模态"？为什么文字、图像、声音都可以被叫做"模态"？ *What is a "modality"? Why can text, images, and sound all be called "modalities"?* 2. 为什么"只会一种语言"的单模态模型注定走不远？ *Why are single-modal models that "only speak one language" bound to fall short?* 3. 多模态到底"多"在哪里？它凭什么在 2021 年之后突然爆发？ *Where exactly is the "multi" in multimodal? Why did it suddenly explode after 2021?*
最后，我们会一起做一个"三 AI 对比小实验"，让你用最普通的手机，就能直观地看见：纯文字 AI、纯图像 AI、图文混合 AI，面对同一张照片时，回答会有多么不同。 *Finally, we'll do a "three-AI comparison experiment" together, using your ordinary phone, to see intuitively how a text-only AI, an image-only AI, and an image-text AI give completely different answers when faced with the same photo.*
读完这一章，你应该能用一句话向朋友讲清楚：多模态大模型，到底是什么。 *After reading this chapter, you should be able to explain to a friend in one sentence: what a multimodal large model actually is.*
让我们从一个真实的小故事开始。 *Let's start with a real little story.*
---
1.2 什么是"模态"？感官的语言——文字、图像、声音都是模态
What is a "modality"? The language of the senses — text, image, and sound are all modalities
在正式给出"模态"这个词的定义之前，我想先邀请你做一件非常私人的事：闭上眼，回想一下你上一次被"惊艳"到的瞬间。 *Before formally defining "modality," I'd like to invite you to do something very personal: close your eyes and recall the last time you were truly "wowed."*
也许是你看了一部画面极美的电影，听了一段让你起鸡皮疙瘩的交响乐，读到了一行让你心头一震的句子，又或者是闻到了小时候外婆家厨房里飘出来的那股腊肉香…… *Maybe you watched an extraordinarily beautiful movie, heard a symphony that gave you goosebumps, read a line that struck your heart — or smelled the cured meat aroma from your grandmother's kitchen when you were a child…*
你会发现，这些"惊艳"的瞬间，几乎都不是单一感官在起作用。 *You'll notice that almost none of these "wow" moments are triggered by a single sense.*
- 看电影时，**视觉**（画面） + **听觉**（配乐） + **文字**（字幕） 联合起来才让你起了一身鸡皮疙瘩；
*When watching a movie, vision (images) + hearing (soundtrack) + text (subtitles) together give you goosebumps;*
- 闻到外婆家腊肉香时，**嗅觉** + **味觉** + **记忆中的画面** 一起把你拉回了童年；
*When smelling grandma's cured meat, smell + taste + the remembered image pull you back to childhood;*
- 读到一行让你心头一震的句子，**视觉（文字）** + **听觉（你脑海中的朗读声）** + **情绪** 三者共振。
人类感知世界的方式，从一开始就是"多模态"的。我们的眼睛、耳朵、皮肤、舌头、鼻子，并不是五个孤立的信息接收器，而是一个彼此协作、互相校验的"感知联盟"。 *How humans perceive the world has been "multimodal" from the start. Our eyes, ears, skin, tongue, and nose are not five isolated information receivers — they are a "perception alliance" that cooperates and cross-validates with each other.*
那么问题来了：科学家们怎么把这种"人类感官的分工"，翻译成机器能懂的语言呢？ *So the question is: how do scientists translate this "division of sensory labor" into a language machines can understand?*
答案，就是"模态 (Modality)"这个词。 *The answer is the word "Modality."*
1.2.1 一个朴素但精准的定义
A simple yet precise definition
所谓"模态"，你可以把它理解为信息存在或传递的形式。 *A "modality" can be understood as the form in which information exists or is transmitted.*
具体来说： *Specifically:*
- 文字，是一种模态。它是人类发明的一种**高度抽象、离散的符号系统**。你看到"苹果"两个字，脑子里立刻浮起一个红红的、圆圆的、可能还有点酸甜的形象——但你看到的其实是两个抽象的符号。
*Text is a modality. It is a highly abstract, discrete symbol system invented by humans. When you see the two characters "苹果" (apple), your mind immediately conjures a red, round, maybe slightly tart-sweet image — but what you actually see is two abstract symbols.*
- 图像，是另一种模态。它是**连续的、高维度的像素阵列**。一张 1024×1024 的彩色照片，包含 3145728 个像素点，每个点又有 RGB 三个数值（0~255）。换句话说，一张图就是一组 9437184 个数字的集合。
*Image is another modality. It is a continuous, high-dimensional pixel array. A 1024×1024 color photo contains 3,145,728 pixels, each with three RGB values (0–255). In other words, an image is a set of 9,437,184 numbers.*
- 声音，又是一种模态。它是**随时间变化的、一维的波形信号**。一段 10 秒、采样率 44.1kHz 的 CD 音质音频，背后是 441000 个连续的振幅数值。
*Sound is yet another modality. It is a one-dimensional waveform signal that changes over time. Ten seconds of CD-quality audio at 44.1 kHz is, behind the scenes, 441,000 continuous amplitude values.*
- 视频，可以理解为**图像 + 时间**的复合模态，本质上是"每秒 24~60 张图像 + 一条音轨"。
*Video can be understood as a composite modality of image + time — essentially "24–60 images per second + an audio track."*
- 更进一步，温度、压力、加速度、脑电波、基因序列……只要是**某种传感器能稳定捕捉、并且能数字化**的信号，都可以被视为一种模态。
*Going further, temperature, pressure, acceleration, EEG, gene sequences… as long as some sensor can stably capture and digitize the signal, it can be considered a modality.*
把这些定义浓缩成一句话，就是本书会反复用到的一句话：
> 模态 = 信息被某种"器官"（生物的或人工的）感知并以某种"形式"组织起来的方式。
> *Modality = the way information is sensed by some "organ" (biological or artificial) and organized in some "form."*
> Text, image, sound, video, temperature, acceleration, even brain waves — all are modalities.
> *文字、图像、声音、视频、温度、加速度，甚至脑电波——都是模态。*
1.2.2 模态的两副"面孔"：离散 vs. 连续
The two "faces" of modalities: discrete vs. continuous
理解了"模态 = 信息的形式"之后，我们还要补一层更深的认知——不同模态之间，是有"性格差异"的。 *After understanding "modality = the form of information," we need one more layer: different modalities have different "personalities."*
大致来说，可以把常见模态分成两类： *Roughly speaking, common modalities can be