强化学习概述 | Introduction to Reinforcement Learning

第 1 章：强化学习概述

1.1 什么是强化学习

强化学习（Reinforcement Learning，简称 RL）是机器学习的一个重要分支，它专注于智能体（Agent）如何在环境中通过试错学习最优决策策略。与传统机器学习方法不同，强化学习不依赖于标注好的输入输出对，而是通过与环境的交互，利用奖励（Reward）或惩罚（Penalty）信号来指导学习过程。

Reinforcement Learning (RL) is a crucial branch of machine learning that focuses on how intelligent agents can learn optimal decision-making strategies through trial and error in an environment. Unlike traditional machine learning methods, RL does not rely on labeled input-output pairs. Instead, agents learn by interacting with the environment and using reward or penalty signals to guide the learning process.

从本质上讲，强化学习模拟了人类和动物学习的基本机制。当一个婴儿学习走路时，她会不断尝试各种动作组合，有些动作帮助她保持平衡并向前移动（获得正向反馈），而有些动作导致她摔倒（获得负向反馈）。通过无数次这样的尝试，婴儿逐渐学会了最优的走路策略。强化学习正是对这一学习过程的数学化和算法化表达。

Essentially, RL simulates the fundamental mechanisms of human and animal learning. When a baby learns to walk, she continuously attempts various combinations of movements—some actions help her maintain balance and move forward (receiving positive feedback), while others cause her to fall (receiving negative feedback). Through countless such attempts, the baby gradually learns the optimal walking strategy. Reinforcement learning is precisely the mathematical and algorithmic expression of this learning process.

强化学习问题可以用一个四元组来形式化描述：$(S, A, P, R)$。其中 $S$ 表示状态空间（State Space），包含智能体可能处于的所有状态；$A$ 表示动作空间（Action Space），包含智能体可以执行的所有动作；$P$ 表示状态转移概率函数（Transition Probability Function），描述在给定当前状态和动作时，下一状态的概率分布；$R$ 表示奖励函数（Reward Function），定义智能体在特定状态执行特定动作后获得的即时奖励。

A reinforcement learning problem can be formally described by a quadruple: $(S, A, P, R)$. Here, $S$ represents the state space, containing all possible states the agent may be in; $A$ represents the action space, containing all actions the agent can perform; $P$ denotes the transition probability function, describing the probability distribution of the next state given the current state and action; and $R$ is the reward function, defining the immediate reward the agent receives after executing a specific action in a specific state.

```mermaid graph LR A[智能体 Agent] -->|动作 Action| B((环境 Environment)) B -->|状态 State| A B -->|奖励 Reward| A A -->|观测 Observation| C[策略 Policy] C --> A ```

上图展示了强化学习的核心交互范式。智能体通过观测（Observation）当前环境状态，根据其策略（Policy）选择并执行动作。环境接收动作后，转移到新的状态并返回奖励信号。智能体根据奖励信号更新其策略，循环往复，直到达到目标状态或满足终止条件。

The figure above illustrates the core interaction paradigm of reinforcement learning. The agent observes the current environment state, selects and executes an action based on its policy. After receiving the action, the environment transitions to a new state and returns a reward signal. The agent updates its policy based on the reward signal, repeating this cycle until reaching a goal state or satisfying termination conditions.

1.2 强化学习的核心要素

1.2.1 智能体与环境

智能体是强化学习系统的决策者，它具备感知环境、做出决策、学习改进的能力。在不同的应用场景中，智能体可以是一个游戏中的玩家、机器人控制系统、自动驾驶汽车，甚至是商业决策系统中的算法引擎。

The agent is the decision-maker in a reinforcement learning system, possessing the capabilities to perceive the environment, make decisions, and improve through learning. In different application scenarios, the agent can be a player in a game, a robot control system, a self-driving car, or even an algorithm engine in a business decision system.

环境是智能体所处的外部世界，它决定了智能体的状态空间、动作空间以及状态转移规则。环境可以是确定性的（Deterministic），也可以是随机性的（Stochastic）。确定性环境中，给定当前状态和动作，下一状态是唯一确定的；而在随机性环境中，相同的当前状态和动作可能导致不同的下一状态，这些转移以概率形式描述。

The environment is the external world in which the agent operates. It determines the agent's state space, action space, and state transition rules. The environment can be deterministic or stochastic. In a deterministic environment, given the current state and action, the next state is uniquely determined; while in a stochastic environment, the same current state and action may lead to different next states, with these transitions described in probabilistic terms.

```python

强化学习环境的Python伪代码示例

class Environment: """强化学习环境基类"""

def __init__(self): self.state = None # 当前状态 self.action_space = None # 动作空间 self.observation_space = None # 观测空间（状态空间）

def reset(self): """重置环境，返回初始状态""" raise NotImplementedError

def step(self, action): """ 执行动作并返回 (next_state, reward, done, info)

参数: action: 智能体选择的动作 返回: next_state: 执行动作后的新状态 reward: 获得的奖励（标量值） done: 布尔值，指示回合是否结束 info: 额外信息（调试用） """ raise NotImplementedError

def render(self): """渲染环境（可视化）""" pass ```

上述代码展示了强化学习环境的标准接口定义（参考OpenAI的Gymnasium库设计）。`reset()`方法用于初始化或重置环境状态，`step()`方法接收智能体的动作输入，返回下一状态、奖励、终止标志和附加信息，这是构建强化学习系统的基础约定。

The code above demonstrates the standard interface definition for a reinforcement learning environment (referencing the Gymnasium library design from OpenAI). The `reset()` method is used to initialize or reset the environment state, while the `step()` method receives the agent's action input and returns the next state, reward, termination flag, and additional information. This is the foundational convention for building reinforcement learning systems.

1.2.2 状态与观测

状态（State）是对环境当前情况的完整描述，记作 $s_t \in S$，其中 $t$ 表示时间步。理想情况下，智能体应该能够获取环境的完整状态信息。但在实际应用中，智能体往往只能获得部分观测（Partial Observation），这被称为部分可观测马尔可夫决策过程（Partially Observable Markov Decision Process，POMDP）。

A state is a complete description of the current situation in the environment, denoted as $s_t \in S$, where $t$ represents the time step. Ideally, the agent should be able to obtain complete state information about the environment. However, in practical applications, the agent often only obtains partial observations, which is known as a Partially Observable Markov Decision Process (POMDP).

以自动驾驶为例，智能体（自动驾驶系统）无法获取道路的完整信息，只能通过车载传感器（摄像头、雷达、激光雷达等）获取局部观测。这些观测可能存在噪声、不完整或具有欺骗性。智能体需要在这些不完美的观测基础上做出决策，这对学习算法提出了更高的要求。

Take autonomous driving as an example: the agent (autonomous driving system) cannot obtain complete information about the road, and can only acquire partial observations through vehicle sensors (cameras, radar, LiDAR, etc.). These observations may contain noise, be incomplete, or even misleading. The agent needs to make decisions based on these imperfect observations, which places higher demands on learning algorithms.

```python import numpy as np

class POMDPEnvironment: """部分可观测环境的示例"""

def __init__(self):

真实状态（智能体不可见）

self.true_state = np.array([0.0, 0.0, 0.0]) # 位置、速度、方向

传感器参数

self.sensor_noise = 0.1

def get_observation(self): """从真实状态生成带噪声的观测""" noise = np.random.randn(3) * self.sensor_noise observation = self.true_state + noise return observation # 智能体只能看到这个

def step(self, action): """执行动作，更新真实状态"""

真实状态转移（智能体未知）

self.true_state = self.true_state + action + np.random.randn(3) * 0.05 return self.get_observation() ```

1.2.3 奖励与回报

奖励函数 $R(s, a, s')$ 定义了在状态 $s$ 下执行动作 $a$，转移到状态 $s'$ 后获得的即时奖励。这个奖励信号是强化学习的核心驱动力，它指导智能体学习什么样的行为是"好"的，什么样的行为是"坏"的。

The reward function $R(s, a, s')$ defines the immediate reward obtained after executing action $a$ in state $s$ and transitioning to state $s'$. This reward signal is the core driving force of reinforcement learning, guiding the agent to learn what behaviors are "good" and what behaviors are "bad".

然而，智能体的目标不是最大化即时奖励之和，而是最大化长期累积奖励（Long-term Cumulative Reward），也称为回报（Return）。回报通常定义为：

However, the agent's goal is not to maximize the sum of immediate rewards, but to maximize long-term cumulative rewards, also known as Return. Return is typically defined as:

$$G_t = R_{t+1} + R_{t+2} + R_{t+3} + \cdots + R_T = \sum_{k=0}^{T-t} R_{t+k+1}$$

其中 $T$ 是终止时间步。对于持续性任务（Continuing Tasks），$T = \infty$，直接求和会导致回报发散。此时需要引入折扣因子（Discount Factor）$\gamma \in [0, 1)$，定义折扣回报（Discounted Return）：

Where $T$ is the terminal time step. For continuing tasks where $T = \infty$, direct summation would cause th