What Is Reinforcement Learning? The Agent-Environment Paradigm

Imagine you are learning to ride a bicycle for the first time. No one hands you a manual with instructions, nor do you have a dataset of millions of cycling examples. Instead, you must discover how to balance, pedal, and steer through trial and error. When you wobble and fall, your body registers pain and failure. When you manage to stay upright for a few seconds, you feel a sense of accomplishment. Gradually, through countless attempts, your muscles learn the subtle adjustments needed to maintain balance. By the time you've mastered cycling, you don't consciously think about each movement—you simply *know* how to do it. This process of learning through interaction with the environment, guided by feedback in the form of rewards and punishments, lies at the heart of reinforcement learning.

Reinforcement learning (RL) represents one of the three fundamental paradigms of machine learning, alongside supervised learning and unsupervised learning. However, unlike its counterparts, reinforcement learning tackles a uniquely challenging problem: how can an autonomous agent learn to make optimal decisions when it must deal with delayed consequences, uncertain outcomes, and a constantly changing environment? In this opening chapter, we will establish the foundational concepts that underpin all of reinforcement learning, introducing you to the agent-environment framework that serves as the conceptual foundation for everything that follows in this book.

The Agent-Environment Framework

At the core of reinforcement learning lies a beautifully simple yet powerful abstraction: the agent-environment interaction. This framework reduces the problem of learning to make decisions into an elegant interaction between two entities. The agent is the learner or decision-maker—the entity that perceives its environment and takes actions within it. The environment is everything outside the agent—the world with which it interacts, including the dynamics that govern how the world responds to the agent's actions.

Consider our bicycle example from the opening paragraph. You, as the rider, are the agent. The bicycle, the road surface, the wind, and the laws of physics that govern how a bicycle behaves—all of these constitute the environment. When you turn the handlebars (your action), the bicycle responds by changing direction (environment's response). When you lean your body weight, the bicycle tilts and either maintains balance or tips further. Every sensation you feel—the pressure on your hands, the tilt of your body, the vibration through the seat—is information from the environment that you can use to inform your next decision.

This agent-environment separation is not merely philosophical; it is a practical tool for thinking about and designing reinforcement learning systems. The key insight is that the agent has control over its own decision-making mechanisms but typically has no direct control over the environment's dynamics. The agent can choose what actions to take, but it cannot unilaterally determine what happens as a result. This distinction is crucial because it frames the learning problem: the agent must learn to predict and influence the environment's responses through its actions.

It is important to note that the boundary between agent and environment is not always obvious or fixed. In some cases, what we consider part of the agent in one context might be considered part of the environment in another. For example, in a robotic system, we might consider the robot's motors as part of the agent (since we control them) but the robot's physical body and sensors as part of the environment. The art of designing reinforcement learning systems often involves thoughtfully defining this boundary in ways that facilitate learning.

The Reinforcement Learning Interaction Loop

Reinforcement learning is fundamentally about learning from interaction. An agent interacts with its environment in a sequence of discrete time steps, and from these interactions, it learns to improve its decision-making behavior. This interaction can be formalized as a cyclical process that repeats continuously during the learning process.

At each time step $t$, the agent receives a representation of the environment's current state, denoted as $s_t$. This state information encodes all the relevant aspects of the environment that the agent can perceive and that are important for making decisions. Based on this state, the agent selects an action $a_t$ according to its current policy $\pi$. The policy can be thought of as the agent's strategy or rule for choosing actions—it defines the behavior of the agent.

After taking the action, two important things happen. First, the environment transitions to a new state $s_{t+1}$ according to its dynamics, which are typically described by a transition function or model. Second, the environment provides the agent with a reward signal $r_{t+1}$, which is a scalar value that indicates how good or bad the immediate outcome was. This reward serves as the primary feedback mechanism that guides the agent's learning.

This process continues iteratively: the agent observes the new state, selects another action, receives another reward, and so on. The sequence of states, actions, and rewards forms a trajectory or episode of interaction. The fundamental goal of reinforcement learning is for the agent to learn a policy that maximizes the total reward it receives over the long run.

```mermaid graph TD A[Start: Initial State s₀] --> B[Agent observes state sₜ] B --> C[Agent selects action aₜ using policy π] C --> D[Environment responds] D --> E[New state sₜ₊₁] D --> F[Reward rₜ₊₁] E --> G{Continue?} F --> G G -->|Yes| B G -->|No| H[End of Episode] ```

The interaction loop illustrated in this diagram is deceptively simple but captures the essence of what makes reinforcement learning both powerful and challenging. The power comes from its generality—this framework can describe learning in a vast array of domains, from board games to robot control to economic decision-making. The challenges arise from the fact that the agent must learn based on scalar rewards that provide only limited information about the quality of its decisions, and it must balance the desire to take actions that yield immediate rewards against the need to explore actions that might lead to greater future rewards.

States, Actions, and Rewards: The Trinity of RL

To understand reinforcement learning deeply, we must examine the three fundamental elements that define an RL problem: states, actions, and rewards. These elements form what might be called the "trinity of reinforcement learning," and mastering their meaning and relationships is essential for anyone working in this field.

States: Information About the World

A state $s$ is a representation of all information that is relevant for making decisions at a given moment. In the bicycle example, your current balance angle, velocity, direction of travel, and the terrain ahead all constitute relevant state information. In a game of chess, the positions of all pieces on the board constitute the state. In a robotic navigation task, the robot's position, orientation, sensor readings, and map information might all be part of the state.

The nature of states in reinforcement learning can vary significantly across different problems. In some cases, we have access to the true state or full state of the environment—a complete description that contains all information needed for optimal decision-making. In other cases, we must work with partial observations or observations $o$ that provide only an incomplete view of the true underlying state. When an agent receives only partial observations, the problem becomes a Partially Observable Markov Decision Process (POMDP), which is significantly more challenging.

Consider a robot navigating a building using only a camer