The Reinforcement Learning Paradigm: Agents, Environments, and Goals

Reinforcement learning (RL) represents a fundamental paradigm in machine learning where an autonomous agent learns to make sequential decisions by interacting with an environment to maximize cumulative numerical rewards. Unlike supervised learning, which relies on labeled datasets providing explicit input-output pairs, or unsupervised learning, which seeks hidden structures in unlabeled data, reinforcement learning operates through a continuous feedback loop of trial and error. The agent perceives the current state of its environment, selects an action according to some behavioral strategy (policy), receives a reward signal indicating the immediate desirability of the outcome, and transitions to a new state. This interaction cycle repeats indefinitely or until a terminal condition is met. The core challenge lies in learning a policy that optimizes long-term cumulative reward, which requires balancing the exploitation of known rewarding actions against the exploration of uncertain but potentially more rewarding alternatives—a dilemma known as the exploration-exploitation trade-off.

The formalization of this interaction process provides the conceptual foundation for the entire field. At its heart, reinforcement learning addresses the problem of sequential decision making under uncertainty, where the consequences of actions extend beyond immediate effects to influence future states and available choices. Consider a robot learning to walk: each step not only determines its current position but also affects its balance and possibilities for subsequent movements. Similarly, in financial trading, a buy decision not only yields immediate profit or loss but also alters portfolio composition and future market positions. These scenarios share a common structure—a decision-maker (agent) operating within a dynamic system (environment) whose state evolves based on the agent's interventions, with the ultimate goal of achieving the best possible outcome over an extended horizon.

The mathematical abstraction that captures this structure is the Markov Decision Process (MDP), which we will formally introduce in Chapter 2. However, before diving into the formalism, it is essential to develop an intuitive understanding of the paradigm's key components and their relationships. The agent serves as the learner and decision-maker; it could be a software algorithm controlling a game character, a physical robot navigating a room, or a recommendation system selecting content for users. The environment encompasses everything external to the agent—the game rules, physical laws, market dynamics, or user preferences—that responds to the agent's actions. The state represents the environment's current configuration, which might be fully observable (as in most board games) or partially observable (as when a robot only has sensor readings rather than complete knowledge of its position). Actions constitute the set of possible interventions the agent can undertake, ranging from discrete choices like moving left/right/up/down to continuous control signals like motor torques. The reward function assigns a scalar value to each state transition, providing the sole guidance signal for learning—positive rewards reinforce desirable behaviors, negative rewards (penalties) discourage undesirable ones.

What distinguishes reinforcement learning from other machine learning paradigms is this reliance on evaluative rather than instructive feedback. In supervised learning, the learning signal explicitly states the correct output for each input (e.g., "this image contains a cat"). In reinforcement learning, the reward signal merely indicates whether the current outcome is good or bad relative to the objective, without specifying what the optimal action would have been. This creates a credit assignment problem: when a reward is received after a sequence of actions, determining which actions contributed most to the outcome becomes challenging. For instance, in a chess game, the final checkmate provides a positive reward, but which of the preceding 50 moves were truly critical? The agent must learn to propagate credit backward through time, attributing value to earlier decisions based on their ultimate consequences.

The objective of maximizing cumulative reward introduces several conceptual layers. The most straightforward is the finite-horizon sum of rewards: $R = r_1 + r_2 + \dots + r_T$, where $T$ is the episode length. However, many problems involve continuing tasks without natural endpoints, leading to the use of discounted return: $G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}$, where $\gamma \in [0,1]$ is a discount factor that weights immediate rewards more heavily than future ones. This discount factor serves both mathematical purposes (ensuring convergence of infinite sums) and practical ones (modeling uncertainty about the future or time preferences). The choice of $\gamma$ fundamentally shapes the agent's behavioral orientation—a value near 1 encourages long-term planning, while a smaller $\gamma$ makes the agent more myopic.

The policy, denoted $\pi(a|s)$, formalizes the agent's behavior as a mapping from states to action probabilities (for stochastic policies) or deterministic selections (for deterministic policies, $a = \pi(s)$). Learning in RL essentially means improving this policy to increase expected cumulative reward. Two primary approaches emerge: policy-based methods, which directly optimize $\pi$ by adjusting its parameters (e.g., via gradient ascent on expected return), and value-based methods, which first learn to estimate the expected return from states or state-action pairs (value functions) and then derive a policy from these values (e.g., by always choosing the action with highest estimated value). Actor-critic methods combine both approaches, maintaining separate policy (actor) and value (critic) components that are trained jointly.

The exploration-exploitation dilemma arises because maximizing cumulative reward requires both using known good actions (exploitation) and trying new actions that might be better (exploitation). A purely exploitative agent might never discover superior strategies, while a purely exploratory agent would not capitalize on learned knowledge. This tension manifests in various algorithmic solutions: $\epsilon$-greedy methods choose the best-known action with probability $1-\epsilon$ and a random action with probability $\epsilon$; softmax (Boltzmann) exploration assigns probabilities proportional to estimated action values; and more sophisticated approaches like Upper Confidence Bound (UCB) algorithms explicitly balance estimated value with uncertainty. The effectiveness of exploration strategies significantly impacts learning speed and final performance, making this a central research theme.

To concretize these abstract concepts, consider three canonical examples that have historically driven RL research. First, game playing—from simple tic-tac-toe to complex Go—provides discrete, fully observable environments with clear reward structures (win/loss/draw). The game of backgammon served as an early testbed for temporal difference learning (TD(λ)), while the 2016 AlphaGo victory over Lee Sedol demonstrated deep reinforcement learning's potential in high-complexity domains. Second, robotic control tasks, such as a simulated bipedal walker or a physical robotic arm, involve continuous action spaces, partial observability (due to sensor noise and latency), and real-world physics constraints. These applications highlight RL's ability to learn complex motor skills without explicit programming. Third, resource management problems—like inventory control, network routing, or financial portfolio optimization—feature stochastic transitions and delayed rewards, requiring agents to learn policies that balance immediate costs against future benefits.

These examples also illustrate RL's position within the broader machine learning landscape. Unlike supervised