Chapter 1: What is Reinforcement Learning? The Magic of Learning by Doing

Imagine a small robot named Robby, with shiny blue buttons for eyes and wheels that whirr with curiosity. Robby has just been placed in a magical, colorful playground—a place where the ground is made of glowing tiles, some trees bear fruit that sparkles like candy, and friendly clouds drift low enough to touch. Robby’s only goal? To learn how to find the hidden treasure chest at the other end of the playground. But Robby doesn’t have an instruction manual. Instead, it learns by doing, by trying, by sometimes bumping into a wall and sometimes discovering a shiny coin along the way. This, dear reader, is the heart of Reinforcement Learning—a way for machines to learn from experience, just like you learned to ride a bike or tie your shoes. In this chapter, we’ll unpack that simple, powerful idea. We’ll meet Robby and its world, watch as it takes its first wobbly steps, and see how every action leads to a consequence—a smiley face or a frowny face—that shapes its future choices. By the end, you’ll not only understand the core loop of reinforcement learning but also design your own mini treasure map, complete with rewards that guide a brave explorer to victory. Ready to begin the adventure? Let’s step into the playground.

---

The Playground: Where Learning Happens

Every great story needs a setting, and in reinforcement learning, that setting is called the environment. Think of the environment as a living, breathing world with its own rules. In Robby’s case, the playground is a grid—a chessboard-like layout of squares, where each square is a state. A state is simply “where you are” and “what you see” at any moment. From Robby’s perspective, one state might be: “I’m standing on a blue tile, and there’s a sparkly tree to my north.” Another state: “I’m on a red tile, and there’s a puddle of mud to my east.” The environment is everything outside the robot: the tiles, the trees, the treasure chest, and even the weather if we want to get fancy. It’s the stage on which our learning drama unfolds.

But an environment isn’t just a static painting; it reacts. When Robby decides to move forward, the environment checks: Is there a wall? Is there a pit? Is there a friendly squirrel that gives a nut? Based on Robby’s action—the choice it makes—the environment changes. Maybe Robby moves from the blue tile to the red tile. Maybe it finds a coin. Maybe it slips on mud and slides back. The environment is like a wise, sometimes strict, teacher who always responds to your moves, but never tells you exactly what to do next.

To make this concrete, let’s picture Robby’s playground in detail. It’s a 5x5 grid, meaning 25 squares total. Some squares are normal paths (gray tiles). Some are obstacles (brown mud pits that slow you down). One square holds the treasure chest (gold and shiny). And scattered around are three reward coins: a shiny silver coin worth +10 points, a dull bronze coin worth +5, and a hidden gem worth +20 but hard to reach. Robby starts in the bottom-left corner, facing north. Its mission: reach the treasure in as few steps as possible while collecting coins. But Robby doesn’t know the layout ahead of time. It must explore, learn, and adapt.

Now, imagine you are Robby. You “see” the square you’re on, and maybe you can “sense” adjacent squares if they’re close. That sensory input—the current arrangement of tiles, coins, and walls—is your state. It’s like a snapshot of the world at this exact second. States are crucial because they tell the agent (Robby) what’s possible. If you’re on a tile surrounded by walls on three sides, your only good action might be to move east. If you’re on a tile with a coin, maybe you should pick it up. The state is the foundation for every decision.

---

The Agent: Our Curious Learner

If the environment is the world, the agent is the learner inside it. Robby is our agent. But an agent doesn’t have to be a robot; it could be a character in a video game, a self-driving car, or even a program that trades stocks. The agent’s job is simple: observe the state, choose an action, and learn from what happens next. What makes an agent special is its policy—its internal strategy for mapping states to actions. At the very beginning, Robby’s policy is random: it might spin in circles, walk into walls, or head toward the treasure by pure luck. But as it experiences more, its policy improves. It starts to remember: “When I see a silver coin to my right, moving right gives me a happy face.” That memory, that association, is learning.

Agents come in many flavors, but for now, think of Robby as a blank slate. It has no built-in map. It doesn’t know that the treasure is always in the top-right corner. All it has are its sensors (which tell it the current state) and its actuators (wheels, arms, etc., which let it perform actions). The magic of reinforcement learning is that, through trial and error, the agent builds its own map—not on paper, but in its decision-making logic.

---

Actions: The Choices That Shape Destiny

Every moment, the agent faces a set of possible actions. In Robby’s grid world, actions are simple: move north, south, east, or west; maybe also “pick up coin” if on a coin square. These are the verbs of our story. Each action is a gamble. Move north? Could lead to a coin, a wall, or open path. The agent must choose one. This choice is guided by its current policy—which might be random at first, but becomes smarter over time.

Actions are the only way an agent can change its state. You can’t think your way to the treasure; you must move your wheels. That’s why reinforcement learning is called “learning by doing.” The agent doesn’t get a lecture on grid navigation; it learns by experiencing the consequences of each step. And here’s a key insight: not all actions are equally good in every state. In a state where a wall blocks north, moving north is a bad action—it yields no progress and maybe a negative reward. In a state where a coin glints east, moving east is a good action. The agent’s journey is about discovering which actions are good in which states.

---

Rewards: The Smiley Faces and Frowny Faces

Now, why would Robby prefer one action over another? Because of rewards. After the agent takes an action, the environment responds with a number—a reward. This is the feedback signal, the “smiley face” or “frowny face” that tells the agent how it did. In our playground:

• Finding the treasure chest: +100 reward (big smiley face!).
• Picking up a silver coin: +10.
• Picking up a bronze coin: +5.
• Finding the hidden gem: +20.
• Bumping into a wall: -1 (mild frowny face).
• Stepping into mud: -5 (bigger frowny face).
• Just moving to an empty tile: 0 (neutral face).

Rewards are the currency of learning. They are the environment’s way of saying, “That was good!” or “That was not so good.” The agent’s ultimate goal is to maximize its total reward over time—not just the next step, but the whole journey. That means sometimes it might take a detour to grab a coin (+10) even if it adds a few steps, because the extra reward outweighs the cost. Or it might avoid mud pits even if the direct path goes near them, because the negative reward hurts its score.

Crucially, rewards are sparse and delayed. Robby doesn’t get a reward until it finds something or hits an obstacle. It might wander for many steps with zero reward before stumbling upon a coin. This sparsity makes learning hard—like trying to learn chess by only being told at the very end if you won or lost. But it’s also what makes reinforcement learning powerful: the agent must figure out which early actions contributed to later success. Did moving north on step 3 help lead to the treasure on step 10? The agent has to connect those dots.

---

The Fundamental Loop: Observe, Act, Learn

Put it all together, and you have the reinforcement learning loop, the heartbeat of our story:

1. Observe State: The agent sees where it is (e.