The Three Pillars of Robotics: A Robot’s First Day

The cold fluorescent hum of the warehouse’s high-bay lighting cuts through the pre-dawn quiet as the 200-kilogram service robot is lowered from its delivery pallet onto the smooth epoxy floor. Its six onboard laser scanners spin to life with a faint whir, casting invisible planes of infrared light across the rows of 3-meter-tall storage racks stacked with cardboard boxes labeled with barcodes and destination codes. The operations supervisor taps a command into her tablet: “Deliver batch #A-729 from rack 12, zone C to the sorting station at the east end of the building. No pre-built map, no pre-defined route—figure it out.” For this robot, and for every autonomous system that will follow it across warehouses, hospitals, city streets, and eventually our homes, this first unplanned task is the moment its three core capabilities—perception, planning, control—are tested for the very first time.

In the first 0.2 seconds after startup, the robot’s perception module goes to work. Raw data streams in from eight different sensors: two 16-channel LiDARs spinning at 10Hz, a 30fps RGB-D camera, a 200Hz IMU (inertial measurement unit) with accelerometer and gyroscope, four wheel encoders sampling at 1kHz, and a UWB (ultra-wideband) anchor for rough absolute positioning. The first LiDAR scan returns 120,000 3D points, most of which map to the hard surfaces of storage racks, the concrete floor, and the metal support pillars lining the aisles. The IMU’s accelerometer confirms the robot is sitting on a flat surface with no tilt, while the wheel encoders report zero velocity, matching the UWB’s initial position estimate of $(12.3, 4.7, 0.0)$ in the warehouse’s local coordinate frame. Within 3 seconds, the SLAM (simultaneous localization and mapping) algorithm fuses these data streams to output its first high-confidence pose estimate: the robot is 2.1 meters west of rack 12, zone C, with an orientation of 0 radians (facing east). It also builds its first partial map of the surrounding 10-meter radius, marking static surfaces (racks, floor, pillars) as occupied space, and flagging a moving object 3.2 meters north of its current position as a dynamic obstacle—later identified as a cleaning staff member pushing a waste cart moving at 0.6 m/s west to east.

This first set of outputs from the perception module is the foundation for every decision the robot will make next. Perception is not simply “collecting sensor data”: it is the process of transforming noisy, raw, high-dimensional sensor measurements into a structured, actionable understanding of both the environment and the robot’s own state within it. For mobile robots, this almost always involves solving two tightly coupled problems: localization (estimating the robot’s own pose relative to a map or the environment) and mapping (building a model of the environment from sensor data). For manipulation robots, perception extends to recognizing objects, estimating their pose, and identifying graspable surfaces. The math behind perception draws heavily from linear algebra, probability theory, and geometric estimation—foundations we will unpack in depth in Chapters 2 through 4, and apply to SLAM and environmental perception in Chapters 5 through 7.

With a confirmed pose and a partial map of its surroundings, the robot’s planning module takes over. The high-level task—deliver a box from rack 12 to the east sorting station—is first decomposed into a sequence of sub-goals: navigate to rack 12, align with the shelf, grasp the target box, navigate to the sorting station, place the box on the conveyor. For the first sub-goal (navigate to rack 12), the global planner runs an A* search on the partially built occupancy grid map, generating a collision-free path that avoids known static obstacles: the path requires the robot to move east 2.1 meters to reach the rack, with a total path length of 2.3 meters accounting for a small detour around a misplaced pallet. But before the robot can start moving, the local planner overrides the first 0.8 meters of the global path: the cleaning staff’s waste cart is predicted to cross the planned route in 2.2 seconds, and the robot’s maximum speed is 1.2 m/s, so a collision is unavoidable if it follows the original path. The local planner generates a new short-term trajectory that first moves 1 meter north to clear the cart’s projected path, then turns east to resume the global route to the rack.

Planning for autonomous systems is fundamentally different from the pre-programmed trajectories used in early industrial robots. The first industrial robot, Unimate, deployed at a General Motors die-casting plant in 1961, had no perception and no online planning: its entire “decision-making” process was a pre-recorded sequence of joint angles, repeated endlessly to lift hot metal parts from production lines and place them in cooling tanks. It worked only because its environment was completely controlled: all obstacles were fixed, all parts were placed in exact predetermined positions, and no unexpected events would occur. Modern mobile robots and autonomous vehicles operate in unstructured, dynamic environments where obstacles move, layouts change, and unexpected events are the norm, not the exception. Planning algorithms must therefore be online, reactive, and able to reason about uncertainty: they must generate paths that are not just collision-free for the current moment, but safe over the entire time horizon of the robot’s movement. We will dive deep into the math and implementation of global and local planning, task planning, and multi-robot planning in Chapters 8 through 10.

The adjusted trajectory from the planning module is a sequence of timed pose targets: at $t=0s$, pose $(12.3, 4.7, 0.0)$; at $t=1s$, pose $(13.3, 4.7, 0.0)$; at $t=2s$, pose $(14.1, 5.2, 0.3)$; and so on, with a target arrival at rack 12 at $t=2.8s$. The control module’s job is to make the robot’s actual pose match these targets as closely as possible, despite inevitable sources of error: wheel slip on the smooth epoxy floor, small unevenness in the ground, sensor noise in pose estimates, and small delays in actuator response. The low-level wheel controller first converts each pose target into target velocities for the left and right drive wheels: to move east at 1 m/s with no turn, both wheels spin at 1 m/s; to turn 0.3 radians east of north, the left wheel slows to 0.7 m/s while the right wheel speeds up to 1.3 m/s. A PID (proportional-integral-derivative) feedback loop runs at 1kHz, comparing the robot’s actual pose (from SLAM and wheel odometry) to the target pose, calculating the error, and adjusting the wheel velocities to minimize that error. When the robot hits a small patch of oil on the floor 1 second into movement, causing 10% wheel slip and a 0.1-meter lateral error from the planned path, the controller increases the right wheel’s velocity by 15% within 10 milliseconds to correct the drift, bringing the robot back to the planned trajectory within 0.3 seconds.

Control is the bridge between the abstract plans generated by the planning module and the physical movement of the robot in the real world. Its core purpose is to reject uncertainty: all real-world systems have unmodeled dynamics, sensor noise, and external disturbances, and control algorithms are designed to compensate for these imperfections to achieve accurate, stable movement. For early industrial robots, control was often simple open-loop control, as their environments were predictable and their tasks were repetitive. For modern autonomous systems, control must account for complex dynamics (the mass, inertia, and friction of the robot’s body), time delays in sensor and actuator response, and unexpected disturbances like wind for aerial robots or uneven terrain for ground robots. We will cover the math of kinematic and dynamic modeling, feedback control, and trajectory optimization for a wide range of robot platforms in Chapters 11 through 13.

The thre