The Autonomous Vehicle Stack: An Architectural Overview

Designing an autonomous vehicle (AV) system is arguably one of the most complex systems engineering challenges of the 21st century. It demands the harmonious integration of artificial intelligence, real-time control theory, robust software engineering, and safety-critical hardware into a cohesive whole capable of operating reliably in the open, unstructured world. At the heart of this endeavor lies the system architecture—the definitive blueprint that specifies the structure, interactions, and constraints of every component, from raw sensor pixels to final actuator commands. This chapter establishes the canonical software and hardware architecture for a production-intent autonomous driving system, providing the foundational framework upon which all subsequent technical deep dives will be built. We will define the four core functional modules—Perception, Prediction, Planning, and Control—delineate their interfaces, and examine the paramount system-level constraints of real-time performance, determinism, fault tolerance, and computational budgets. Furthermore, we will analyze the critical trade-offs between centralized, decentralized, and federated computing topologies, culminating in the presentation of a reference pipeline that will serve as a consistent conceptual model throughout this book.

The Core Functional Modules: Perception, Prediction, Planning, Control

The functional decomposition of an autonomous driving system into four primary modules is nearly universal across industry and academia. This division, often called the "sense-plan-act" paradigm extended with a prediction layer, creates a logical separation of concerns that simplifies development, testing, and certification. Each module has a well-defined responsibility, a set of inputs and outputs, and operates within specific latency and accuracy boundaries. The integrity of the entire system hinges on the precise specification and reliable implementation of these interfaces.

Perception: From Raw Sensors to Environmental Representation

The perception module is the system's primary interface with the physical world. Its fundamental responsibility is to answer the question: "What is around me?" It transforms raw, noisy, and asynchronous data from a suite of sensors—cameras, laser radars (lidar), radio detection and ranging (radar), and inertial measurement units (IMU)—into a consistent, time-synchronized, and semantically rich representation of the vehicle's surroundings. This representation, often called the "world model" or "environment model," must include the position, velocity, geometry, classification, and semantic intent (e.g., traffic light state, lane markings) of all relevant objects and features.

Internal Sub-Modules and Data Flow: A modern perception pipeline is itself a multi-stage architecture. 1. Sensor Preprocessing & Calibration: Raw sensor data undergoes correction for distortions (e.g., lens distortion in cameras, beam divergence in lidar), noise filtering, and time synchronization. Multi-sensor extrinsic (relative pose) and intrinsic (internal parameters) calibration is performed offline and validated online to ensure spatial alignment of all data streams. 2. Object Detection: This stage identifies instances of relevant classes (vehicles, pedestrians, cyclists, traffic signs) in the sensor data. For cameras, this typically involves convolutional neural networks (CNNs) like YOLO or Faster R-CNN operating on 2D images. For lidar, point cloud-based detectors like PointPillars or VoxelNet process 3D points directly. Radar detection focuses on point targets with associated radial velocity. 3. Object Tracking: Detection outputs from successive time steps are associated into persistent tracks. Algorithms like Kalman Filters (linear) or Unscented Kalman Filters (non-linear) estimate the state (position, velocity, acceleration) of each tracked object, fusing multi-sensor measurements and predicting the object's state between detections. Data association—matching new detections to existing tracks—is a critical challenge, often solved with Hungarian algorithm or joint probabilistic data association (JPDA). 4. Semantic Segmentation & Free Space Estimation: Beyond bounding boxes, the system requires a dense understanding of the scene. CNNs like DeepLab or SegNet assign a class label (road, sidewalk, building, sky) to every pixel in a camera image. Lidar and radar contribute to estimating the drivable area, often by voxelizing the 3D space and classifying each voxel. 5. Sensor Fusion: This is the core orchestration layer. Fusion can occur at different levels:

• **Low-Level (Early) Fusion:** Raw or minimally processed data from multiple sensors are combined before detection (e.g., projecting lidar points onto the image plane and using both modalities as input to a single network). This maximizes information use but is computationally heavy and requires precise calibration.
• **High-Level (Late) Fusion:** Each sensor modality runs a complete detection pipeline independently, and the resulting object lists are fused. This is more modular and robust to single-sensor failure but can lose valuable correlations.
• **Intermediate (Feature-Level) Fusion:** Features extracted from different sensors are combined before the final detection/classification stage. This is a common compromise, exemplified by architectures that project all sensor data into a unified Birds-Eye View (BEV) representation—a 2.5D grid from a top-down perspective—which simplifies fusion and subsequent planning.

The output of the perception module is a unified, time-stamped list of *Tracked Objects* and a *Local Map* (e.g., lane boundaries, drivable area, traffic signs). Each tracked object includes a unique ID, 3D position (x, y, z), 3D velocity (vx, vy, vz), 3D bounding box dimensions (length, width, height), classification confidence, and a predicted short-term trajectory (e.g., next 1-2 seconds).

```mermaid graph TD A[Raw Sensor Data<br/>Camera, Lidar, Radar, IMU] --> B[Preprocessing & Calibration]; B --> C[Object Detection]; B --> D[Semantic Segmentation]; C --> E[Object Tracking]; D --> F[Free Space Estimation]; E --> G[Sensor Fusion<br/>BEV Representation]; F --> G; G --> H[Unified World Model<br/>Tracked Objects + Local Map]; ```

Prediction: Anticipating the Future States of Dynamic Agents

While perception describes the present state, the prediction module projects the future behavior of all dynamic agents in the scene—other vehicles, pedestrians, cyclists. Its core question is: "What will they do next?" This is a profoundly difficult problem because human behavior is stochastic, interactive, and influenced by the AV's own intended actions. Prediction must output a probabilistic distribution over future trajectories for each agent, often for a horizon of 3-8 seconds.

Key Approaches and Challenges:

• **Physics-Based Models:** Simple kinematic models (constant velocity, constant acceleration) are fast and require no learning but fail to capture intentional behavior (e.g., a car preparing to turn).
• **Pattern-Based Models:** These use large datasets of human driving trajectories to learn common behaviors. They can be implemented with generative models like Gaussian Mixture Models (GMMs) or, more powerfully, deep learning models.
• **Interaction-Aware Models:** The state-of-the-art recognizes that agents do not move in isolation. Models use graph neural networks (GNNs) or social pooling mechanisms (e.g., Social-LSTM) to encode interactions between all agents. The predicted trajectory of one agent depends on the predicted trajectories of others, creating a joint distribution.
• **Context-Aware Models:** Prediction must incorporate static context (road layout, traffic signs, lane connectivity) and dynamic context (traffic lights, presence of emergency vehicles). This context is typically provided by the percepti