Foundations of AI Agent Architecture

Foundations of AI Agent Architecture

The design of an autonomous agent begins not with code, but with a fundamental question: what architectural components must a system possess to generate behavior that appears purposeful, adaptive, and goal-directed? This question has haunted artificial intelligence research since its inception, but it has acquired new urgency in the era of large language models. The architectural patterns that emerged from classical AI—reactive systems, model-based planners, utility maximizers, and hierarchical decomposers—provide essential conceptual vocabulary. Yet the introduction of LLMs as the cognitive substrate for agents fundamentally transforms how we compose these components and conceive their interactions. Understanding the architecture of modern AI agents therefore requires us to synthesize classical design patterns with contemporary implementation realities.

This chapter establishes the conceptual and architectural foundations for everything that follows. We examine how agent systems evolved from simple stimulus-response mechanisms to complex architectures capable of multi-step reasoning, persistent memory, and tool use. We analyze the role of large language models as both enabling technology and architectural constraint. We introduce a unified framework that positions task planning, memory systems, tool interfaces, and orchestration mechanisms as interdependent components within a coherent architectural whole. Through detailed examination of seminal agent patterns and concrete production systems, we develop the design vocabulary and mental models necessary for understanding agent architecture at a principled level.

Historical Evolution: From Reactive Systems to Goal-Directed Agents

The lineage of AI agent architecture traces through several distinct paradigms, each addressing different aspects of the autonomy problem. Understanding this evolution illuminates why contemporary agent systems take their current form and reveals the trade-offs embedded in each architectural choice.

The Reactive Paradigm

Early AI agents operated on a simple principle: perceive and act. Rodney Brooks' subsumption architecture, developed in the late 1980s, exemplified this approach. Rather than constructing world models and planning from first principles, Brooks proposed layering simple behaviors—avoid obstacles, explore, wander—into hierarchical structures where higher layers could inhibit or override lower ones. The architecture was compelling because it was efficient, robust, and didn't require the computational infrastructure necessary for classical planning.

Reactive systems remain relevant today in specific niches: low-level robot control, game AI for simple entities, and certain real-time response requirements. The fundamental limitation, however, is their inability to pursue goals that require temporal coordination across multiple steps. A reactive agent cannot plan a route through a building it has never seen, cannot negotiate a multi-party conversation toward a desired outcome, and cannot decompose a complex request into meaningful sub-goals. These capabilities require something beyond direct perception-action coupling.

The lesson from reactive systems is not that they are obsolete but that they occupy a specific niche in the broader architecture of intelligent agents. Lower-level components often remain reactive—handling immediate sensory processing, managing reflex behaviors, enforcing safety constraints—while higher-level reasoning layers provide goal direction and temporal coherence.

Model-Based Agents and Internal Representation

The recognition that agents need internal representations led to model-based architectures. Rather than directly mapping perception to action, model-based agents maintain beliefs about world state, update those beliefs through perception, and select actions based on reasoning about the consequences of potential choices.

The distinction between model-based and model-free approaches has deep roots in both AI and reinforcement learning. A model-based agent explicitly represents entities, relationships, and dynamics; it can simulate outcomes before committing to actions. This capability enables look-ahead reasoning, plan repair when circumstances change, and more sophisticated goal-directed behavior. The cost is computational: maintaining and reasoning over internal models requires significant processing resources.

Contemporary agent architectures inherit this distinction. When an LLM-based agent "thinks through" a problem, it is implicitly constructing a model—a representation of the task, the available resources, and the likely consequences of various approaches. The model may not be explicitly stored as a data structure, but it exists in the activation patterns and attention mechanisms of the neural network. This has profound implications for agent architecture, as we shall explore throughout this chapter.

Utility-Based and Decision-Theoretic Foundations

Utility-based agents introduced a rigorous framework for action selection. Rather than pursuing fixed goals with hardcoded priorities, utility-based agents evaluate potential outcomes according to a utility function—an explicit representation of preferences over states. This perspective, rooted in decision theory and economics, provides a principled mechanism for handling trade-offs, uncertainty, and competing objectives.

The utility framework solves several problems that plague simpler goal-based architectures. An agent pursuing multiple goals can weigh them against each other rather than imposing arbitrary ordering. Actions can be evaluated not just on whether they achieve a goal but on how well they do so. The agent can tolerate partial goal achievement and select actions that make "good enough" progress toward multiple objectives.

Modern agent systems rarely implement explicit utility functions in the decision-theoretic sense, but the underlying reasoning patterns persist. LLM-based agents evaluate multiple potential responses, often implicitly assessing which best satisfies user intent, maintains safety constraints, and preserves consistency with prior outputs. The language model itself functions as a kind of utility estimator, guiding generation toward responses that score highly on learned preference dimensions.

Hierarchical Architectures and Temporal Abstraction

The recognition that intelligent action operates at multiple temporal scales led to hierarchical agent architectures. Rather than reasoning about primitive actions at every level of decision-making, hierarchical agents decompose tasks into abstractions: high-level goals decompose into sub-goals, sub-goals into plans, plans into actions. This temporal abstraction reduces the computational burden of planning and enables more sophisticated behavioral organization.

The Options framework in reinforcement learning and Hierarchical Task Networks (HTNs) in classical planning both address this hierarchical decomposition problem. The key insight is that useful behaviors often repeat across contexts—the "go to location" routine might be invoked whether the agent is seeking resources, fleeing danger, or pursuing any number of higher-level objectives. By encoding reusable skills at multiple abstraction levels, hierarchical architectures enable both efficiency and generalization.

Contemporary LLM-based agents exhibit implicit hierarchical structure even when not explicitly engineered. High-level reasoning about strategy and approach occurs in the model's longer-range attention patterns, while more immediate responses to user input reflect faster, more direct processing. The challenge for agent architects is to make this implicit hierarchy explicit and controllable, enabling better composition, debugging, and systematic improvement.

Large Language Models as Cognitive Substrates for Autonomous Agents

The introduction of large language models transformed ag