The Evolution and Core Architecture of Database Systems

The journey of database systems from hierarchical models to today’s distributed, globally replicated data stores is not merely a timeline of technological adoption but a narrative of persistent tension between abstract data modeling and the unforgiving physics of storage hardware. Each paradigm shift—from the pointer-chasing navigational models of the 1960s to the declarative elegance of SQL, and from the single-node, ACID-compliant monoliths to the eventually consistent, horizontally scaled NoSQL stores—was driven by a specific confluence of application demands, hardware capabilities, and theoretical insights. Understanding this evolution is not an academic exercise; it is a prerequisite for diagnosing the performance, scalability, and reliability characteristics of any modern data system. The canonical architecture of a contemporary Database Management System (DBMS), comprising the storage engine, query processor, transaction manager, and execution engine, emerged as a direct response to the challenges identified across this historical arc. This chapter will trace that development, defining each architectural component and the core design philosophies that shape them, particularly the fundamental trade-off between log-structured and page-oriented storage layouts. We will also examine the critical, often overlooked, systems-level concerns of process and thread models, memory architecture, and the buffer pool—the linchpin that connects logical data access to physical storage.

1.1 The Pre-Relational Era: Navigation and Hierarchies

Before the relational model crystallized as the dominant paradigm, the 1960s and early 1970s were defined by *navigational* or *non-relational* databases. These systems were designed to mirror the physical storage structures of the time, primarily magnetic tapes and early disks with high seek costs. Data was accessed by following pointers or paths, a necessity given the lack of a high-level, set-oriented query language. Two major models emerged: the hierarchical model and the network model.

The hierarchical model, epitomized by IBM's Information Management System (IMS), organized data in a strict tree-like structure. Each record (segment) had a single parent, enabling efficient one-to-many relationships (e.g., a customer to many orders). Access was performed via a procedural API that involved *get unique* (direct access by key) and *get next* (sequential traversal within a hierarchy). While extremely fast for its intended workloads—large-scale, batch-oriented transaction processing in industries like banking and airline reservations—its rigidity was a profound limitation. Many-to-many relationships required awkward duplication of data or the creation of artificial hierarchies, and ad-hoc querying was virtually impossible. The physical data layout was tightly coupled with the logical view; changing the hierarchy often required reorganizing the physical storage.

The network model, formalized by the Conference on Data Systems Languages (CODASYL) and implemented in systems like IDMS and Total, generalized the hierarchy by allowing a record to have multiple parents, thus supporting many-to-many relationships directly. Data was organized into *sets*, and applications navigated these sets using a *currency indicator* (a pointer to the current record) and commands like `GET`, `FIND`, and `OBTAIN`. This provided greater modeling flexibility but at a significant cost: application code became heavily laden with complex navigation logic, tightly binding the program to the exact physical database structure. Any schema change rippled through the application layer. Both models suffered from a lack of data independence; the logical view was inextricably linked to the physical storage path.

Case Study: IBM IMS (1968) IMS remains in production today, a testament to its engineering for reliability and throughput on early hardware. Its core design—a hierarchical file structure stored on direct-access storage devices (DASD)—used a technique called *HDAM* (Hierarchical Direct Access Method) or *HSAM* (Hierarchical Sequential Access Method). HDAM, for instance, used a hashing algorithm on the root segment's key to calculate a Relative Block Address (RBA), providing near-direct access. For non-root segments, the system followed physical pointers (Direct Random Access, DRA) stored within the parent record. This pointer-chasing was efficient on the slow disks of the era because it minimized seeks within a tightly coupled hierarchy but made ad-hoc queries across hierarchies impossible. The lesson here is that architecture is a direct response to hardware constraints: when I/O is catastrophically slow and expensive, minimizing random access by encoding relationships physically is a rational, if inflexible, choice.

1.2 The Relational Revolution and the Birth of the DBMS

The seminal 1970 paper "A Relational Model of Data for Large Shared Data Banks" by Edgar F. Codd at IBM's San Jose Research Laboratory proposed a radical departure: data should be presented to users and applications as *relations* (tables), with access achieved via a non-procedural, *declarative* query language. The physical storage details would be hidden behind a *logical* schema, achieved through what Codd called *data independence*. This separation of concerns is the foundational principle of the modern DBMS.

The relational model’s power lies in its simplicity and mathematical foundation (first-order predicate logic). A relation is an unordered set of tuples, each sharing the same attributes. Queries express *what* data is desired, not *how* to retrieve it. This abstraction enabled the development of the query processor, a component responsible for translating a high-level statement (e.g., SQL) into an efficient execution plan. The shift also necessitated a new storage organization. Since relationships were no longer encoded by physical pointers but by *data values* (foreign keys), the system needed a mechanism to find related tuples quickly. This led to the widespread adoption of indexes, most notably the B-tree (and its variants like B⁺-tree), which provided ordered, logarithmic-time access on any attribute.

The first generation of relational database management systems (RDBMS) in the late 1970s and early 1980s—such as IBM's System R, Ingres (UC Berkeley), and Oracle (then Relational Software Inc.)—began to define the core architectural components we recognize today. System R, in particular, was a research prototype that established the blueprint: a *parser* to build a query tree, a *optimizer* (based on relational algebra equivalence rules and cost estimation) to choose the lowest-cost plan, and an *executor* to iterate through the plan, calling a *storage manager* to fetch data pages. The storage engine became responsible for mapping logical relations and indexes to physical pages on disk, managing a buffer pool (or cache) in main memory to avoid unnecessary I/O, and ensuring atomicity and durability via a write-ahead log (WAL).

This era cemented the canonical four-component architecture: 1. Storage Engine: Manages on-disk data layout (pages, extents), indexes, and the buffer pool. It provides a page-oriented interface to higher layers. 2. Query Processor: Includes a parser, optimizer, and executor. It transforms SQL into an executable plan of relational operators (scan, join, aggregate). 3. Transaction Manager: Ensures ACID properties (Atomicity, Consistency, Isolation, Durability) through concurrency control (e.g., locking, Multi-Version Concurrency Control - MVCC) and recovery mechanisms (logging, checkpoints). 4. Execution Engine: The runtime that processes the operator tree, often using a "pull" model (火山模型) where each operator requests tuples from its child.

Design Philosophy: Page-Oriented Storage The relational model, combined with the page-oriented nature of disks (which read and write