ACID Properties:
All reliable database systems adhere to four ACID principles:
- Atomicity - Transactions succeed completely or fail without effect
- Consistency - Every transaction moves the database between valid states
- Isolation - Concurrent transactions don't interfere with each other
- Durability - Committed transactions survive any system failure
Durability is the safety net that makes databases trustworthy. WAL is the most widely used technique to implement this critical property.
The Unbreakable Promise of Database Durability
When you withdraw money from an ATM or confirm an online purchase, you trust that these transactions will persist even if the system crashes immediately afterward. This ironclad guarantee is powered by a fundamental database mechanism called Write-Ahead Logging (WAL).
Write-Ahead Logging Explained
The core principle is beautifully simple: Record changes to a log before modifying the actual database. This approach provides:
- Crash recovery - The log can replay all committed transactions
- Performance - Sequential log writes are faster than random database updates
- Consistency - The log contains everything needed to reconstruct state
The WAL Workflow
When a transaction begins in a database system that uses a Write-Ahead Log (WAL) for durability and recovery, it follows a precise sequence to ensure atomicity and consistency, even in the event of a system crash. Initially, the transaction starts and any intended changes (such as updates, inserts, or deletes) are not directly applied to the database’s inmemory structures. Instead, these changes are first written to the WAL. This log acts as a sequential, append only file that records every modification along with essential metadata such as the transaction ID, operation type, affected rows or pages, and a timestamp or log sequence number (LSN).
After the changes are logged, the WAL entries are flushed to stable storage (typically disk) to ensure they are safely persisted. This step is crucial because it guarantees that the database can recover the changes later, even if a power failure or crash occurs before the in-memory changes are fully applied. Only after this flush completes are the changes actually applied to the inmemory data structures, such as buffer pool pages or caches. This approach decouples logical operations from physical storage and avoids data loss during mid transaction failures.
Once all changes have been safely logged and applied inmemory, the transaction commits. At this stage, a special commit record is appended to the WAL. This record marks the logical end of the transaction and is also flushed to disk. The presence of the commit record in the WAL indicates that the transaction was successful and can be safely replayed during recovery. If a crash occurs after the commit record is written but before changes are fully propagated to the main database files, the recovery process uses the WAL to redo the changes. Conversely, if no commit record exists, the changes are rolled back during recovery. To summarize the WAL workflow:
- Transaction begins
- Changes are written to the WAL (with metadata)
- WAL entries are forced to stable storage
- Changes are applied to in-memory database structures
- Transaction commits
- Commit record is added to WAL
The Architecture of a Modern WAL System
Building a production-grade WAL requires solving several key challenges:
1. Log Structure Design
- Fixed-size records vs. variable-length entries
- Binary formats for efficiency
- Metadata storage (timestamps, checksums)
2. Concurrency Management
- Handling simultaneous writes
- Reader/writer coordination
- Locking strategies
3. Durability Mechanisms
- fsync policies (synchronous vs. batched)
- Checksums for corruption detection
- Storage media considerations
4. Resource Management
- Log file rotation
- Cleanup of obsolete logs
- Space reclamation
5. Recovery Protocols
- Crash recovery procedures
- Partial transaction handling
- Checkpointing strategies
WAL in Practice: Key Implementation Patterns
Modern WAL implementations typically include:
- Segmented Logs - Breaking logs into manageable files
- Checksumming - Detecting data corruption early
- Concurrent Access - Supporting multiple readers/writers
- Buffered Writes - Balancing performance and durability
- Background Sync - Periodic persistence without blocking
+----------------+-------------------+-------------------+
| Entry Size | Log Entry | Entry Size |
| (int32) | | (next entry) |
+----------------+-------------------+-------------------+
Each Log Entry Structure (pb.WAL_Entry):
-----------------------------------------------------
| LogSequenceNumber (uint64) |
| Data ([]byte) |
| CRC (uint32) - computed as CRC32(data + LSN byte) |
-----------------------------------------------------
Log File Layout (Sequential Binary Format):
-----------------------------------------------------------
| Size_1 | Entry_1 | Size_2 | Entry_2 | Size_3 | Entry_3 | ...
| (int32)| (bytes) | (int32)| (bytes) | (int32)| (bytes) |
-----------------------------------------------------------
Example:
--------------------------------------------------------------------
| 52 | [Seq:1, Data:"foo", CRC] | 58 | [Seq:2, Data:"barbaz", CRC]..
--------------------------------------------------------------------
My implementation of WAL in Go is on github.com/rushikeshg25/wal-go follows the same design patterns.
The next time you make an online transaction, remember there's an elegant logging system working tirelessly to keep your data safe.