Storage Subsystem Architecture: NVMe vs. SAN for Enterprise PostgreSQL

Storage architecture is often the single most decisive factor in determining whether an enterprise PostgreSQL cluster reaches its full throughput potential or suffers from severe I/O bottlenecks. While modern multi-core processors and fast memory subsystems can execute complex queries in microseconds, every committed transaction must ultimately hit physical, non-volatile storage to satisfy PostgreSQL’s Write-Ahead Logging (WAL) durability guarantees.

When architecting bare-metal or dedicated virtualized infrastructure, database architects typically choose between two enterprise storage paradigms: Direct-Attached NVMe (Non-Volatile Memory Express) storage arrays and centralized Storage Area Networks (SAN).

Understanding how PostgreSQL's storage engine interacts with drive queue depths, latency profiles, and file system layouts under both paradigms is critical for delivering high-throughput, low-latency database infrastructure.

1. Architectural Comparison: Direct-Attached NVMe vs. Enterprise SAN

The fundamental difference between Direct-Attached NVMe and SAN lies in the physical proximity of storage controllers to the CPU memory bus and how I/O commands are queued.

Plaintext

+-----------------------------------------------------------------------+
|                    DIRECT-ATTACHED NVMe ARCHITECTURE                  |
|                                                                       |
|   [ CPU / RAM ] <---> PCIe Bus (Gen 4/5) <---> [ Enterprise NVMe ]    |
|   - Latency: ~10 - 50 Microseconds                                    |
|   - Command Queues: Up to 64,000 queues (64k commands/queue)          |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
|                        ENTERPRISE SAN ARCHITECTURE                    |
|                                                                       |
|   [ CPU / RAM ] <---> Host Bus Adapter (HBA) <---> Fiber Channel      |
|                              |                                        |
|                              v                                        |
|                   [ Centralized SAN Array ]                           |
|   - Latency: ~0.5 - 3 Milliseconds (Network/Switch Jitter)            |
|   - Command Queues: Managed via SCSI/FC Protocol Stacks               |
+-----------------------------------------------------------------------+

Storage Subsystem Latency Metrics

• Direct-Attached PCIe NVMe: Communicates directly over the PCIe bus without protocol translation, delivering ultra-low read/write latencies ranging from 10 to 50 microseconds.

• Fiber Channel / iSCSI SAN: Traverses host bus adapters (HBAs), storage switches, and network fabrics, introducing protocol overhead that pushes average latencies to 0.5 to 3 milliseconds.
  

2. PostgreSQL Storage Engine & Write Patterns

PostgreSQL exhibits two distinct types of physical storage I/O operations that behave differently depending on the storage subsystem:

Sequential Write-Ahead Logging (WAL)

Every INSERT, UPDATE, or DELETE transaction writes a sequential record to pg_wal and executes an fsync() system call to ensure data is physically flushed to disk before acknowledging the COMMIT to the client.

• NVMe Advantage: Near-zero write queues allow ultra-fast fsync() completions, maximizing Single-Thread Commit Throughput (TPS).

• SAN Limitation: Network latency and SAN controller flush acknowledgments can introduce write-stall delays during high-frequency transaction bursts unless backed by battery-backed NVMe/NVDIMM write caches on the SAN array.
  

Random Data Page I/O (8KB Page Reads & Shared Buffers Flush)

When PostgreSQL executes sequential scans or index lookups not cached in shared_buffers, it reads 8KB pages from data files (base/). Background checkpointer and bgwriter processes flush dirty pages back to storage periodically.

• NVMe Parallelism: Supports tens of thousands of parallel command queues, allowing PostgreSQL parallel query workers to execute asynchronous reads without queue saturation.

• SAN Centralization: Offers snapshot capabilities, block-level deduplication, and thin provisioning, but shares physical controller bandwidth across multiple enterprise hosts.

3. Storage Layout Strategy: Isolate WAL and Data Tablespaces

Regardless of whether you deploy NVMe or SAN, never place PostgreSQL Write-Ahead Logs (pg_wal) and primary data files (base) on the same physical storage volume.

# Recommended Enterprise File System Structure
/var/lib/postgresql/data/
├── base/          --> Mounted on High-Capacity NVMe / SAN Data Array
├── pg_wal/        --> Mounted on Ultra-Low Latency Isolated NVMe Volume
└── pg_stat_tmp/   --> Mounted on RAM-backed tmpfs

Strategic Storage Drive Mapping

1. pg_wal Mount: Dedicate a RAID 1 or RAID 10 array of Enterprise NVMe drives specifically for pg_wal. This isolates sequential transaction flushes from random data reads.

2. base Mount: Deploy a high-capacity NVMe RAID 10 array or high-performance SAN volume for table and index storage.

3. pg_stat_tmp Mount: Map temporary statistics collector data to a RAM disk (tmpfs) to prevent unnecessary physical disk writes:

   Bash

   mount -t tmpfs -o size=1g,noatime tmpfs /var/lib/postgresql/data/pg_stat_tmp

4. Architectural Decision Matrix: When to Choose NVMe vs. SAN

Requirement / Metric	Direct-Attached NVMe Arrays	Centralized Enterprise SAN
Transaction Throughput (TPS)	Ultra-High (>100k TPS)	Moderate to High
I/O Latency SLAs	Sub-millisecond (<50µs)	1ms - 5ms
Storage Snapshots & Clones	Manual / OS-Level (LVM, ZFS)	Native Hardware Acceleration
Disaster Recovery Replication	DB-Level (Streaming Replication)	Array-Level Synchronous Mirroring
Storage Scalability	Limited by Host Drive Bays	Virtually Unlimited Elasticity
High Availability Pattern	Patroni / CloudNativePG Clusters	Hardware-Level Controller Failover

5. Linux Storage Stack Optimization Rules

When configuring Linux storage devices for PostgreSQL, apply these kernel-level adjustments to minimize queue delays:

Set I/O Scheduler to none (NVMe)

Modern NVMe drives handle internal command queuing in hardware. Disable OS-level queue scheduling algorithms to prevent unnecessary CPU overhead:

# Set NVMe storage scheduler to 'none'
echo none > /sys/block/nvme0n1/queue/scheduler

Increase Block Device Read-Ahead (For Analytical Scans)

For large analytical reporting workloads (OLAP) reading large tables sequentially, increase the block device read-ahead size:

# Set read-ahead to 4096 sectors (2MB) for data volumes
blockdev --setra 4096 /dev/nvme0n1

Summary

• Choose Direct-Attached NVMe for mission-critical OLTP workloads, high-concurrency financial transactions, and modern High Availability architectures (such as Patroni or CloudNativePG) where local, sub-millisecond storage performance is the top priority.

• Choose Enterprise SAN if your infrastructure requires centralized storage management, multi-terabyte elasticity, hardware-accelerated block snapshots, or seamless integration into legacy enterprise virtualization management layers.