Write Amplification vs. Read Tail Latency in LSM-Tree Storage Engines on Modern NVMe

Log-Structured Merge-trees (LSM-trees) trade background compaction I/O for sequential write speed. However, on modern high-IOPS NVMe SSDs, simultaneous L0→L1 and L1→L2 compaction streams compete directly with foreground point lookups for flash channel bandwidth and SSD controller garbage collection (GC) cycles.

1. Key-Value Separation for Payloads > 4 KB

Repeatedly rewriting 16 KB values across levels L1 through L6 drives write amplification factors (WAF) above 20. By storing only keys and compact 24-byte blob pointers in the LSM tree while appending large values to immutable blob files (WiscKey / RocksDB BlobDB), compaction bandwidth drops by over 75%.

2. Rate-Limiting Compaction Writes with Token Buckets

options.rate_limiter.reset(NewGenericRateLimiter(
    /* rate_bytes_per_sec = */ 250ULL * 1024 * 1024,
    /* refill_period_us   = */ 100 * 1000,
    /* fairness           = */ 10,
    RateLimiter::Mode::kWritesOnly,
    /* auto_tuned         = */ true
));

Pacing background flushes and compactions at sub-100ms refill intervals prevents flash translation layer (FTL) buffer exhaustion and keeps foreground p99 point-lookup latency under 400 µs.