When NVMe SSDs Beat More HBM for LLM KV Cache—and When They Don't

Offloading LLM KV cache to NVMe SSDs provides substantial capacity and cost advantages for high concurrency and long contexts, but it comes with a notable latency penalty. The decision hinges on workload latency tolerance, concurrency needs, and context length.

Daron Krikorian

October 27, 2026 · 6 min read

Offloading the KV cache of a large language model to NVMe SSDs is a viable way to serve far more concurrent users and much longer contexts on the same GPU, but it buys that capacity with a real latency penalty. A PCIe 4.0 NVMe drive delivers roughly 7 GB/s of sequential throughput with per-access latency in the 100-microsecond to 1-millisecond range—about 500 times slower in bandwidth than GPU HBM, as Spheron's engineering write-up on LMCache offloading puts it. Whether that trade is worth making depends almost entirely on your workload's latency tolerance, concurrency, and context length.

The KV Cache Bottleneck in LLM Inference

The memory pressure in modern LLM serving does not come primarily from model weights. It comes from the key-value cache: the attention states a model stores for every token it has processed so far, so it can respond coherently to new prompts without recomputing the entire history. Memblaze's technical explainer likens it to the "previously on…" recap before an episode—it lets the model recall what happened without rewatching everything.

The problem is that this cache grows linearly with context length and multiplies with concurrency. Using LLaMA2-70B as an example—80 transformer layers, 64 attention heads of 128 dimensions, FP16 precision—Memblaze calculates roughly 2.6 MB of KV cache per token. That scales to about 2.6 GB at 1,000 tokens and roughly 21.3 GB at the 8,192-token maximum for a single session. Ten concurrent sessions can then exceed 200 GB in total cache footprint, far beyond a typical GPU memory budget.

This is why researchers describe KV cache management as a first-class problem for inference. As context windows scale toward millions of tokens and active sessions multiply, the aggregate cache footprint quickly exceeds GPU HBM capacity. The arithmetic is unforgiving: GPU memory must hold both model weights and the cache, and the cache is often the term that blows the budget first.

How NVMe Offloading Works as a Solution

The architectural idea is tiering. Hot cache blocks—the ones being actively read and appended to—stay in HBM. Warm and cooler blocks, such as the cached prefixes of long conversations waiting for a user's next turn, are written out to NVMe SSDs and reloaded on demand. Because inference often involves temporal gaps between conversation turns, there is a natural window in which a session's history can sit on slower storage without anyone noticing.

The economics of the tier follow from the hardware. Enterprise NVMe SSDs now offer multi-terabyte capacities per drive, so a single server can hold tens of terabytes of cache at a small fraction of the cost of an additional HBM-equipped GPU. Samsung's PM1753, for instance, is specified at up to 14.5 GB/s of sequential throughput and 3.3 million random read IOPS—bandwidth and parallelism the company positions as suited to KV cache offloading at scale.

Notably, the access pattern is favorable. A 2025 characterization from the atlarge research group on model and KV cache offloading workloads found they use large I/O sizes—around 128 KiB—and are dominated by reads. That profile is closer to sequential streaming than to the small random accesses that punish SSDs, which is why the researchers see an opportunity to optimize storage devices, including NVMe interfaces like ZNS and FDP that give more control over data placement and parallelism, for this emerging class of workload.

Performance Impact: Latency, Throughput, and I/O Bottlenecks

The catch is that the SSD tier is not automatically beneficial. Recent research on SSD-backed KV cache, published on arXiv as the Tutti project, measured vLLM with LMCache on Llama3-8B across HBM, DRAM, and SSD tiers at 64K sequence length and a 75% hit rate. DRAM stayed close to HBM performance, but SSD and GPU Direct Storage paths incurred large GPU bubbles—idle gaps where the GPU waits on data.

The same measurements show the picture can invert entirely: as LLM engines optimize inference computation, restoring KV cache from SSDs stopped being beneficial in newer vLLM versions (v0.12.0 versus v0.17.0 in their comparison) due to a severe I/O bottleneck. In other words, a faster engine makes recomputation relatively more attractive, and a slow storage path can erase the offloading advantage altogether.

Part of the blame sits in the software stack. The overhead of traversing the virtual file system, filesystem, block layer, and device driver introduces latency that hinders full utilization of the NVMe SSD—a long-standing bottleneck in the storage domain that has motivated numerous kernel-bypass solutions. Another issue is the disruption of sequential locality at the device level. So the effective bandwidth you get from an SSD tier is usually well below the drive's headline specification.

The latency penalty is not always disqualifying, though. Spheron's analysis notes that NVMe is viable for cold KV blocks that would otherwise be evicted: loading from NVMe is far cheaper than recomputing a 128K-token prefix from scratch, which takes roughly 11 seconds on an H100.

KV Cache Storage Strategy Decision Matrix

The matrix below maps the three workload dimensions that drive the choice—latency requirement, concurrency, and context length—onto the two strategies, using the performance and cost evidence above. Read each row as a decision input, not a standalone verdict; the recommendation only holds when all three rows point the same way.

KV cache storage tier selection by workload characteristic
Workload characteristic	HBM-only strategy	NVMe offloading strategy	Recommendation and trade-offs
Required latency (interactive TTFT)	Fastest path; cache reads stay in HBM	Per-access latency of 100 µs–1 ms; SSD/GDS paths measured with large GPU bubbles; can be worse than recomputation in optimized engines	Latency-critical interactive workloads: stay in HBM (or DRAM, which measured close to HBM)
Concurrency level	Capped by HBM; 10 sessions on LLaMA2-70B can exceed 200 GB	Expands capacity beyond HBM; reported on the order of 10x more users per H100	High-concurrency, latency-tolerant serving: offload cold blocks to NVMe
Context length	Cache grows linearly with tokens; ultra-long contexts exhaust HBM quickly	Warm/cooler prefixes preserved on SSD for later reuse; reload beats ~11 s recomputation of a 128K prefix	Long-context workloads with reuse: offload; one-shot short contexts: no benefit
Cost-effectiveness	Scaling means more HBM-equipped GPUs—prohibitively expensive for capacity alone	Local NVMe capacity is nearly free relative to GPUs; watch the overflow threshold where external storage ($0.20/GB/mo) spend begins	Offload wins on cost per served user until overflow or latency requirements dominate

KV cache storage tier selection by workload characteristic

Workload characteristic

HBM-only strategy

NVMe offloading strategy

Recommendation and trade-offs

Required latency (interactive TTFT)

Fastest path; cache reads stay in HBM

Per-access latency of 100 µs–1 ms; SSD/GDS paths measured with large GPU bubbles; can be worse than recomputation in optimized engines

Latency-critical interactive workloads: stay in HBM (or DRAM, which measured close to HBM)

Concurrency level

Capped by HBM; 10 sessions on LLaMA2-70B can exceed 200 GB

Expands capacity beyond HBM; reported on the order of 10x more users per H100

High-concurrency, latency-tolerant serving: offload cold blocks to NVMe

Context length

Cache grows linearly with tokens; ultra-long contexts exhaust HBM quickly

Warm/cooler prefixes preserved on SSD for later reuse; reload beats ~11 s recomputation of a 128K prefix

Long-context workloads with reuse: offload; one-shot short contexts: no benefit

Cost-effectiveness

Scaling means more HBM-equipped GPUs—prohibitively expensive for capacity alone

Local NVMe capacity is nearly free relative to GPUs; watch the overflow threshold where external storage ($0.20/GB/mo) spend begins

Offload wins on cost per served user until overflow or latency requirements dominate

Two boundary conditions deserve emphasis. First, the SSD tier's benefit is contingent on engine speed: the Tutti measurements show cache restoration becoming unbeneficial between vLLM versions as compute got faster, so re-benchmark on your actual engine version. Second, the cost advantage erodes once aggregate cache exceeds local NVMe and spills to external storage, which is where Klockwood's model shows the hourly cost gap narrowing.