Why Storage Is Becoming the Bottleneck for AI Inference

An inference request arrives. The model is loaded. The GPU is ready. And then: the system stalls. Not because compute is absent. Because context is.

This is the failure mode that doesn't show up clearly in GPU utilisation dashboards. It shows up in latency spikes, throughput collapse at scale and inference pipelines that seem fine until they aren't. The common diagnosis is "we need more GPUs." The real diagnosis is usually "we need a better memory hierarchy."

At NVIDIA's recent partner sessions, the message was direct: inference context is the new bottleneck. Storage must be rearchitected. Not optimised. Rearchitected. That distinction matters.

At Hyperstack, we've been building storage infrastructure for GPU-intensive AI workloads long enough to recognise this problem when we see it. Our storage portfolio — NVMe scratch, Shared Storage Volumes, Secure Object Storage, and parallel filesystem — maps directly onto NVIDIA's four-tier hierarchy

In our latest blog, we discuss why the hierarchy matters and what goes wrong when the architecture is treated as an afterthought.

The KV Cache Problem Nobody's Talking About

Key-value (KV) cache sits at the centre of transformer inference. Every token in a conversation's context window gets encoded into a KV pair that the model needs to attend to when generating the next token. Short conversations mean a small cache. Long-context agentic workflows with multi-turn history, retrieved documents, and shared session state? The cache grows fast.

GPU HBM (high-bandwidth memory) is where you want the KV cache to live. Access is fast, latency is low, and inference throughput holds. But HBM is finite: typically 80GB per H100. A single long-context request can consume gigabytes of KV cache. Run multiple concurrent sessions, and you're rationing HBM between active requests, model weights, activations, and everything else the GPU needs to hold.

When HBM fills up, context spills. Where it spills, and how fast you can retrieve it, determines whether your inference pipeline is fast, acceptable or broken.

This is not a future concern. It's a present operational constraint for any team running production inference at scale.

NVIDIA's Four-Tier Context Memory Hierarchy

NVIDIA has formalised the solution as a four-tier hierarchy for context memory management. Each tier serves a specific function based on access speed, capacity, and cost:

• Tier 1: Active KV (GPU HBM). The hottest context. Active sessions, tokens being attended to right now. This is where latency is measured in microseconds. Nothing should compete with active KV for this space.

• Tier 2: Staging/Spillover KV (system memory / CPU DRAM). Context that's been recently active but isn't being attended to this instant. Lower bandwidth than HBM, but still fast enough to pull back without killing throughput. This tier absorbs burst load and prevents HBM eviction under pressure.

• Tier 3: Warm KV Reuse (local storage / NVMe). Context that's worth keeping but not worth holding in memory. Think: sessions from earlier in the day that might resume, or frequent retrieval patterns you can predict. Local NVMe gives you high-throughput read access at a fraction of the cost of DRAM. Getting this tier right is where most teams leave performance on the table.

• Tier 4: Cold/Shared KV (networked storage/object storage). Cold or shared KV sits on networked or object storage and acts as the long-term memory layer for AI systems. It stores structured data — what NVIDIA describes as the “ground truth” layer for agentic AI — including datasets, model artefacts, long-session state, and retrieval corpora. Access speeds are slower than local memory tiers but storage capacity is effectively unlimited and can be shared across nodes and persisted between sessions.

Why Traditional Storage Architectures Fail at Inference Scale

The instinct is to scale storage the same way you'd scale any other resource — more capacity, more nodes, more replicas. NVIDIA's explicit warning from the session was direct: scaling traditional storage increases cost and power at inference scale. The problem isn't capacity. It's architecture.

Most storage systems weren't designed for the access patterns that inference generates. Training reads data sequentially in large batches. Inference reads context randomly, at low latency, with high concurrency. A storage layer built for training throughput will have the wrong characteristics for inference access patterns.

There's also the cross-node problem. In a multi-GPU inference cluster, context can't stay local to one node. It needs to be accessible wherever the request lands. That means shared context requires networked storage with low enough latency that retrieval doesn't become the bottleneck. Object storage optimised for throughput doesn't automatically mean object storage optimised for low-latency random read at scale.

The practical failure mode is when teams optimise GPU and network performance, then discover that storage access patterns are causing stalls, inconsistent latency, and throughput that doesn't scale linearly with GPU count. By that point, they're re-architecting under production pressure.

Mapping the Hierarchy to a Real Storage Architecture

The four-tier hierarchy only works if each tier is implemented with storage that matches its performance and access requirements. Hyperstack's storage portfolio maps onto each layer directly. Here's how:

Tier 1 (Active KV) maps to GPU HBM management

This is a scheduling and orchestration problem, not a storage product decision. The question is how efficiently your cluster can evict and reload KV pairs as requests come and go. Dedicated single-tenant GPU environments have an advantage here: no noisy-neighbour effects competing for HBM bandwidth.

Tier 2 (Staging KV) maps to CPU DRAM and system memory

The key requirement is bandwidth and low-latency transfer back to GPU memory. NVLink and PCIe bandwidth constraints apply here. Your cluster architecture determines how effective this tier can be.

Tier 3 (Warm KV) maps to local NVMe scratch storage

High-throughput, low-latency local NVMe is the right fit for a context that needs to be retrievable quickly but isn't hot enough for memory. This is also where checkpoint writes land during training runs. The same storage tier serves both use cases. Hyperstack's NVMe scratch is provisioned for exactly this pattern: high-throughput staging, fast checkpoint writes, and KV spillover that needs to come back quickly. The requirement is sustained sequential throughput and random read latency measured in microseconds, not milliseconds.

Tier 4 (Cold/Shared KV) maps to object storage and shared storage volumes

This tier handles the persistent, cross-node, durable layer: datasets, long-horizon session state, model artefacts, retrieval corpora, and audit logs in regulated environments. The requirement is durability, capacity, and the ability to serve reads across multiple compute nodes without becoming a bottleneck at the network layer. Hyperstack's Secure Object Storage (with server-side encryption and durable retention) maps directly here. For workloads requiring shared, high-throughput file access across nodes (distributed training, parallel retrieval), Hyperstack's parallel filesystem is the right choice over object storage. Shared Storage Volumes (SSVs) sit between these two: persistent block storage that retains datasets, checkpoints, and artefacts across node restarts and workload migrations, covering the cases where you need more than scratch but less than a full parallel filesystem.

Each tier requires different engineering decisions. Getting one tier right but neglecting another is how you end up with a cluster that benchmarks well in isolation but degrades under production load.

Why Single-Tenant Matters for Context Storage

For teams running inference in regulated industries (finance, healthcare, defence), the storage architecture problem has an additional constraint layer that some public clouds  can't resolve.

Context is data. In a regulated deployment, that means access controls, auditability, data residency and in some cases, legal jurisdiction constraints on where context can persist. A KV cache that spills to shared networked storage in a multi-tenant environment isn't just a performance risk. It's a compliance risk. InfoSec and audit teams will want to know exactly where inference context lives, who can access it, and what the retention and deletion behaviours are.

This is where Hyperstack's Secure Private Cloud changes the conversation. In a shared cloud environment, you don't control where Tier 3 or Tier 4 context lives relative to other tenants' data. The storage may be logically separated but the physical infrastructure is shared. Secure Private Cloud is a single-tenant, dedicated deployment: segregated infrastructure, access boundaries defined as part of the build and operational logs designed for regulated environments. Every storage tier in the hierarchy sits inside an environment InfoSec and audit teams can actually examine.

Region and data residency are addressed at the deployment level. Secure Private Cloud can be deployed in-country or in a specific jurisdiction where regulatory requirements demand it, without forcing a redesign of the workload or the storage architecture. The storage tiers stay intact. The compliance constraints get met.

What Good Storage Architecture Looks Like

The teams that get this right don't treat storage as a later decision. They design it alongside GPU, network and orchestration choices. The performance of the whole system is determined by the weakest tier in the hierarchy.

To give an idea, that means:

• Sizing NVMe scratch for peak KV spillover load, not average load. Average load will look fine in testing. Production spikes are where the gaps appear.
• Separating scratch from persistent volumes. Ephemeral NVMe scratch for hot-path operations; Hyperstack SSVs for state that needs to survive node restarts or workload migration.
• Treating object storage as the shared context layer, not just data backup. Retrieval corpora, session state and model artefacts belong here. Only if your object storage has the latency characteristics and access patterns required for inference retrieval, not just training data ingestion.
• Validating the full stack before scaling. GPU throughput, network bandwidth, and storage IOPS need to be tested together under inference load conditions. A system that performs well on GPU benchmarks alone can still fail at the storage tier when real workloads hit.

Build the Storage Architecture Before You Need It

Storage architecture for inference is almost always designed too late. Teams procure GPUs, configure networking, choose an orchestration layer and then ask what storage is available. By that point, the constraints are already set.

NVIDIA's framing of the four-tier hierarchy is useful precisely because it forces the conversation earlier. Context memory management is not an infrastructure detail. It's a first-order architectural decision that determines what throughput your inference cluster can actually deliver, at what latency, and at what cost per token.

The teams that think about storage when they're designing their cluster, not when they're debugging production latency spikes, are the ones that don't have to re-architect under pressure.

Hyperstack's storage including NVMe scratch, SSVs, Secure Object Storage and parallel filesystem, covers every tier of that hierarchy. Deployed inside a Secure Private Cloud, those tiers run on dedicated, single-tenant infrastructure with the governance controls, auditability and data residency options that regulated AI workloads require.

If you're designing an inference cluster in a regulated sector and want to talk through how to architect the storage layer correctly from the start, we're ready to have that conversation.