The Race to Run AI at Scale Is Now a Race for More Tokens per Watt

For many years, the AI infrastructure conversation revolved around one question: how many GPUs do you have? That question is now out of date. The one replacing it is how many tokens your server produces per watt. It reflects where the money in AI now actually gets made.

Training builds a model once. Inference runs it millions of times a day for every user, every query, every action an agent takes. As enterprises move out of pilot programmes and into production, inference is where the AI budget lands and where the AI revenue gets generated.

This change shows up in how the market is now talking about infrastructure. A year ago, most procurement conversations started and ended with GPU count and generation. Today, the conversation includes tokens per watt, cost per million tokens and latency at concurrency because those are the numbers that decide whether an AI product is actually profitable to run.

Training Got the Headlines. Inference Pays the Bills

Training a frontier model is a headline event. It happens occasionally, draws coverage and consumes enormous compute for weeks or months at a stretch. Inference is different. It happens every time someone sends a query, every time an agent takes a step, every time a customer-facing product calls a model. It is constant.

That volume difference is why the industry has started reframing what counts as good infrastructure. Inference is the workload companies actually get paid for and tokens are the commodity that measures it. A cluster is no longer judged mainly on how fast it trains a model. It is judged on how many tokens it can generate reliably per dollar and per watt.

The Metric That Actually Matters Now

Independent benchmarking from SemiAnalysis has been tracking exactly this efficiency question across hardware generations: how many tokens a system produces per watt and what each token costs to generate. Its most recent results show the NVIDIA GB300 NVL72 system, built on the Blackwell Ultra architecture, delivering up to 50 times more tokens per watt and 35 times lower cost per token than the NVIDIA Hopper platform. The largest cost reduction shows up at the low-latency end of the curve, which is exactly where agentic applications operate.

Those figures come from a specific benchmark setup: FP4 precision combined with disaggregated serving, where the prefill and decode stages of inference are split across the system rather than run together on the same GPUs. The real takeaway is not that one chip wins every scenario. It is that the industry has settled on the metric that decides the argument: tokens produced per watt, per dollar, per server.

Token Costs Are Falling Faster Than Most Budgets Assume

The economics reinforce the point. Running inference on NVIDIA Blackwell has already cut cost per token by up to 10 times for some providers compared with Hopper, and Gartner forecasts that by 2030, running inference on a one-trillion-parameter model will cost providers over 90 percent less than it did in 2025. But the demand is not slowing down to compensate. Coding assistants and agentic applications, which are especially token-hungry because they run multi-step reasoning over long context windows, have driven programming-related traffic from roughly 11 percent of total volume to more than half within a year, according to OpenRouter's State of AI report. Inference is not slowing down. It is only growing faster.

Power Is the New Constraint

For years, capacity planning was a chip supply problem. Rent enough GPUs and you had scaled. That logic breaks down once power becomes the limiting factor, which is where the market is heading. In a power-constrained world, the question stops being how many GPUs and starts being how many tokens per megawatt. Two clusters with an identical GPU count can produce very different amounts of usable inference output depending on networking, cooling design and software optimisation layered on top. That is a different sizing exercise than the one most infrastructure teams built their planning around when training was the priority.

Inference at Scale Is Now a Distributed Systems Problem Too

Serving a large model to real users is no longer a single-GPU decision. Long context windows and mixture-of-experts architectures often exceed the memory of one GPU or even one server, which means inference itself now gets split across multiple GPUs and nodes with prefill and decode stages routed to the hardware best suited to each. Inference infrastructure is starting to resemble the distributed systems problem training already was, except the binding constraints are latency and concurrency rather than raw throughput on a single fixed job.

That has a direct implication for cluster design. A network fabric chosen purely for training's all-reduce patterns will not necessarily hold up under the GPU-to-GPU communication patterns that high-concurrency inference needs. High-bandwidth interconnects such as NVLink matter as much for inference now as they once did for training.

The Investment Case Is Settled. The Bottleneck Has Moved

The demand side of this argument is mostly settled. In McKinsey's January 2025 survey, 92 percent of executives said they expected to increase AI spending over the following three years, and separate research from Deloitte shows organisations shifting proof-of-concept work into production at a faster rate than a year ago. The debate over whether to invest in AI is mostly over.

What remains unresolved for most teams is whether the infrastructure behind that investment can support production traffic profitably. A proof of concept can run on a few shared GPUs without anyone paying much attention to the cost per token. Production traffic is different. Once real users depend on the system, every inefficiency in the stack shows up directly in your unit economics.

This is why the shift toward inference economics has landed with users rather than just chip watchers. Enterprises already committed budget to AI. What they are now looking for is the infrastructure that lets that budget produce a return at the volume a production product actually needs, rather than the volume a pilot could get away with.

Allocation Is Becoming the Real Constraint

The cost-per-token conversation assumes you can get the hardware in the first place. But the demand for current-generation GPUs is running well ahead of what the supply chain can turn out and lead times on data-centre GPUs are stretching toward the better part of a year. This is not a single vendor's problem. It is an industry-wide demand signal, and it is changing how procurement teams have to think.

The old pattern was easy: decide you need capacity and reserve the latest architecture on a one-year term, done. That pattern is breaking now. Teams planning on that timeline are finding out mid-procurement that the market no longer moves at that pace. The teams getting ahead of it are the ones working with a partner who has already secured allocation, rather than sourcing it cold once the need becomes urgent.

What This Means for How You Plan a Cluster

Training and inference put different demands on infrastructure. Training benefits from maximum throughput for a fixed and predictable workload, while inference depends on low latency, high concurrency and consistent performance as traffic fluctuates. That is the exact point where shared infrastructure can start to show its weaknesses. Bursty and unpredictable workloads make noisy-neighbour problems much harder to ignore.

That difference has practical consequences for how you size and specify a cluster. Before your next procurement conversation, it is worth asking:

• What is the cost per million tokens at your target latency, not just at peak theoretical throughput?

• Is the network fabric, RoCE Ethernet or InfiniBand, tuned for your serving pattern or only for a training job it was originally sized for?

• Are GPUs reserved and dedicated to your workload or pooled and oversubscribed alongside other tenants?

• Can storage keep pace with checkpoint reloads and caching during live serving, not just during a training run?

Those four questions matter more than a GPU count because they are the ones that determine what a cluster actually costs you per token in production.

It also means validating those numbers before you commit to a design, not afterwards. Run real workloads on different networking, storage and GPU configurations for a defined period before approving the production build. This tells you what your specific model and traffic pattern actually deliver.

How Secure Private Cloud Helps

NVIDIA Blackwell/NVIDIA Blackwell Ultra clusters on Hyperstack Secure Private Cloud are built around exactly this shift: dedicated, single-tenant capacity where throughput per server and tokens per watt are the numbers that matter, not a shared pool where your inference latency depends on what another tenant happens to be running that hour.

Because Secure Private Cloud deployments are single-tenant by design, resources are fully reserved rather than oversubscribed. That matters more for inference than it did for training. A training job that runs slower on a noisy day still finishes overnight. A production inference endpoint that spikes in latency during a noisy period costs you users in real time.

Deployments are built on Spectrum-X RoCE or InfiniBand fabrics with NVIDIA ConnectX-8 SuperNICs, selected according to the scale and latency profile of the workload rather than a default choice. Storage tiers, from local NVMe scratch for checkpoint staging to persistent Shared Storage Volumes for datasets and artefacts, are chosen the same way to match how a specific inference workload actually reads and writes data.

For regulated industries running inference on sensitive data, knowing exactly where workloads run and who can access the supporting infrastructure is important. Hyperstack Secure Private Cloud gives dedicated, single-tenant infrastructure hosted in EU or Canadian jurisdictions under EU law. For teams that need more flexibility, AI Studio on the Hyperstack platform provides serverless inference for open-weight models you control, without requiring you to manage the underlying infrastructure.

The Question Worth Asking Now

Inference is not a phase that follows training. For most enterprises moving out of pilot mode this year, it is the workload their AI budget now depends on. The infrastructure question worth asking is no longer how many GPUs. It is how many tokens your cluster can produce per watt, per dollar, under real production load.

And that question has a timing problem attached to it: the supply side is not waiting. On Hyperstack, the NVIDIA B300s land on-demand soon with reserved private clusters on Secure Private Cloud following in Q4. If you know you'll need serious compute soon, this is the window to line it up before lead times stretch further and the decision gets made for you by whatever's left in the queue.

Model it for your workload

Want to model that number for your own workload? Talk to the Hyperstack team about what NVIDIA Blackwell and Blackwell Ultra clusters on Secure Private Cloud would look like for your inference traffic.

FAQs

What does tokens per watt mean?

Tokens per watt measures how efficiently AI infrastructure generates output using available power.

Why is inference more important than training?

Inference powers every AI interaction, making it the primary driver of ongoing infrastructure costs and revenue.

Why does cost per token matter?

Cost per token determines whether AI applications remain profitable as user demand and inference volumes increase.

How does NVIDIA Blackwell improve inference?

NVIDIA Blackwell delivers significantly higher inference efficiency with lower cost per token than previous GPU generations.

Why are dedicated GPU clusters better for inference?

Dedicated clusters provide predictable performance and avoid latency caused by shared infrastructure.