AI Infra Summit 2026 Insights: What Santa Clara Told Me About the Inference Era

I spent last week in Santa Clara at AI Infra Summit, splitting my time between the Hyperstack booth, a lot of hallway conversations and a session of my own on the Expo Theater 1 stage.

Most of what I heard came back to one idea: the industry has stopped treating AI infrastructure as a race to accumulate training compute and started treating it as an operating problem, where the hard questions are about serving models efficiently, finding enough memory, keeping power in check and staying in control of the whole stack.

Here are the five takeaways I brought home with me and what I think they mean for anyone planning AI capacity over the next 18 months.

1. Inference is now the main event and training has taken a back seat

What struck me most in Santa Clara was how far the conversation has moved past scaling raw compute and into inference, which is where the real competition now sits. I saw it in almost every conversation I had on the floor, where nobody was asking how many GPUs they could string together for one giant training run and everyone was asking how to serve models efficiently, affordably and sustainably once real users arrive.

I think this is a healthy shift for the industry, because training is episodic and you can plan for it, budget for it and live with the occasional delay. Inference runs every hour of every day and its cost scales with usage rather than ambition, so once a product takes off, the inference bill becomes the line item that decides whether the unit economics hold together.
It is a shift we planned for, and it is the reason Hyperstack Secure Private Cloud gives enterprises dedicated, single-tenant GPU clusters with zero oversubscription. Inference latency is never at the mercy of whoever else is sharing the hardware, because nobody else is. Our Dedicated Cloud tier lets teams consume that capacity as GPU VMs through the same Hyperstack portal and API used for our public cloud regions and serve inferencing endpoints on hardware nobody else touches. A fixed monthly contract gives finance a number they can plan against, which matters a great deal when inference volume is the thing growing fastest.

For teams that are not ready for a 12-month dedicated commitment, the NVIDIA B300 is available on-demand on Hyperstack at $7.40/hr. That is a practical way to benchmark production inference on current-generation hardware before deciding how much capacity to lock in.

2. Memory is the constraint you cannot buy your way out of quickly

If inference was the headline, memory was the problem sitting underneath it. Memory remains the primary bottleneck in AI computing, and the manufacturing intensity of HBM is now squeezing commodity DRAM supply, which is expected to keep memory markets tight through 2028.

That second point deserves more attention than it gets. Producing HBM is far more manufacturing-intensive than producing standard DRAM, so every increase in accelerator memory draws capacity away from the rest of the market. For infrastructure buyers, the consequence is that the servers around the GPUs become harder to source and more expensive alongside the GPUs themselves.

Memory is what inference actually runs on. Model weights, KV caches and context all compete for the same HBM, and when a model does not fit, you either shard it across more GPUs or accept lower throughput. For most inference work, the memory profile of the hardware tells you more than the headline FLOPS figure does.

An eight-GPU NVIDIA B300 system carries 2.3 TB of HBM3e with 64 TB/s of aggregate bandwidth, which lets teams keep larger models and longer contexts resident in memory instead of splitting them across more nodes. When caches and datasets outgrow GPU memory, storage effectively becomes part of the memory hierarchy. In Secure Private Cloud, Ceph is the default for flexible, general-purpose storage, and for demanding workloads we offer NVIDIA-certified parallel file systems from WEKA, VAST and DDN with RDMA support, so GPUs can talk directly to storage.

The longer memory stays tight, the more it pays to secure capacity through a partner that already has allocation in hand. As an NVIDIA partner, we secure allocation ahead of demand, which spares customers from sourcing parts cold on a procurement cycle that still assumes normal lead times.

3. Agentic workloads punish every weak layer in the stack

The third theme was agentic workloads, which are becoming far more demanding to serve. They bring dynamic contexts, very large caches and rapid autonomous turns, and the view I heard again and again was that scaling them now requires optimisation across the whole stack rather than at any single layer.

I agree with that and I think it changes how teams should evaluate infrastructure. A chatbot answers one request and then waits for the user, whereas an agent plans, calls tools, reads the results and calls the model again, often many times for a single task, with its context growing at every step. Latency compounds across those turns and cache pressure builds with them, so any inefficiency in the network, storage or software layer gets multiplied.

A GPU benchmark on its own will not tell you how an agentic workload behaves in production. What matters is how the GPUs, the fabric, the storage and the software work together under sustained and unpredictable load.

This is where owning more of the stack pays off, and it is why Hyperstack selects and tunes drivers and kernels to work together, with GPUs running at 100% of bare-metal performance inside our virtualised environment. NICs are passed through directly using SIOV, so InfiniBand or Spectrum-X fabric runs at full bare-metal line rate, and every Secure Private Cloud cluster gets a backend fabric purpose-designed for the GPU-to-GPU traffic that training and inference depend on.

Customers who would rather consume an API than manage infrastructure can use our Managed Orchestration tier, where we run the Kubernetes or Slurm layer ourselves. Owning more of the stack is what allows us to offer pre-provisioned spares, predictive failure handling and a higher SLA at that tier.

4. Power has moved from facilities footnote to a first-order planning input

The number I keep coming back to since getting home is the power forecast I heard in Santa Clara. Data centre electricity demand could exceed 60 gigawatts by 2030, reaching 12 to 15% of total consumption.

That changes the economics of everything above it. When power is the scarce input, the useful measure becomes how much work you get from every watt, and idle or underused GPUs become a waste of the scarcest resource in the building.

It changes how buyers should read a capacity offer as well. A cluster that exists on a roadmap but cannot be powered on time is worth very little, so questions about power availability, cooling and energy source now belong at the start of a procurement conversation rather than the end.

Our data centres in Europe and Canada run on 100% renewable energy, and our capacity is liquid-cooled and purpose-built for high-density GPU workloads, so customers inherit that work instead of taking on a multi-year facilities project themselves. Utilisation is the other half of the answer. In the Dedicated Cloud tier, full-stack control lets customers repurpose idle capacity for spot or R&D use and split capacity across departments or business units, so the GPUs they are paying for and powering keep doing useful work.

5. Physical AI raised the stakes on trust, safety and privacy

The final theme was physical AI, which has now moved onto factory floors, alongside a wave of new model designs producing capabilities we have not seen before. The people I spoke to in this space were clear that it brings a new set of challenges in trust, safety and privacy, and that is exactly the ground I covered in my own session.

When AI moves from a chat window into a production line, a vehicle or a hospital system, the question of who controls the model and the data behind it stops being theoretical. The data involved is often sensitive, operational or regulated, and a failure or a sudden loss of access has physical consequences.

On the Thursday morning at Expo Theater 1, I spoke about what sovereignty and openness really require at the infrastructure level, and why most teams are checking the wrong things. “Open source” and “open weights” are different things with different legal and technical guarantees, and a model with published weights can still carry licence terms that limit how you use, modify or deploy it.

Real sovereignty sits across four layers: data, infrastructure, model openness and operational control. Getting one right while ignoring the others gives you the appearance of control without the substance. Data residency means little if the model underneath can be switched off by someone else, and an open-weight model means little if it runs on shared infrastructure you cannot inspect or govern.

The encouraging part is that the old trade-off between sovereignty and capability has mostly disappeared. Open-weight models have closed enough of the gap that running them on infrastructure you control is a serious option for production workloads, including the physical AI use cases that need it most.

I closed with a reference architecture for sovereign inference and the economics of owning GPU capacity versus renting it. The short version is that renting on-demand makes sense while workloads are variable or still being proven, and dedicated capacity makes sense once utilisation is steady enough that a long-term commitment costs less than paying for flexibility you no longer use.

This is the combination we built Secure Private Cloud around. Every environment is carved out for a single customer and nothing about the GPU fabric, storage or compute is ever comingled between customers. Hyperstack holds SOC 2 Type II attestation and our 24/7 help desk is staffed by engineers based in Nottingham. Customers choose how much of the stack they want to own through four consumption tiers, from Metal Only through to a fully managed Dedicated Cloud, and pairing that with Hyperstack AI Studio for deploying and fine-tuning open-weight models puts all four layers of sovereignty under their control.
If you missed the session, I am happy to walk through the reference architecture with you one-to-one.

Where that leaves us

Put the five themes together and they describe an industry moving from building capacity to operating it well. Inference is where the spend now goes, while memory and power are the constraints that decide how much of it you can actually get. Agentic workloads expose every weak layer in the stack, and physical AI turns control of models and data into a safety question as much as a compliance one.

Each of those problems has a practical answer on Hyperstack, depending on where your priorities sit. If your priority is predictable inference performance and cost, that is Hyperstack Secure Private Cloud on dedicated hardware with a fixed monthly contract. If you need current-generation compute now, the NVIDIA B300 is available on-demand today, with dedicated clusters for teams ready to commit. And if control is the deciding factor, Secure Private Cloud with open-weight models through AI Studio covers all four layers of sovereignty.

If any of this is on your roadmap, come and talk to me directly. Tell me what you are trying to run, your timeline and how much control you need  and I will give you an honest view of what is realistic and what it would take. Book a cloud consultation today.