Inference Autoscaling: The Complete Guide to Scaling LLM Workloads on GPUs

If you are the engineer or platform lead who gets paged when your LLM endpoint slows during a traffic spike, inference autoscaling is the part of your stack that decides whether users notice.

Our blog covers what inference autoscaling scales, which metrics to use, how to reduce cold starts, how to structure autoscaling on Kubernetes and manage costs using Hyperstack GPU VMs and On-Demand Kubernetes.

Why Static GPU Capacity Breaks Under LLM Traffic

Your LLM endpoint handled launch week without trouble, then a single customer pointed a batch job at it on a Tuesday afternoon. Time to first token climbed from a few hundred milliseconds to several seconds. The GPUs you had sized for average traffic became a bottleneck for every other user on the platform.

That is the problem inference autoscaling exists to solve. LLM traffic is bursty by nature, with daily peaks, product launches, agent loops that fan out into dozens of calls and batch jobs that arrive without warning. A fixed fleet sized for the peak burns money for most of the day and a fleet sized for the average fails at exactly the moment demand matters most.

What Inference Autoscaling Scales

Inference autoscaling adds and removes model-serving capacity in response to live demand so that latency stays within your targets and idle GPU hours stay low. That means adjusting two different things that move at very different speeds.

• The first is the number of model replicas, which are the pods running an inference server such as vLLM, SGLang or NVIDIA Triton Inference Server with a copy of the model loaded into GPU memory. Adding a replica on a GPU you already have takes as long as loading the model, which can be tens of seconds for a small model and several minutes for a large one.

• The second is the number of GPU nodes on which those replicas run. Adding a node means provisioning a GPU VM, joining it to the cluster, validating drivers and pulling the serving image. The timescale is minutes rather than seconds.

Treat replica and node scaling as separate control loops: replicas absorb short bursts on existing capacity, while nodes follow sustained demand and maintain headroom for the next burst.

LLM requests process prompts during prefill and generate tokens during decode with KV cache memory growing alongside context length. Request costs can vary more than tenfold, making request count alone a poor measure of load.

Scale on Queue Depth and Latency

GPU utilisation is the first metric most teams reach for and it is usually the wrong one. The standard utilisation figure reported through NVIDIA DCGM measures the share of time at least one kernel was running on the GPU, not how much of the GPU's compute or memory bandwidth that kernel used. Continuous batching in modern inference servers keeps that number high across a wide range of real load, so a replica can look saturated with plenty of spare capacity left.

The better signals come from the inference server itself. vLLM, for example, exposes Prometheus metrics for running requests, waiting requests, KV cache usage and latency histograms for time to first token and time per output token. These map to what your users feel and to what your GPUs are actually short of.

Cold Starts Decide How Fast You Can Really Scale

Your autoscaling policy can only react as fast as a new replica can start serving. For LLMs, that startup time is dominated by three steps: pulling the container image, loading model weights from storage into GPU memory and warming up the inference engine with memory profiling and CUDA graph capture.

The numbers add up quickly. A 70-billion-parameter model in 16-bit precision carries roughly 140 GB of weights, and serving images with CUDA libraries and frameworks often run to several gigabytes. Pulling both across a network on every scale event can turn a five-minute burst into one that your new capacity never catches.

Four techniques consistently shorten the path from scale decision to first token:

A Reference Architecture for Inference Autoscaling on Kubernetes

Kubernetes has become the default control plane for inference autoscaling because it already separates the two loops described above. A typical production stack has five layers that work together:

This KEDA configuration scales a vLLM Deployment using per-replica targets of five waiting requests and 35 running requests. The running-request trigger protects busy replicas when queues empty; replace 35 with a load-tested value. A ten-minute stabilisation window and a limit of one replica removed every two minutes prevent abrupt scale-in.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-server
spec:
  scaleTargetRef:
    name: vllm-llama
  minReplicaCount: 1
  maxReplicaCount: 8
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600
          policies:
            - type: Pods
              value: 1
              periodSeconds: 120
  triggers:
    - type: prometheus # requests waiting (demand)
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{model_name="llama"})
        threshold: "5"
    - type: prometheus # requests running (in-flight load)
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_running{model_name="llama"})
        threshold: "35" # example value: set yours from load testing

The interaction between the pod and node layers is where most real-world problems appear. A burst triggers the pod autoscaler, new pods go pending because no GPU is free, the node layer requests a GPU VM and the new pods start only once that node is ready and the model has loaded. If that chain takes eight minutes and your bursts last five, the system scales up just in time to scale back down.

Two habits make that chain far more predictable in production. The first is to separate node groups by GPU type so that a pod requesting an NVIDIA H100 never waits on a node group that only offers smaller cards. The second is to keep a small buffer of spare GPU capacity, often held by low-priority placeholder pods that are evicted the moment a real replica needs the GPU.

Understanding the Costs of Scaling Decisions

Every inference autoscaling policy is a trade between idle GPU hours and latency risk, and the right point on that curve comes down to what a slow response costs your business. An internal summarisation tool can tolerate a cold start, but a customer-facing assistant with a contractual latency target cannot.

Aggressive scale-in saves money on paper and then gives it back through repeated cold starts, failed requests during scale events and engineering time spent chasing latency spikes. A conservative baseline that covers normal daytime load with autoscaling handling the peaks above it is usually cheaper in total than a fleet that tries to track demand minute by minute.

Autoscaling is also not the answer for every workload. A single model with flat traffic may need nothing more than a fixed replica count and good monitoring and adding autoscaling there only introduces new failure modes. At the other end, a baseline that sits at hundreds of GPUs around the clock favours reserved or dedicated capacity for that steady layer with on-demand capacity covering the bursts above it.

Running Inference Autoscaling on Hyperstack GPU VMs and Kubernetes

Hyperstack gives inference teams two building blocks for this architecture. On-Demand GPU VMs provide NVIDIA B300, NVIDIA B200, NVIDIA H200, NVIDIA H100, NVIDIA A100, NVIDIA L40 and NVIDIA RTX A6000 capacity among other options, launching in minutes, priced per hour and charged by the minute. A VM bills until you delete it, so remove burst capacity when demand falls: stopping a VM does not stop its charges, and hibernating it stops GPU charges while its storage keeps billing.

Structuring Autoscaling on Hyperstack Kubernetes

Node groups let you run different VM flavours and configurations within the same cluster. This helps you match GPU and CPU resources to specific workloads.

For example:

• NVIDIA H100 GPU VMs: Run large models.
• Smaller GPU VMs: Handle embeddings and reranking.
• CPU nodes: Manage gateways and routing.

You can create and scale node groups through the Hyperstack API to respond to sustained pod-level demand. Automatic node scaling through Cluster Autoscaler is also planned.

How Node Scaling Works Today

Until automatic node scaling is available, you need to manage node capacity through your own automation:

• Monitor pending GPU pods to identify when more capacity is needed.
• Use the Create Node API to add workers.
• Drain nodes before deleting them.
• Run scaling operations one at a time. Every existing node must be in the ACTIVE state before another operation begins.

Keep These Platform Limits In Mind

• Each node group supports 1 to 20 workers.
• A cluster must always have at least one node group.
• To scale GPU capacity to zero, retain a small CPU-only node group and delete the GPU node group.
• Your automation must also restore the placeholder buffer by adding another node when a real replica takes its place.

Reducing Cold Starts

Hyperstack selects the Kubernetes node image automatically, so you cannot bake your model-serving images into it.

Instead, use a DaemonSet to pre-pull container images onto standby nodes before demand increases. This reduces the time workers spend downloading images when new pods need to start.

Managing Persistent Model Weights

The Hyperstack CSI Driver provisions persistent volumes through standard Persistent Volume Claims (PVCs). There are a few limitations to consider:

• Volumes use ReadWriteOnce, meaning each volume can be mounted by only one node at a time.
• Each worker node can attach a maximum of 10 volumes.

To retain cached model weights during scale-in, configure your model server as a StatefulSet:

1. Use volumeClaimTemplates: Give each replica its own persistent volume so cached weights survive pod replacement.
2. Create a headless Service: StatefulSets require one to manage network identity.
3. Set podManagementPolicy: Parallel: This allows replicas to start together rather than waiting for each one to become ready before starting the next.
4. Update the KEDA target: Then point the ScaledObject at the StatefulSet because KEDA targets a Deployment by default.

spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: vllm-llama

A replica that comes back after scaling in reuses the weights it already cached. Replicas created for the first time still start cold and retained volumes keep billing until you delete them.

We also offer an open-source LLM Inference Toolkit on GitHub for teams that want a starting point for their serving stack.

When you need hundreds of GPUs running continuously, Hyperstack Secure Private Cloud provides dedicated, single-tenant infrastructure with contracts of 12 months or more. On-Demand GPU VMs can then handle extra demand when workloads increase, giving you a cost-effective way to manage both steady and changing workloads.

Test Your Inference Autoscaling Policy
on Hyperstack

The fastest way to know whether your scaling signals hold up is to run them against real traffic on real hardware. Sign up for Hyperstack, launch an On-Demand Kubernetes cluster with a GPU node group in minutes and watch how queue depth, cache pressure and time to first token behave as you push load through your serving stack.

FAQs

What is inference autoscaling?

Inference autoscaling automatically adds and removes model-serving replicas and GPU nodes based on live demand, keeping LLM latency within targets during traffic spikes and cutting idle GPU spend when demand falls.

What metric should I use to autoscale LLM inference?

Queue depth per replica is the most reliable primary signal. Add running requests per replica so busy replicas are not removed when nothing is queued, add KV cache usage for long-context traffic and track p95 time to first token to confirm scaling decisions protect user latency.

Why is GPU utilisation a poor autoscaling metric for LLMs?

Standard GPU utilisation only shows whether a kernel was running, not how hard the GPU worked. Continuous batching keeps it high across most load levels, so it rarely reflects true serving capacity.

How do you reduce cold start time for LLM inference?

Pre-pull serving images, cache model weights, quantise where accuracy allows and keep a warm replica for latency-sensitive endpoints.

Can Kubernetes autoscale GPU inference workloads?

Yes. The Horizontal Pod Autoscaler or KEDA scales inference pods on custom metrics, and a node autoscaler or provider API adds and removes GPU nodes when pods cannot be scheduled.

Should LLM inference endpoints scale to zero?

Scale to zero suits low-traffic workloads that tolerate cold starts, with a proxy such as the KEDA HTTP Add-on holding requests during startup. Latency-sensitive endpoints usually need warm replicas.

How does Hyperstack support inference autoscaling?

Hyperstack On-Demand Kubernetes runs on GPU VMs with NVIDIA-optimised drivers, supports node groups per GPU flavour and lets teams scale node groups through the API using their own automation, with worker billing only while active. Automatic node scaling is planned for the platform.