Deploy Kimi K3 on GPU Cloud for Multi-Node 2.8T Inference

Kimi K3 is Moonshot AI's 2.8 trillion parameter flagship, and since the weights were published on 27 July 2026 anyone can serve it themselves. It is a sparse Mixture-of-Experts model built on Kimi Delta Attention and Attention Residuals, activating 104 billion parameters through 16 of 896 experts per token, with native vision through MoonViT-V2 and a one million token context window. The checkpoint ships in native MXFP4, and when we pulled it the 96 safetensors shards measured 1,560.94 GB. That one number decides everything about serving it. The official vLLM recipe and the vLLM launch blog both put the floor at Blackwell: at least one 8x NVIDIA B300 node, or an NVIDIA GB300 NVL72, with 16x NVIDIA B200 also supported. NVIDIA H100 is not on that list.

This guide serves Kimi K3 on NVIDIA H100 anyway: 32 NVIDIA H100 80GB GPUs across four Hyperstack nodes, tensor parallel 32, expert parallel 32, at $64 per hour. Every log line, memory figure and timing below comes from a single run on 28 July 2026 that reached a live endpoint eleven minutes after the containers started, then answered chat, called two tools in one turn, read an image and returned its chain of thought, with the full one million token context window intact. If you have read our technical deep dive into Kimi K3, this is the deployment half of that story: the arithmetic that fixes the cluster shape, the launch command, and what the model does once it answers.

Kimi K3 in the Numbers That Decide a Deployment

Before any hardware is booked, five figures from the model card set the shape of the deployment: the parameter count, the activated parameter count, the quantisation format, the layer composition and the context window. Everything else in this guide follows from them.

Two of those rows are worth pausing on. MXFP4 stores roughly four bits per parameter plus a shared block scale, so 2.8 trillion parameters land at about 1.4 TB before overheads, and at 1.56 TB in practice once every tensor and the vision tower are counted. 16 of 896 is an extreme sparsity ratio, and it is why a 2.8 trillion parameter model has a compute cost closer to a 104 billion parameter one. The memory cost, however, is the full 2.8 trillion, because every expert has to be resident somewhere.

For context on what that memory buys, here is Kimi K3 against the strongest proprietary models on a representative slice of the published results.

The architecture you are about to shard across 32 GPUs

Kimi K3 is built on two architectural updates. Kimi Delta Attention gives the model an efficient foundation for scaling attention along sequence length, and Attention Residuals retrieve representations selectively across depth rather than accumulating them uniformly. The Mixture-of-Experts block is a Stable LatentMoE, trained stable at this sparsity by four techniques: Quantile Balancing, which derives expert allocation from router-score quantiles instead of a heuristic; Per-Head Muon, which optimises attention heads independently; the Sigmoid Tanh Unit for activation control; and Gated MLA for attention selectivity.

Kimi K3 also produced a motion-graphics explainer of its own architecture, which is worth watching before you shard it across 32 GPUs.

A 3Blue1Brown-style explainer of the Kimi K3 architecture, made by Kimi K3. Video: Moonshot AI, Kimi K3 release blog.

Why 1.56 TB of MXFP4 Does Not Fit Where You Expect

Read this section before provisioning anything, because it decides the shape of the cluster and it is where most of the money goes.

Start with the weights

MXFP4 packs four bits per parameter plus a shared exponent per block, so 2.8 trillion parameters imply roughly 1.4 TB before any overhead. The download settles the question exactly. Our verification script walked the Hugging Face manifest, resolved every symlink and added up the real bytes on disk on each of the four nodes:

repo             : moonshotai/Kimi-K3
snapshot dir     : /ephemeral/hf/hub/models--moonshotai--Kimi-K3/snapshots/9f62e4e9fffbd0a83ddd60e1c209d828994b3569
safetensors      : 96/96 present
total bytes      : 1,560,936,091,448 (1560.94 GB)
config.json      : OK
.incomplete files: 0
RESULT: COMPLETE

That is 1,560,936,091,448 bytes across 96 shards, identical on all four nodes, and it is the number every later calculation uses. It is worth pausing on the second half of that sentence, because a tensor-parallel deployment does not split the download the way it splits the model.

Now count the GPUs

An NVIDIA H100 "80GB" card is a rounded label. On our nodes nvidia-smi reported 81,559 MiB, which is 79.65 GiB, and the PyTorch allocator counted a total capacity of 79.19 GiB per card. Thirty-two of them come to about 2,534 GiB, or roughly 2.72 TB.

Set 1.56 TB of weights against 2.72 TB of memory and the deployment fits, with about 1.16 TB left for the KV cache, the recurrent state that Kimi Delta Attention needs, the activations, the CUDA context and the communication buffers. That arithmetic is what makes 32 NVIDIA H100 GPUs the right shape for this model, and the measured per-GPU budget later in this guide lands almost exactly where it predicts.

Why adding more NVIDIA H100 GPUs does not help

The obvious response to a tight fit is to add nodes. On Kimi K3 that does not work, and the reason is arithmetic rather than budget. Tensor parallelism has to divide the model evenly, which means it has to divide both the 96 attention heads and the 7168 hidden size. The greatest common divisor of 96 and 7168 is 32, so the valid tensor-parallel sizes are 1, 2, 4, 8, 16 and 32, and no more. A tensor-parallel size of 64 would ask for 1.5 attention heads per rank.

So TP 32 is not the largest shape that happens to work. It is the only one that clears both tests, and four nodes of eight NVIDIA H100 GPUs is the only way to reach it on this generation. Adding a fifth node cannot lower the per-GPU weight load, because 40 does not divide 96 either, and data parallelism replicates the model rather than sharding it. If TP 32 does not fit, the answer is a larger card, not a longer invoice.

One Model, Four Machines: the 32-GPU Topology

Kimi K3 has no sanctioned pipeline-parallel strategy, so a multi-node deployment runs tensor parallelism across the node boundary, with expert parallelism laid over the top. That is a different shape from the one we used for Tencent Hy3, where pipeline parallelism kept the heavy collectives inside each node and put a single activation tensor on the wire. Here every rank participates in every layer, so the topology is flat: thirty-two equal ranks, one process group, one model.

Three things follow from that, and they shape every command in the deployment steps below. All four nodes have to sit in the same environment, because that is what puts them on a shared private network. Collectives are pinned to that private interface with NCCL_SOCKET_IFNAME and run over TCP sockets, since on-demand nodes are joined by ordinary Ethernet rather than InfiniBand. And every node needs its own full copy of the weights on local NVMe, because each rank loads its shard of every tensor from disk.

The Kernel Path on Hopper, and the Settings That Fit 1.56 TB

This deployment uses SGLang, which publishes a Kimi K3 cell for four NVIDIA H100 nodes in its deployment cookbook. Two things about that configuration are worth understanding before you run it, because they explain both the flags and the performance.

MXFP4 weights, a mixed-precision kernel

Kimi K3 stores its expert weights in MXFP4, and the MoE backends that execute 4-bit weights natively are built around Blackwell FP4 hardware, which is why the published guidance points at NVIDIA B300 and NVIDIA GB300. On Hopper the expert matrix multiplies run instead through Marlin, the mixed-precision W4A16 kernel that multiplies 16-bit activations by 4-bit weights and hides the dequantisation cost behind the matrix multiply itself. The engine announces the choice on the first line of weight loading:

[TP0 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled
          (moe_runner_backend=marlin, quant_method=Mxfp4MoEMethod).

That single line in the log explains the throughput figures later in this guide. It is also why --moe-runner-backend marlin appears in the launch command rather than being left to auto: on Hopper it is the path that runs, so it is worth stating.

Three settings that keep the load inside the budget

Loading 1.56 TB through 32 allocators at once is the part of this deployment that is sensitive to configuration. Three of the settings in the launch command exist for exactly that, and they are the ones to keep if you change anything else:

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True switches the PyTorch caching allocator onto CUDA virtual memory management, so segments grow and shrink in place instead of being pinned as separate cudaMalloc blocks that can never merge. On a load that streams 96 shards through the allocator, that is what keeps the resident footprint close to the size of the weights themselves.
• --ep-size 32 states expert parallelism explicitly rather than leaving it to be inferred, so each rank holds exactly its own slice of the 896 experts and no more.
• --mem-fraction-static 0.85 sizes the static pool, and it is the single lever to reach for if you want a larger KV cache or more headroom.

With those in place the load runs to completion, and the head node reports where every gigabyte went. The log arrives in four stages, and each one answers a different question.

First, the cluster forms. The tokeniser loads, then all 32 ranks find each other and agree on the process group. Forty-one seconds for a rendezvous across four machines is normal, and almost no memory has been touched yet:

[2026-07-28 07:39:14] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2026-07-28 07:39:14] Applying special tokens cache patch for Kimi tokenizer
[2026-07-28 07:39:14 TP0 EP0] Init torch distributed begin.
[2026-07-28 07:39:58 TP0 EP0] Init torch distributed ends. elapsed=41.49 s, mem usage=0.47 GB

Then the weights load. This is the long part, and the middle line is the kernel choice from the section above. Note the pair of numbers at either end: 78.23 GB available before, 18.60 GB after, so the weights themselves cost 59.63 GB per card:

[2026-07-28 07:39:58 TP0 EP0] Load weight begin. avail mem=78.23 GB
[2026-07-28 07:39:58 TP0 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled
     (moe_runner_backend=marlin, quant_method=Mxfp4MoEMethod).
[2026-07-28 07:45:16 TP0 EP0] Load weight end. elapsed=317.50 s,
     type=KimiK3ForConditionalGeneration, quant=compressed-tensors,
     avail mem=18.60 GB, mem usage=59.63 GB.

Next the memory pools. The KV cache is only 3.62 GB, but available memory falls by 7.03, and the difference is the recurrent state that Kimi Delta Attention keeps per sequence. The attention backends are chosen here too, one for decode and a different one for prefill:

[2026-07-28 07:45:18 TP0 EP0] KV Cache is allocated. dtype: torch.bfloat16,
     #tokens: 140352, KV size: 3.62 GB
[2026-07-28 07:45:18 TP0 EP0] Memory pool end. avail mem=11.57 GB
[2026-07-28 07:45:18 TP0 EP0] Using hybrid attention backend for decode and prefill:
     decode_backend=flashmla, prefill_backend=fa3.

Finally the server comes up. The line to read is the first one, because it is where the engine states what it will actually accept:

[2026-07-28 07:49:32 TP0 EP0] max_total_num_tokens=140352, chunked_prefill_size=8192,
     max_prefill_tokens=16384, max_running_requests=49, context_len=1048576,
     available_gpu_mem=10.34 GB
[2026-07-28 07:49:32] INFO:     Application startup complete.
[2026-07-28 07:49:32] INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2026-07-28 07:50:15] The server is fired up and ready to roll!

Put those four stages together and the per-GPU budget is settled: 59.63 GB of weights, 7.03 of pools, about 1.2 for CUDA graph capture, and 10.34 GB still free. Across all 32 cards nvidia-smi showed 70,960 MiB in use of 81,559 MiB, and the server advertised the full context_len=1048576.

How to Deploy Kimi K3 on Hyperstack

Now, let us walk through the step-by-step process of standing up the four-node cluster and serving Kimi K3 across all 32 GPUs.

Step 1: Accessing Hyperstack

First, you will need an account on Hyperstack.

• Go to the Hyperstack website and log in.
• If you are new, create an account and set up your billing information. Our documentation can guide you through the initial setup.

Step 2: Deploying Four 8x NVIDIA H100 Virtual Machines

From the Hyperstack dashboard, launch four identical GPU virtual machines. The node count is fixed by the tensor-parallel arithmetic above: tensor parallel 32 needs 32 GPUs, and nothing smaller divides the model.

• Initiate Deployment: Click the "Deploy New Virtual Machine" button on the dashboard.

• Select Hardware Configuration: Choose an 8x NVIDIA H100-80G-PCIe-NVLink flavour, listed in the flavour reference as n3-H100x8-NVLink. Our run used the Spot variant, n3-H100x8-NVLink-spot, at $2.00 per GPU per hour, which is $16.00/hr per node and $64.00/hr for the cluster. Take the NVLink variant rather than plain PCIe: per NVIDIA's H100 PCIe product brief it bridges each card to a single adjacent card at 600 GB/s, and nvidia-smi topo confirmed the NV12 pairs on our nodes.

• Choose the Operating System: Select the "Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker" image. The driver line has to match what the engine container expects, and this image also ships Docker and the NVIDIA container runtime ready to use.
• Environment: Choose an environment carrying NVIDIA H100 NVLink stock. Our run used CANADA-1. All four nodes must land in the same environment, because that is what puts them on a shared private network, and the entire deployment depends on it.
• Select a Keypair: Choose an existing SSH keypair. Use the same keypair for all four nodes.
• Network Configuration: Assign a Public IP to every node so you can reach them over SSH, and note each node's private IP once it is running.
• Review and Deploy: Check the settings, click "Deploy", and repeat until all four nodes exist with identical settings.

Step 3: Configure the Firewall

The four nodes have to talk to each other freely on the private network, and you need to reach the API from your own machine. Add these firewall rules:

Every node's private IP is shown on its detail page in the dashboard, so you can fill in that third rule without logging in anywhere. It is also the rule people forget. NCCL negotiates its ports dynamically, so pinning it to a single port does not work and the range has to be wide. The source does not have to be wide, though. Scope it to the peer nodes' private addresses, or at most to your environment's own subnet, rather than to the whole 10.0.0.0/8 range: a cloud private network is shared, and "not reachable from the internet" is not the same as "reachable only by my four machines".

Step 4: Accessing Your Nodes

Once all four virtual machines are running, copy their Public IP addresses from the dashboard and connect to each in a separate terminal.

# Connect to each node using your private key and the node's public IP
ssh -i [path_to_your_ssh_key] ubuntu@[your_node_public_ip]

You will also need three pieces of information on every node: its private IP, the name of its private network interface, and confirmation that the large disk is mounted. Run this on all four:

# Private IP and the name of the private interface
ip -o -4 addr show | awk '{print $2, $4}' | grep -E ' 10\.'

# All eight NVIDIA GPUs present, with their real memory size
nvidia-smi --query-gpu=index,name,memory.total --format=csv

# The two disks: a small root and a very large NVMe
df -h / /ephemeral

ens6 10.0.0.219/24

index, name, memory.total [MiB]
0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
...
7, NVIDIA H100 PCIe, 81559 MiB

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        96G   12G   84G  13% /
/dev/vdb1       6.3T   28K  6.0T   1% /ephemeral

On our NVIDIA H100 nodes the private interface was ens6 on the 10.0.0.0/24 subnet, and the head node's private IP was 10.0.0.219. Do not assume these values match yours. The interface name in particular varies by flavour, so read both from the commands above and substitute them wherever this guide shows ens6 or a 10.0.0.x address. Both values go into the launch command in Step 8, so it is worth writing them down now.

The disk check is what shapes the rest of the guide. The machine has two disks and only one of them matters here: the root filesystem is roughly 96 GB, while the multi-terabyte NVMe disk is mounted separately at /ephemeral and offers around 6.3 TB on this flavour. It is documented under ephemeral storage. Kimi K3 is 1.56 TB and the container image adds another 13.3 GB on top, so nothing about this deployment fits on the root disk.

Step 5: Point Docker and the Model Cache at /ephemeral

Two things have to move onto the big disk: the Hugging Face cache that will hold the weights, and Docker's own storage, because container images are pulled onto the root disk by default. Everything in this step runs on all four nodes.

Start with the directories. /ephemeral/hf becomes the Hugging Face cache, and it is the path that gets mounted into the container later so the engine finds the weights already present:

# The Hugging Face cache and Docker's storage both go on the big NVMe disk
sudo mkdir -p /ephemeral/hf /ephemeral/docker
sudo chown -R ubuntu:ubuntu /ephemeral/hf

Next, move Docker itself. The data-root key tells the daemon to store images and layers on the NVMe disk rather than the 96 GB root partition, and it only takes effect on a restart:

# Docker data-root -> /ephemeral (root is only ~96 GB; the image alone is 13.3 GB)
sudo systemctl stop docker docker.socket
echo '{"data-root":"/ephemeral/docker"}' | sudo tee /etc/docker/daemon.json
sudo systemctl start docker
sleep 4

Finally, clear anything Docker has already written to the root disk, then check both halves landed. On a fresh node the prune finds little to remove, and it earns its place when you are re-running on a node that has pulled images before. docker info should now report Docker Root Dir: /ephemeral/docker:

# Reclaim whatever the previous storage driver left on the root disk
docker system prune -af

# Then confirm both halves of the move
docker info | grep 'Docker Root'
df -h /

Step 6: Download Kimi K3 and Pull the Engine Image

Every node needs its own complete copy of the weights, because every rank loads its shard of every tensor from local disk. The image pull and the model download are independent, so start them together on all four nodes. First make sure the Hugging Face client is present:

# Ubuntu 24.04 marks its Python as externally managed, hence the flag
python3 -m pip install -q --break-system-packages huggingface_hub hf_transfer

Kick off the image pull first, because it is the quicker of the two and nothing about it depends on the weights:

# Start the image pull in the background. It is 13.3 GB and it does not
# depend on the weights, so there is no reason to wait for it.
nohup docker pull lmsysorg/sglang:kimi-k3 >/tmp/pull.log 2>&1 &

Then start the weights. Three details in this command are doing the work: HF_TOKEN authenticates the download, because Hugging Face applies tighter rate limits to anonymous traffic than to signed-in users, so a token is what sustains full speed across 1.56 TB; HF_HOME puts the cache on the NVMe disk rather than the root partition, which is the whole point of the previous step; and max_workers=16 is what fills the pipe.

# Authenticate, then pull 1.56 TB of weights onto the NVMe disk.
# max_workers=16 is what fills a multi-gigabit pipe on this flavour.
export HF_TOKEN="YOUR_HF_TOKEN_HERE"
nohup env HF_HOME=/ephemeral/hf HF_HUB_ENABLE_HF_TRANSFER=1 HF_TOKEN="$HF_TOKEN" \
  python3 -c "from huggingface_hub import snapshot_download; snapshot_download('moonshotai/Kimi-K3', max_workers=16)" \
  >/tmp/dl.log 2>&1 &

Both are detached, so you can watch the disk fill rather than the terminal. On our nodes the transfer peaked at 1.8 GB/s and finished in about 32 minutes, an average of roughly 0.8 GB/s across the whole checkpoint:

# Watch the disk fill. Expect roughly 1.5 TB over about half an hour.
watch -n 30 'df -h /ephemeral | tail -1'

Step 7: Verify All 96 Shards Before You Serve

A 32-rank cluster spends about eleven minutes loading before it is in a position to notice that one of the 96 shards is short. Confirming the download first takes seconds on an idle node and gives you a definite answer before the expensive part begins, so it is worth the keystrokes on a checkpoint this size.

The script reads your token from ~/hf_token.txt, so save it there first with echo "$HF_TOKEN" > ~/hf_token.txt. It then builds in four parts, starting with the arguments and the header that authenticates the API call:

#!/usr/bin/env python3
# verify_k3_download.py - confirm a node has the COMPLETE Kimi K3 checkpoint
# before we try to serve. Run ON the node.
import argparse, glob, json, os, urllib.request

ap = argparse.ArgumentParser()
ap.add_argument("--repo", default="moonshotai/Kimi-K3")
ap.add_argument("--hf-home", default="/ephemeral/hf")
ap.add_argument("--token-file", default=os.path.expanduser("~/hf_token.txt"))
a = ap.parse_args()

tok = ""
if os.path.exists(a.token_file):
    tok = open(a.token_file).read().strip()
h = {"User-Agent": "Mozilla/5.0"}
if tok:
    h["Authorization"] = "Bearer " + tok

Next it asks Hugging Face what should be there. The repository metadata lists every file in the repo, and filtering it to .safetensors gives the 96 names to look for:

# expected manifest from HF
req = urllib.request.Request(f"https://huggingface.co/api/models/{a.repo}", headers=h)
meta = json.loads(urllib.request.urlopen(req, timeout=60).read())

expected = [s["rfilename"] for s in (meta.get("siblings") or [])]
exp_st = sorted(f for f in expected if f.endswith(".safetensors"))

Then it finds the local copy. The downloader writes into a snapshot directory named after the commit hash and fills it with symlinks into the blob store, so the size of each link has to be read through os.path.realpath rather than from the link itself:

snap = os.path.join(a.hf_home, "hub", "models--" + a.repo.replace("/", "--"), "snapshots")
dirs = glob.glob(os.path.join(snap, "*"))
if not dirs:
    print(f"MISSING: no snapshot dir under {snap}")
    raise SystemExit(2)
d = dirs[0]

local, total = {}, 0
for f in exp_st:
    p = os.path.join(d, f)
    if os.path.exists(p):
        sz = os.path.getsize(os.path.realpath(p))
        local[f] = sz
        total += sz

Finally it compares the two and exits non-zero if anything is missing, so the whole thing can go in front of your launch command:

missing = [f for f in exp_st if f not in local]
incomplete = glob.glob(os.path.join(d, "*.incomplete"))
cfg_ok = os.path.exists(os.path.join(d, "config.json"))

print(f"repo             : {a.repo}")
print(f"snapshot dir     : {d}")
print(f"safetensors      : {len(local)}/{len(exp_st)} present")
print(f"total bytes      : {total:,} ({total/1e9:.2f} GB)")
print(f"config.json      : {'OK' if cfg_ok else 'MISSING'}")
print(f".incomplete files: {len(incomplete)}")

ok = (not missing) and (not incomplete) and cfg_ok
print("RESULT:", "COMPLETE" if ok else "INCOMPLETE")
raise SystemExit(0 if ok else 1)

Run it on every node. What you want to see, with the same snapshot hash on all four:

repo             : moonshotai/Kimi-K3
snapshot dir     : /ephemeral/hf/hub/models--moonshotai--Kimi-K3/snapshots/9f62e4e9fffbd0a83ddd60e1c209d828994b3569
safetensors      : 96/96 present
total bytes      : 1,560,936,091,448 (1560.94 GB)
config.json      : OK
.incomplete files: 0
RESULT: COMPLETE

Step 8: Launch Kimi K3 Across All Four Nodes

Unlike a pipeline-parallel deployment, there is no separate follower command here. All four nodes run the same command, and only two values change between them: --node-rank and SGLANG_HOST_IP. Rank 0 is the head, and it is the only node that will answer HTTP, but it holds no more of the model than the others.

Set the five values first. Only the first two differ between machines, so the safest way to work is to open four terminals, paste this block into each, and change the two lines:

# Run this on EVERY node. Only RANK and SELF_PRIV change between them.
export RANK=0                     # 0 on the head, then 1, 2 and 3
export SELF_PRIV="10.0.0.219"     # THIS node's private IP
export HEAD_PRIV="10.0.0.219"     # the head node's private IP, on all four
export NIC="ens6"                 # the private interface from Step 4
export HF_TOKEN="YOUR_HF_TOKEN_HERE"

# Clear any container left from an earlier launch
docker rm -f sgl 2>/dev/null || true

Next collect the engine arguments. Putting them in a variable keeps them separate from the container plumbing, and it makes the difference between the four nodes a single line rather than a diff across thirty:

# Everything the engine itself needs, collected in one variable so the
# docker command below stays readable.
K3_ARGS="--trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 32 \
  --ep-size 32 \
  --nnodes 4 \
  --node-rank $RANK \
  --dist-init-addr $HEAD_PRIV:20000 \
  --moe-runner-backend marlin \
  --decode-attention-backend flashmla \
  --mem-fraction-static 0.85 \
  --dist-timeout 3600 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 8000"

Then launch. Everything here is container-level: the GPUs, host networking and shared memory, the weights mounted where the engine looks for them, and the environment the engine reads at start-up:

docker run -d --name sgl --gpus all --network host --ipc=host --shm-size 32g \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env NCCL_CUMEM_ENABLE=1 \
  --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --env SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \
  --env SGLANG_K3_ATTN_RES_MODE=jit \
  --env SGLANG_MOE_FUSED_GATE_RADIX=1 \
  --env SGLANG_HOST_IP=$SELF_PRIV \
  --env NCCL_SOCKET_IFNAME=$NIC \
  --env GLOO_SOCKET_IFNAME=$NIC \
  --env NCCL_IB_DISABLE=1 \
  lmsysorg/sglang:kimi-k3 \
  sglang serve $K3_ARGS

Start rank 0 first, then the other three within a minute or so of each other. The rendezvous on port 20000 waits for all four, and --dist-timeout 3600 gives it an hour before it gives up, which is deliberate on a load this size.

Here is what each part of those three blocks is for:

Step 9: Verify the Deployment

Follow the head node while the cluster forms:

docker logs -f sgl

The timeline below is our run, from the first container log line to the readiness message. Four moments in it are the ones to watch for, and they carry a filled marker.

Now confirm the model is being served, from your own machine:

# Use the HEAD node's PUBLIC IP here, not the private one
export HEAD_IP="your.head.node.public.ip"

curl -s http://$HEAD_IP:8000/v1/models

{
  "object": "list",
  "data": [
    {
      "id": "moonshotai/Kimi-K3",
      "object": "model",
      "owned_by": "sglang",
      "root": "moonshotai/Kimi-K3",
      "max_model_len": 1048576
    }
  ]
}

The max_model_len of 1,048,576 is the line to look for. A 2.8 trillion parameter model is live on 32 GPUs with its full one million token context window intact, not a truncated one.

Finally, look at the memory across all four nodes. This is 2.8 trillion parameters resident on thirty-two NVIDIA H100 GPUs:

=== node 0 (head, rank 0) ===
0, NVIDIA H100 PCIe, 71790 MiB, 81559 MiB
1, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...
=== node 1 (rank 1) ===
0, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...
=== node 2 (rank 2) ===
0, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...
=== node 3 (rank 3) ===
0, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...

Thirty-one of the thirty-two cards read 70,960 MiB of 81,559 MiB. One card reads 71,790 MiB, about 830 MiB more, which is consistent with the rank that also carries the HTTP server process. That leaves about 10.3 GB of headroom on every card, matching the engine's own available_gpu_mem=10.34 GB exactly. Kimi K3 is live.

Talking to Kimi K3

The endpoint is OpenAI compatible, so every example below is a standard chat completion. What is specific to Kimi K3 is not the transport but the behaviour: it always thinks, it returns that thinking in a separate field, and it expects you to hand that thinking back on the next turn.

The first request, and the field most clients ignore

curl -s http://$HEAD_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "moonshotai/Kimi-K3",
    "messages": [{"role": "user", "content": "Who are you? One sentence."}],
    "max_tokens": 256
  }'

The reply is a standard chat completion with one extra field. Here is the message it returned:

"message": {
  "role": "assistant",
  "content": "Hello! I'm Kimi, an AI assistant developed by Moonshot AI.",
  "reasoning_content": "They specify background identity: current assistant is Kimi,
                        developed by Moonshot AI. Need answer naturally, short
                        sentence. Final only.",
  "tool_calls": null
}

The answer is in content as usual, but reasoning_content is populated alongside it. That field exists because the server was launched with --reasoning-parser kimi_k3, and without it the thinking would arrive inline in the answer. Now look at what it cost:

"usage": {
  "prompt_tokens": 137,
  "completion_tokens": 135,
  "total_tokens": 272,
  "reasoning_tokens": 109
}

reasoning_tokens reports 109 of the 135 completion tokens. On a question this small, 81 per cent of the generated tokens were thinking, and a client that reads only content pays for all of them while seeing none of them.

Every Python example from here on shares one small helper. It is the standard library only, so there is nothing to install, and it is the same request shape the shell examples above send:

import json
import time
import urllib.request

BASE = "http://<HEAD_NODE_PUBLIC_IP>:8000/v1"
MODEL = "moonshotai/Kimi-K3"
KEY = "EMPTY"

def post(body, timeout=1800):
    req = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.load(r), time.time() - t0

With that in place a chat request is a dictionary. The only field that is not standard OpenAI is reasoning_effort, and it sits at the top level of the body rather than inside chat_template_kwargs as some other models expect:

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain what a mixture-of-experts model is, simply, in 2 sentences."},
]

resp, dt = post({"model": MODEL, "messages": messages, "max_tokens": 1024,
                 "temperature": 0.8, "top_p": 1.0,
                 "reasoning_effort": "high"})     # low, high or max

Reading the reply is where the difference shows. The two halves come back on separate keys, so you can log the thinking and show the answer:

m = resp["choices"][0]["message"]
content = m.get("content") or ""
reasoning = m.get("reasoning_content") or ""

u = resp.get("usage", {})
ct = u.get("completion_tokens", 0)

print("--- thinking ---")
print(reasoning)              # the chain of thought lands here
print("--- answer ---")
print(content)                # the final answer lands here
print(f"    [{ct} tok in {dt:.1f}s = {ct / dt:.1f} tok/s]")

[2026-07-28 07:57:13] Kimi K3 supports thinking_effort low/high/max;
                      ignoring reasoning_effort='no_think'.

Thinking effort: what the knob actually changes

The interesting question is not whether the effort setting works, but what it moves. We put the same word problem through at low and at high and compared the token accounting:

prompt = ("Three friends split a bill. Ana pays twice what Ben pays, and Ben pays 3 more "
          "than Cara. The total is 43. How much does each pay?")

for effort in ("low", "high"):
    resp, dt = post({"model": MODEL, "max_tokens": 2000, "reasoning_effort": effort,
                     "messages": [{"role": "user", "content": prompt}]})
    u = resp.get("usage", {})
    print(f"  [{effort:>4}] reasoning_tokens={u.get('reasoning_tokens')} "
          f"completion={u.get('completion_tokens')} {dt:.1f}s")

The token counts move sharply, and the wall clock with them:

  [ low] reasoning_tokens=54 completion=202 33.6s
  [high] reasoning_tokens=184 completion=344 56.6s

The thinking itself is where the difference is visible. At low it is one line. At high the model restates the problem, solves it, checks the total and then considers whether non-integer amounts are acceptable:

# the whole of the low-effort thinking:
Cara = c, Ben = c+3, Ana = 2c+6. Total 4c+9 = 43, c=8.5. Ben 11.5, Ana 23. Total 43

# the opening of the high-effort thinking:
The user wants me to solve a word problem. Let me work through it.
Let Cara pay x. Ben pays 3 more than Cara: Ben = x + 3.
Ana pays twice what Ben pays: Ana = 2(x + 3) = 2x + 6.
Total: x + (x + 3) + (2x + 6) = 43 ...
Non-integer amounts, but that's fine (money can be .50).

# both answers identical: Cara 8.50, Ben 11.50, Ana 23.00

That is a clean result for capacity planning. The effort setting is a latency and cost dial, not a verbosity dial. At low the model wrote a single compressed line of working. At high it restated the problem, solved it, checked the total and then explicitly considered whether non-integer amounts were acceptable. On a problem this size the extra thinking bought nothing, which is exactly the sort of thing worth measuring on your own workload before you default the whole fleet to max.

Preserved thinking history, which is a requirement rather than an option

Kimi K3 was trained in preserved thinking history mode. The model card is direct about what happens otherwise: if the harness fails to pass back the historical thinking content, or if a session running on another model is switched over to K3 mid-conversation, generation quality may become highly unstable. In practice that means the assistant message you append to messages has to carry reasoning_content as well as content and tool_calls.

Here is a probe that can only be answered correctly if the previous turn's thinking survived the round trip. Turn one is an ordinary request:

msgs = [{"role": "user", "content": "Tell me three random numbers."}]

resp1, dt1 = post({"model": MODEL, "messages": msgs, "max_tokens": 1024,
                   "temperature": 0.8, "top_p": 1.0, "reasoning_effort": "high"})
m1 = resp1["choices"][0]["message"]

This next block is the one that matters. The assistant message goes back onto messages with its reasoning attached, rather than being rebuilt from content alone:

# pass the assistant message back AS-IS, including reasoning_content
assistant_turn = {"role": "assistant", "content": m1.get("content") or ""}
if m1.get("reasoning_content"):
    assistant_turn["reasoning_content"] = m1["reasoning_content"]
msgs.append(assistant_turn)

Turn two then asks about something that only ever existed inside that thinking:

msgs.append({"role": "user",
             "content": "What other numbers did you have in mind while thinking? Name them."})

resp2, dt2 = post({"model": MODEL, "messages": msgs, "max_tokens": 1024,
                   "temperature": 0.8, "top_p": 1.0, "reasoning_effort": "high"})
m2 = resp2["choices"][0]["message"]

print(f"    turn1: {(m1.get('content') or '')[:90]}")
print(f"    turn2: {(m2.get('content') or '')[:120]}")

    turn1: 17, 482, 96
    turn2: I didn't have specific alternates in mind, I just picked three.
           Three more random ones could be: 8, 157, 642.

The model read its own prior reasoning, correctly reported that it had not held back any alternatives, and offered three new numbers instead of inventing a memory. That is the behaviour you want, and it is the behaviour you lose if the assistant turn is rebuilt from content alone.

Agentic tool calling with the kimi_k3 parser

Because the server was launched with --tool-call-parser kimi_k3, the model decides for itself when to reach for a tool and the parser converts its output into standard OpenAI tool_calls. The first of the two tools is a calculator, and it walks the expression as an abstract syntax tree rather than calling eval, so anything other than arithmetic raises a ValueError rather than reaching a shell:

import ast
import operator

ops = {
    ast.Add:  operator.add,
    ast.Sub:  operator.sub,
    ast.Mult: operator.mul,
    ast.Div:  operator.truediv,
    ast.Pow:  operator.pow,
    ast.USub: operator.neg,
}

The calculator itself recurses over that tree. Anything that is not a number or one of the five permitted operators raises rather than evaluating:

def calculate(expression: str):
    def ev(n):
        if isinstance(n, ast.Constant):
            return n.value
        if isinstance(n, ast.BinOp):
            return ops[type(n.op)](ev(n.left), ev(n.right))
        if isinstance(n, ast.UnaryOp):
            return ops[type(n.op)](ev(n.operand))
        raise ValueError("bad expr")

    return {"result": ev(ast.parse(expression, mode="eval").body)}

The second tool is a stub, and a dispatch map ties the two names back to the two functions:

def get_weather(city: str):
    fake = {"Tokyo": "22C, clear", "London": "14C, rain", "Cairo": "35C, sunny"}
    return {"city": city, "weather": fake.get(city, "18C, partly cloudy")}

IMPL = {"calculate": calculate, "get_weather": get_weather}

Then describe the same two functions to the model. This is the standard OpenAI tools schema, with nothing Kimi-specific in it:

CALCULATE_TOOL = {
    "type": "function",
    "function": {
        "name": "calculate",
        "description": "Evaluate a basic arithmetic expression.",
        "parameters": {
            "type": "object",
            "properties": {
                "expression": {"type": "string", "description": "e.g. '47*89'"},
            },
            "required": ["expression"],
        },
    },
}

The second entry has the same shape, and the two go into the list the request will carry:

WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}

TOOLS = [CALCULATE_TOOL, WEATHER_TOOL]

The loop itself is the usual pattern. Send the task, and stop as soon as a reply comes back without tool calls, because that reply is the answer:

task = ("What is 47 * 89? Also, what's the weather in Tokyo? "
        "Use your tools, then give me one combined answer.")
messages = [{"role": "system", "content": "You are a helpful assistant with tools."},
            {"role": "user", "content": task}]

for turn in range(6):
    resp, dt = post({"model": MODEL, "messages": messages, "tools": TOOLS,
                     "max_tokens": 1024, "temperature": 0.7, "reasoning_effort": "high"})
    m = resp["choices"][0]["message"]
    tcs = m.get("tool_calls") or []
    print(f"[turn {turn}] tool_calls={len(tcs)} "
          f"reasoning={len(m.get('reasoning_content') or '')} chars")

    if not tcs:
        print(m.get("content", ""))
        break

If there were tool calls, the assistant turn goes back onto messages first, and it carries the same three things every time:

    # echo the assistant message back AS-IS (content + reasoning_content + tool_calls)
    at = {"role": "assistant", "content": m.get("content") or "", "tool_calls": tcs}
    if m.get("reasoning_content"):
        at["reasoning_content"] = m["reasoning_content"]
    messages.append(at)

Then each requested tool runs and its result is appended as a tool message, which is what the model reads on the next pass:

    for tc in tcs:
        fn = tc["function"]["name"]
        try:
            args = json.loads(tc["function"].get("arguments") or "{}")
            result = IMPL[fn](**args)
        except Exception as e:
            args, result = {}, {"error": str(e)}

        print(f"    -> {fn}({args}) = {result}")
        messages.append({"role": "tool", "tool_call_id": tc.get("id", fn),
                         "name": fn, "content": json.dumps(result)})

Kimi K3 requested both tools in a single turn rather than serialising them into two round trips. Its reasoning for that turn, 116 characters of it, reads: "The user wants 47*89 and the weather in Tokyo. I should call both tools in the same block since they are independent."

[turn 0] tool_calls=2 reasoning=116 chars
    -> calculate({'expression': '47*89'}) = {'result': 4183}
    -> get_weather({'city': 'Tokyo'}) = {'city': 'Tokyo', 'weather': '22C, clear'}
[turn 1] tool_calls=0 reasoning=37 chars

Here are both results:

- 47 x 89 = 4,183
- Tokyo weather: 22C and clear

So, 47 times 89 equals 4,183, and it's currently a pleasant clear day at 22C in Tokyo!

The whole exchange took 36.25 seconds across two round trips. Parallel tool calls in one turn are the behaviour you want from an agentic model, and the kimi_k3 parser delivers it with no orchestration code of your own.

Native vision, tested against an image we generated ourselves

Kimi K3 is natively multimodal through MoonViT-V2, so images go into the same endpoint as text. Rather than describe a photograph and grade the answer by eye, we built the test image in the script: a 128 by 128 PNG whose left half is pure red and whose right half is pure blue, written byte by byte with nothing but the standard library. The correct answer is therefore known before the request is sent.

import base64
import struct
import zlib

def make_png(w=128, h=128):
    rows = b""
    for _ in range(h):
        row = b"\x00"                                    # filter byte
        for x in range(w):
            row += b"\xff\x00\x00" if x < w // 2 else b"\x00\x00\xff"
        rows += row

    def chunk(tag, data):
        c = struct.pack(">I", len(data)) + tag + data
        return c + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)

    ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)    # 8-bit truecolour
    return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
            + chunk(b"IDAT", zlib.compress(rows, 9)) + chunk(b"IEND", b""))

Sending it is the same call as any other, with the image as a second content part rather than a separate endpoint:

b64 = base64.b64encode(make_png()).decode()

resp, dt = post({"model": MODEL, "max_tokens": 400, "reasoning_effort": "low",
                 "messages": [{"role": "user", "content": [
                     {"type": "text", "text": "What do you see in this image? "
                                              "Name the colours and how they are arranged."},
                     {"type": "image_url",
                      "image_url": {"url": f"data:image/png;base64,{b64}"}}]}]})

ans = (resp["choices"][0]["message"].get("content") or "").strip()
print(f"  latency {dt:.1f}s | usage={resp.get('usage')}")
print(f"  ANSWER: {ans[:300]}")

  latency 18.4s | usage={'prompt_tokens': 174, 'total_tokens': 272,
                          'completion_tokens': 98, 'reasoning_tokens': 54,
                          'prompt_tokens_details': {'cached_tokens': 0, 'image_tokens': 25}}
  ANSWER: The image shows two solid color blocks arranged side by side: the left half
          is red and the right half is blue, meeting along a sharp vertical line down
          the center.

That answer is objectively correct, including the vertical boundary down the centre. The usage block is the interesting part for anyone budgeting a vision workload: the whole image cost 25 image tokens, and the request completed in 18.4 seconds. Vision on Kimi K3 is not a bolt-on adapter, and it is priced in tokens like everything else.

Throughput on 32 NVIDIA H100 GPUs

Single-stream decode across our three chat prompts averaged 5.8 tokens per second, with individual prompts at 5.9, 5.8 and 5.6. The more useful question for a sparse MoE with room for 49 concurrent requests is what happens when requests arrive together, so we fired four at once. The script is standard library only, so it runs anywhere:

import json
import threading
import time
import urllib.request

BASE = "http://<HEAD_NODE_PUBLIC_IP>:8000/v1"
MODEL = "moonshotai/Kimi-K3"
N = 4

One worker sends one request and records what it got back:

def one(i, res):
    body = {"model": MODEL, "max_tokens": 128, "temperature": 0.9,
            "messages": [{"role": "user",
                          "content": f"Write one interesting paragraph about topic #{i}: the deep ocean."}]}
    req = urllib.request.Request(
        BASE + "/chat/completions", data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json", "Authorization": "Bearer EMPTY"})

    t0 = time.time()
    with urllib.request.urlopen(req, timeout=600) as r:
        resp = json.load(r)

    res[i] = {"completion_tokens": resp["usage"]["completion_tokens"],
              "latency_s": time.time() - t0}

Then start all four at once and time the batch as a whole, which is the number that matters for a server rather than for a single user:

res = {}
ths = [threading.Thread(target=one, args=(i, res)) for i in range(N)]

t0 = time.time()
for t in ths:
    t.start()
for t in ths:
    t.join()
wall = time.time() - t0

toks = sum(v["completion_tokens"] for v in res.values())
agg = toks / wall
print(json.dumps({"concurrency": N, "wall_s": round(wall, 2),
                  "total_completion_tokens": toks,
                  "aggregate_tok_per_s": round(agg, 1),
                  "per_stream_tok_per_s": round(agg / len(res), 1)}, indent=2))

{
  "concurrency": 4,
  "wall_s": 36.55,
  "total_completion_tokens": 512,
  "aggregate_tok_per_s": 14.0,
  "per_stream_tok_per_s": 3.5
}

What sets that figure, and what would move it

Those numbers come from a bring-up configuration, and every choice in it is a deliberate trade for a fast, cheap start rather than for peak throughput:

• The MoE path is Marlin. Every expert matrix multiply dequantises MXFP4 into 16 bits on the way through, because the native MXFP4 backends target Blackwell.
• The cards are PCIe rather than SXM. NVLink bridges pairs of cards, and the remaining hops inside each node go through the PCIe host bridge.
• Collectives run over Ethernet. Twenty-four of the thirty-two ranks sit on a different machine from rank 0, so their all-reduces cross the private network.
• Speculative decoding is off. Kimi K3 ships a draft path that the published recipes pair with the model, and we ran without it to keep the launch minimal.
• Every request thinks. Kimi K3 has no no-think mode, so even a one-line answer pays for a reasoning pass. On our first request that was 109 of 135 tokens.

For scale at the other end of the range, the vLLM launch blog reports 111 tokens per second per user on TP 8 and 118 on TP 16 at batch size 1, rising to 331 and 370 with DSpark speculative decoding, measured on NVIDIA GB300 NVL72. That is a different hardware class, with native FP4 kernels and NVLink across the whole domain, and the distance between the two is a fair picture of what each of the trades above is worth.

So this is the shape of deployment to reach for when the job is evaluation, integration and correctness work: proving an agent harness against the model itself, measuring how your prompts behave at each thinking effort, testing vision inputs, and validating the tool-call plumbing end to end, on hardware you can book by the minute. When throughput becomes the requirement, the levers in order of effect are more memory per card, then native FP4, then speculative decoding.

Cost and Tearing Down the Cluster

At $2.00 per GPU per hour on Spot, thirty-two NVIDIA H100 GPUs cost $64.00 per hour. Here is where that hour goes on a clean run that follows this guide:

The download dominates, at about 62 per cent of the bill, and it is the one phase that does not get cheaper with faster GPUs. Everything after it is minutes. Add whatever time you intend to spend actually using the endpoint on top, at roughly $1.07 per minute.

When you are finished, stop the containers and delete the virtual machines. This is the step you must not skip:

# On each node, stop the container first so the NVIDIA GPUs are released cleanly
docker rm -f sgl

# Then delete all four virtual machines from the Hyperstack dashboard,
# or through the API, and confirm they have gone.
curl -s -X DELETE https://infrahub-api.nexgencloud.com/v1/core/virtual-machines/$VM_ID \
  -H "api_key: $HYPERSTACK_API_KEY"

Serve an open 3T-class model

Run Kimi K3 across thirty-two NVIDIA H100 GPUs

Four nodes on Spot at $64.00 per hour. Our run downloaded 1.56 TB, launched 32 ranks and reached a live one-million-token endpoint eleven minutes after the containers started.

32x NVIDIA H100TP 32 x EP 321M contextReady in 11 minutes

Launch an NVIDIA GPU cluster on Hyperstack today.

FAQs

What hardware do you need to run Kimi K3?

The checkpoint is 1,560.94 GB across 96 shards and every expert has to stay resident, so the constraint is total GPU memory rather than compute. The official vLLM recipe asks for at least one 8x NVIDIA B300 node. This guide serves it on 32x NVIDIA H100 80GB instead, across four Hyperstack nodes.

What is the maximum tensor-parallel size for Kimi K3?

Thirty-two. Tensor parallelism has to divide both the 96 attention heads and the 7168 hidden size, and their greatest common divisor is 32. Adding a fifth node therefore cannot lower the weights each GPU carries.

Why does the Kimi K3 endpoint always return reasoning_content?

Kimi K3 always thinks: it accepts an effort of low, high or max, and nothing turns thinking off. Launching with --reasoning-parser kimi_k3 puts that thinking in its own field, and you have to pass it back on the next turn or later replies become unstable.

How fast is Kimi K3 on 32x NVIDIA H100?

We measured 5.8 tokens per second single stream and 14.0 aggregate at four-way concurrency, on the Hopper Marlin path without speculative decoding. For contrast, the vLLM launch blog reports 111 per user on TP 8 at batch size 1, measured on NVIDIA GB300 NVL72.

How much does it cost to run Kimi K3 on Hyperstack?

Four 8x NVIDIA H100 Spot nodes are $2.00 per GPU per hour, so $64.00 per hour for the 32-GPU cluster. A clean run reaches a live endpoint in about 52 minutes, or roughly $55. Delete the machines afterwards, because the ephemeral disk holding the weights goes with them.