Run DeepSeek OCR on Hyperstack with your Own UI

Qwen3.8 Max is a 2.4 trillion parameter mixture-of-experts model from the Qwen team, and 95 billion of those parameters are awake for any one token. Alibaba put the hosted version in front of developers on 3 August 2026 and published the weights shortly afterwards, which is the first time a Qwen Max class model has been released for anyone to run on their own hardware. The FP8 checkpoint is 2.50 TB across 213 safetensors shards, so a Qwen3.8 Max deployment is a multi-node exercise before it is anything else.

This guide deploys it on Hyperstack across eight nodes of eight NVIDIA H100 80GB PCIe GPUs: eight virtual machines, 64 GPUs in total, running SGLang with tensor parallel 16 by pipeline parallel 4. Every figure below comes from one session on 12 August 2026 that pulled 2.50 TB of weights, reached a live OpenAI-compatible endpoint in under six minutes of engine start-up, and then answered chat, reasoning, tool-calling and concurrency workloads with zero failed requests across the sweep.

There is one thing worth settling before the hardware. The hosted Qwen3.8 Max API does not publish an activated parameter count, which leaves anyone modelling cost to guess at the compute behind a token. The open checkpoint states it in config.json: 95 billion active, ten routed experts of 512 plus one shared expert, 92 layers. That is what makes a Qwen3.8 Max deployment something you can size and price rather than estimate. If you have followed our Kimi K3 multi-node deployment, this is the same class of exercise with a different bottleneck, and the reasons are arithmetic rather than memory.

Inside Qwen3.8 Max: 2.4 Trillion Parameters, 95 Billion Awake

Qwen3.8 Max is a sparse model with a hybrid attention stack. Both halves of that sentence change how you deploy it. Take them one at a time before booking any hardware.

Sparsity Is What Makes 2.4 Trillion Parameters Servable

Every parameter has to be resident in GPU memory, because the router picks its experts per token and per layer, and there is no useful way to predict which ones. What sparsity buys is compute rather than capacity. The forward pass touches 95 billion parameters, so the cluster is sized by the weight of the model and the speed is set by a far smaller number.

The Hybrid Attention Stack Sets the Concurrency Limit

The 92 layers are not uniform. Three out of every four use Gated DeltaNet, a linear attention layer that carries a fixed-size recurrent state per sequence. The fourth is gated full attention with a conventional key-value cache. The configuration expresses this as full_attention_interval: 4, which gives 69 linear layers and 23 full attention layers.

The practical consequence arrives at serving time. A conventional transformer runs out of concurrency when the key-value cache fills, so the limit moves with context length. Here the recurrent state is a fixed cost per sequence regardless of how long that sequence becomes, and it is the larger of the two pools. When the engine reported its capacity, the ceiling on concurrent requests came from the recurrent state pool rather than from the key-value cache, and that shapes the whole throughput picture further down.

What the Published Scores Say

Qwen positions this release around long-horizon agentic work, coding and research rather than raw single-turn question answering, and the published headline scores follow that emphasis.

The Qwen3.8 Max Deployment Architecture: Why 64 NVIDIA H100 GPUs

Sizing a cluster for a 2.50 TB checkpoint looks like a division problem and is not one. Memory rules out the small clusters, and then a second constraint rules out two more that memory was perfectly happy with. Getting to the right answer takes both filters, in that order.

The First Filter Is Memory, and It Is the Forgiving One

Weights are only part of what a GPU has to hold. The recurrent state pool, the key-value cache and the captured CUDA graphs all come out of the same 80 GB, and whatever is left is what serves the running batch. We sized this deployment against a floor of roughly 25 GB free per GPU after weights, which turns the question from a theoretical lower bound into a usable one.

Four nodes fail this outright. The weights alone would need 78.1 GB of an 80 GB card, which leaves nothing for anything else. Five nodes clear the weights and then fall under the free-memory floor. Six and seven nodes pass comfortably.

The Second Filter Is Divisibility, and It Is the Strict One

A parallel placement is only legal if the numbers divide. The pipeline degree has to divide the layer count evenly, and the tensor degree has to divide the attention head count and the expert count. Applying that to six and seven nodes removes both, even though both fit the weights with room to spare.

The arithmetic behind that is short enough to check by hand. Qwen3.8 Max has 92 layers, and 92 factorises as 2 squared times 23, so a pipeline degree has to be one of 1, 2, 4, 23, 46 or 92. It also has to divide the GPU count. On 48 GPUs that leaves pipeline degrees of 1, 2 and 4, giving tensor degrees of 48, 24 and 12. None of those divides 64 query heads. On 56 GPUs the tensor degrees are 56, 28 and 14, and again none divides 64. Six and seven node clusters are not tight, they are impossible.

Why Pipeline Parallel 4 and Tensor Parallel 16

Three placements are legal on 64 GPUs: tensor 64 by pipeline 1, tensor 32 by pipeline 2, and tensor 16 by pipeline 4. The choice between them is about where the traffic goes.

A tensor-parallel group communicates on every layer, so its members want to be as close together as possible. At tensor 64 the group spans all eight machines, and every collective in every layer crosses the network across 64 ranks. At tensor 16 the group spans two machines, and the four pipeline stages exchange activations only at the three stage boundaries. Twenty-three layers per stage divides evenly. The SGLang cookbook lists a verified four-node NVIDIA H200 placement for this same FP8 checkpoint, which is the same weights on cards large enough to need half as many GPUs. Our guide to tensor and pipeline parallelism covers the general trade-off between the two axes if you want the background.

What the Fabric Decides About Your Launch Flags

The launch command carries four settings that exist because of how the GPUs are wired, not because of anything about Qwen3.8 Max. Reading the topology first makes all four obvious.

      GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0
GPU0   X    NV12  PHB   PHB   PHB   PHB   PHB   PHB   PHB
GPU1  NV12   X    PHB   PHB   PHB   PHB   PHB   PHB   PHB
GPU2  PHB   PHB    X    NV12  PHB   PHB   PHB   PHB   PHB
GPU3  PHB   PHB   NV12   X    PHB   PHB   PHB   PHB   PHB
...
NIC0  PHB   PHB   PHB   PHB   PHB   PHB   PHB   PHB    X

  NV#  = Connection traversing a bonded set of # NVLinks
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge

A tensor-parallel group of 16 covers eight cards on one machine and eight on another, so the group is a multi-node group by definition. Two single-node optimisations then switch themselves off, and the engine says so during startup.

[18:15:14 PP0 TP0] Init torch distributed begin.
[18:15:40 PP0 TP0] CustomAllreduce is disabled because this process group
                   spans across nodes.
[18:15:40 PP0 TP0] Init torch distributed ends. elapsed=26.60 s, mem usage=0.47 GB
[18:15:46 PP0 TP0] multimem all-gather disabled because the TP group spans
                   across nodes.

Neither line calls for action. Both are the engine selecting the portable path because the fast path is only correct within one machine. What does call for action is telling the collectives which interface to use, because a multi-homed node will otherwise pick the wrong one and the rendezvous will not complete.

How to Deploy Qwen3.8 Max on Hyperstack, Step by Step

Nine steps take a Qwen3.8 Max deployment from an empty account to an endpoint answering requests. Everything up to Step 5 is ordinary virtual machine work repeated eight times. The launch itself is Steps 6 and 7, and Step 7 is where the flags specific to this model appear.

Step 1: Create Eight Virtual Machines

From the Hyperstack dashboard, open the virtual machines page and start a new deployment. The getting started guide covers the account setup if this is a first deployment.

Start from the virtual machines page and select deploy a new virtual machine.

Four choices matter, and one of them matters more than the rest.

The image picker. Choosing the CUDA and Docker variant removes every driver step from this deployment.

Step 2: Open the Three Ports That Matter

A multi-node deployment needs the nodes to talk to each other freely and needs you to talk to exactly one of them. Attach these firewall rules to every machine.

Step 3: Point the Model Cache at the Ephemeral Disk

Once the machines are active, connect to each one and take stock. The root disk is 96 GB, which is not where 2.50 TB of weights is going. This flavour ships an ephemeral NVMe disk mounted at /ephemeral, and that is the destination.

# From your workstation, once the eight virtual machines are ACTIVE
ssh -i ~/.ssh/[YOUR KEY] ubuntu@[NODE PUBLIC IP]

# Confirm what you are standing on before anything else
nvidia-smi --query-gpu=index,name,memory.total --format=csv
ip -o -4 addr show | awk '{print $2, $4}' | grep ' 10\.'
df -h / /ephemeral

index, name, memory.total [MiB]
0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
...
7, NVIDIA H100 PCIe, 81559 MiB

ens6 10.0.0.210/24

Every card reports 81,559 MiB and the private interface is ens6, holding a 10.x address. The third command is the one that decides where the weights go: the root disk is 96 GB, so /ephemeral has to take the 2.50 TB checkpoint. Confirm it has the room before starting the download.

# The root disk is 96 GB and the checkpoint is 2.50 TB, so the cache
# has to live on the ephemeral NVMe disk. Do this on all eight nodes.
sudo mkdir -p /ephemeral/hf
sudo chown -R ubuntu:ubuntu /ephemeral/hf

python3 -m pip install -q --break-system-packages huggingface_hub hf_transfer

Step 4: Pull the Serving Image and the Weights

The container image and the checkpoint can be fetched at the same time, and on eight nodes both should be started in the background so a dropped SSH session does not take the download with it. The image comes from the SGLang project and carries the kernels this model needs.

# Pull the serving image in the background while the weights download.
# Images stay on the root disk; only the checkpoint goes to /ephemeral.
nohup docker pull lmsysorg/sglang:qwen38 > /tmp/pull.log 2>&1 &

The weights need a Hugging Face token with access to the Qwen3.8-2.4T-A95B-FP8 repository. Create one from your account settings and keep it out of your shell history.

# 2.50 TB across 213 safetensors shards. hf_transfer keeps the NVMe disk fed;
# 32 workers saturated the link on our nodes without stalling.
export HF_HOME=/ephemeral/hf
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_TOKEN=[YOUR HUGGING FACE TOKEN]

nohup python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('Qwen/Qwen3.8-2.4T-A95B-FP8', max_workers=32)
" > /tmp/dl.log 2>&1 &

Step 5: Confirm Every Node Before You Commit to a Launch

A 64-rank job starts by having all 64 ranks meet each other. Every rank waits for the others, so the cluster is only as ready as its least ready node. Checking takes about twenty seconds per machine, and it belongs immediately before the launch rather than earlier in the session.

# Run this on every node and read the number, not the vibe. A shard that is
# still a .incomplete temporary file fails the launch twenty minutes in,
# after the rendezvous has already committed the other 63 ranks.
SHARDS=$(find /ephemeral/hf -name '*.safetensors' | wc -l)
PARTIAL=$(find /ephemeral/hf -name '*.incomplete' | wc -l)
echo "$(hostname): $SHARDS/213 shards, $PARTIAL incomplete"

q38-0: 213/213 shards, 0 incomplete
q38-1: 213/213 shards, 0 incomplete
q38-2: 213/213 shards, 0 incomplete
q38-3: 213/213 shards, 0 incomplete
q38-4: 213/213 shards, 0 incomplete
q38-5: 213/213 shards, 0 incomplete
q38-6: 213/213 shards, 0 incomplete
q38-7: 213/213 shards, 0 incomplete

The shard count is the check that pays for itself most often, because a download that stopped short looks completely normal until the engine reaches the missing file twenty minutes into loading. Three more checks round it out.

# One last look at every node, immediately before the launch rather than
# minutes before it. Twenty seconds here protects a 64-rank rendezvous.
timeout 25 nvidia-smi -L | grep -c '^GPU'          # expect 8
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader | sort -u
ip -o -4 addr show | awk '$4 ~ /^10\./ {print $2; exit}'
find /ephemeral/hf -name '*.safetensors' | wc -l   # expect 213

Step 6: Set the Launch Environment

Four values differ per node. Everything else in the launch is identical across the cluster, which is what makes this straightforward to script.

# Run on every node. RANK is 0 on the head node, then 1 through 7.
RANK=[THIS NODE RANK]
SELF=[THIS NODE PRIVATE IP]
HEAD=10.0.0.210          # the private address of rank 0
NIC=ens6                 # the interface holding the 10.x address

The head address is the private address of rank 0 and is the same on every node. The rank is the one value that has to be unique across the cluster, so set it from the machine name rather than by hand if you are scripting this.

Step 7: Launch All Eight Ranks

The arguments split into three groups: the topology, the backends the hardware calls for, and the parsers that shape the responses. Building them into one variable keeps the launch readable and keeps every node running the same string.

# The topology and the two backend choices that follow from the hardware
Q38_ARGS="--trust-remote-code \
  --model-path Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tp-size 16 --pp-size 4 --nnodes 8 \
  --node-rank $RANK \
  --dist-init-addr $HEAD:20000 --dist-timeout 1800 \
  --linear-attn-prefill-backend flashinfer \
  --linear-attn-decode-backend flashinfer \
  --mamba-full-memory-ratio 0.95 \
  --mamba-ssm-dtype bfloat16 \
  --mamba-radix-cache-strategy extra_buffer \
  --max-prefill-tokens 8192 --page-size 64 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --host 0.0.0.0 --port 30000"

The container itself needs host networking so the ranks can reach each other on the private addresses, the checkpoint mounted from the ephemeral disk, and enough shared memory for the loader. Run this on all eight nodes.

docker rm -f q38 2>/dev/null || true

docker run -d --name q38 --gpus all --network host --ipc=host \
  --shm-size 32g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$(cat $HOME/hf_token.txt)" \
  --env SGLANG_HOST_IP=$SELF \
  --env NCCL_SOCKET_IFNAME=$NIC \
  --env GLOO_SOCKET_IFNAME=$NIC \
  --env NCCL_IB_DISABLE=1 \
  --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  lmsysorg/sglang:qwen38 \
  sglang serve $Q38_ARGS

Step 8: Watch the Start-up

The engine reports its memory at five points, and those five numbers are the clearest signal that a launch is going well. Follow the head node.

# On the head node. The interesting lines are the memory checkpoints.
docker logs -f q38 2>&1 | grep -E "Init torch|Load weight|Cache is allocated|\
Memory pool|CUDA graph (begin|end)|max_total_num_tokens|Uvicorn"

The ranks meet first. Twenty-six seconds for 64 processes to join one group is normal, and the two lines about collectives are the engine adapting to a group that spans machines rather than anything going wrong.

[18:15:44 PP0 TP0] Load weight begin. avail mem=78.05 GB
Multi-thread loading shards: 100% | 213/213 [00:41<00:00, 16.52it/s]
[18:16:29 PP0 TP0] Load weight end. elapsed=45.02 s, type=Qwen3_5MoeForCausalLM,
                   quant=fp8, avail mem=41.73 GB, mem usage=36.32 GB.

Free memory falling from 78.05 GB to 41.73 GB is the moment the deployment becomes real. That 36.32 GB per GPU is the FP8 checkpoint spread across 64 cards. Our first load took 5 minutes 38 seconds reading cold from disk. On a relaunch with the page cache still warm the first rank finished in 45 seconds and the last in 55, and it is the slowest rank that decides when the cluster moves on.

[18:16:45 PP0 TP0] max_running_requests is capped to 541 by the mamba state
                   cache (max_mamba_cache_size=2164, 4 state slots per request).
[18:16:46 PP0 TP0] Mamba Cache is allocated. max_mamba_cache_size: 2164,
                   conv_state size: 0.28GB, ssm_state size: 9.51GB
[18:16:46 PP0 TP0] KV Cache is allocated. dtype: torch.bfloat16,
                   #tokens: 1803072, K size: 4.30 GB, V size: 4.30 GB
[18:16:46 PP0 TP0] Memory pool end. avail mem=22.78 GB

This is where the hybrid attention stack shows up in the accounting. The recurrent state pool takes 9.79 GB and the key-value cache takes 8.60 GB, and it is the recurrent pool that sets the ceiling on concurrent requests. The engine states the limit and the arithmetic behind it in one line: 2,164 state slots, four slots per request, so 541 requests can run at once.

[18:19:48 PP0 TP0] Capture target prefill CUDA graph end. elapsed=182.25 s,
                   mem usage=3.87 GB, avail mem=18.76 GB.
[18:20:27 PP0 TP0] Capture target decode CUDA graph end. elapsed=38.97 s,
                   mem usage=1.42 GB, avail mem=17.34 GB.
[18:20:27 PP0 TP0] max_total_num_tokens=1803072, chunked_prefill_size=8192,
                   max_prefill_tokens=8192, max_running_requests=541,
                   context_len=262144, available_gpu_mem=17.34 GB
[18:20:28] INFO:     Uvicorn running on http://0.0.0.0:30000

Capturing CUDA graphs is the longest phase of a warm start. The prefill capture walks 58 different token counts and the decode capture walks 52 batch sizes, and together they account for more than half the time between launching the container and answering a request.

Engine startup timings (s): load_weight=54.98, kv_cache_allocation=1.63,
  scheduler_e2e=317.47,
  cuda_graph={prefill=182.60, decode=38.97, target_verify=0.00},
  tokenizer_e2e=328.19

Step 9: Send the First Request

The endpoint speaks the OpenAI chat completions API on port 30000 of the head node. No other rank needs to be reachable from outside the environment.

# From your workstation, against the head node
curl -s http://[HEAD NODE IP]:30000/v1/models | head -c 200

curl -s http://[HEAD NODE IP]:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3.8-2.4T-A95B-FP8",
    "messages": [{"role": "user", "content": "Reply with exactly: OK"}],
    "max_tokens": 512
  }'

While that runs, the cluster looks like this. Roughly 63,400 MiB in use of the 81,559 MiB on each card, at low temperature, because a mixture-of-experts model at low concurrency waits on memory traffic rather than saturating the compute.

index, name, memory.used [MiB], memory.total [MiB], temperature.gpu, power.draw [W]
0, NVIDIA H100 PCIe, 63552 MiB, 81559 MiB, 26, 76.01 W
1, NVIDIA H100 PCIe, 63396 MiB, 81559 MiB, 24, 78.40 W
2, NVIDIA H100 PCIe, 63396 MiB, 81559 MiB, 36, 84.20 W
...
7, NVIDIA H100 PCIe, 63490 MiB, 81559 MiB, 25, 78.86 W

Qwen3.8 Max Inference: Reading What Comes Back

Qwen3.8 Max reasons before it answers, on every request. That single fact changes how a client has to be written, and it is the part of a Qwen3.8 Max implementation most likely to catch a team that has ported code from a model without chain of thought.

The Chain of Thought Arrives in Its Own Field

Because the server was launched with --reasoning-parser qwen3, the reasoning does not appear inline in the answer wrapped in tags. It comes back in message.reasoning_content, and the answer in message.content is clean. A client can log the reasoning, show it, or discard it, without parsing anything.

import json
import urllib.request

BASE = "http://[HEAD NODE IP]:30000/v1"
MODEL = "Qwen/Qwen3.8-2.4T-A95B-FP8"


def ask(prompt, max_tokens=4096):
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.7,
        "top_p": 0.8,
        "max_tokens": max_tokens,
    }
    request = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request, timeout=1800) as response:
        return json.load(response)

def show(prompt, max_tokens=4096):
    payload = ask(prompt, max_tokens)
    choice = payload["choices"][0]
    message = choice["message"]
    usage = payload["usage"]

    # The qwen3 reasoning parser puts the chain of thought in its own field,
    # so the answer arrives clean and the reasoning can be logged or dropped.
    thinking = message.get("reasoning_content") or ""
    answer = message.get("content") or ""

    print("finish_reason  :", choice["finish_reason"])
    print("reasoning      :", usage["reasoning_tokens"], "tokens")
    print("completion     :", usage["completion_tokens"], "tokens")
    print("answer         :", len(answer), "characters")
    return answer

The usage block carries a reasoning_tokens field alongside completion_tokens, which is the number to watch. Reasoning is not billed separately in a self-hosted deployment, but it does consume the same budget as the answer.

Reasoning and the Answer Share One Token Budget

This is the behaviour to design around. max_tokens covers the reasoning and the answer together, not the answer alone. We sent the same code prompt at two budgets.

{
  "prompt_tokens": 112,
  "completion_tokens": 2048,
  "reasoning_tokens": 1871,
  "total_tokens": 2160
}

{
  "prompt_tokens": 111,
  "completion_tokens": 6144,
  "reasoning_tokens": 6144,
  "total_tokens": 6255
}

At 6,144 the model spent the entire allowance thinking and emitted no answer tokens at all. A budget sized for a model without chain of thought does more than clip the tail here. It can return an empty response, or code that looks complete and is not.

# A budget sized for a model without chain of thought can return nothing at
# all here, because reasoning and the answer share one allowance. Treat a
# length stop as a retry signal rather than as a finished response.
answer = show("Write a Python function chunked(iterable, n) that yields "
              "lists of length n, lazily, raising ValueError if n < 1.",
              max_tokens=4096)

if not answer.strip():
    print("empty answer: raise max_tokens or lower reasoning_effort")

Reasoning Effort Is the Control Worth Wiring Up

The Qwen3.8-2.4T-A95B model card documents three reasoning levels, low, medium and xhigh, with xhigh as the default. Every measurement below was taken at the default, which is the most expensive setting and the right one to measure against. It is also the first lever to reach for when a workload does not need deep deliberation on every turn.

# reasoning_effort is the documented control. The default is xhigh, which is
# what every measurement above was taken at.
{
  "model": "Qwen/Qwen3.8-2.4T-A95B-FP8",
  "messages": [{"role": "user", "content": "..."}],
  "reasoning_effort": "low",
  "max_tokens": 4096
}

The shape of that chart is the useful part. On a two-sentence factual question the model spent 105 tokens thinking and 65 answering. On a code task it spent 1,871 thinking and 177 answering. Reasoning grows with difficulty far faster than the answer does, so a fixed budget that suits simple traffic will fail on hard traffic rather than degrade on it.

Tool Calling Works End to End

The qwen3_coder parser turns the native tool syntax of this model into OpenAI-shaped tool_calls. We tested it with a loop that executes the tools and feeds each result back, because a broken parser still produces plausible-looking first-turn output.

TOOLS = [{
    "type": "function",
    "function": {
        "name": "plan_cluster",
        "description": "Smallest cluster that leaves more than headroom_gb "
                       "free per GPU and admits a legal (tp, pp) split.",
        "parameters": {
            "type": "object",
            "properties": {
                "weights_tb": {"type": "number"},
                "vram_gb": {"type": "number"},
                "gpus_per_node": {"type": "integer"},
                "layers": {"type": "integer"},
                "headroom_gb": {"type": "number"},
            },
            "required": ["weights_tb", "vram_gb", "gpus_per_node", "layers"],
        },
    },
}]

messages = [
    {"role": "system", "content": "You are a deployment engineer. Call a tool "
                                  "for every factual number."},
    {"role": "user", "content": TASK},
]

for turn in range(8):
    payload = chat(messages, TOOLS)
    message = payload["choices"][0]["message"]
    calls = message.get("tool_calls") or []

    # Echo the assistant turn back so the model sees its own calls
    messages.append({k: v for k, v in message.items()
                     if k != "reasoning_content"})
    if not calls:
        print(message["content"])
        break

    for call in calls:
        name = call["function"]["name"]
        args = json.loads(call["function"]["arguments"] or "{}")
        result = IMPLEMENTATIONS[name](**args)
        messages.append({"role": "tool", "tool_call_id": call["id"],
                         "name": name, "content": json.dumps(result)})

The task was to size a cluster for this exact model using only tool results. It made four dependent calls in the right order, taking each argument from the previous result, across eight turns and 54.6 seconds.

lookup_model {"name": "qwen3.8-fp8"}
  -> {"weights_tb": 2.5, "layers": 92, "total_params_b": 2400}

lookup_gpu   {"flavor": "h100-80g"}
  -> {"vram_gb": 80, "gpus_per_node": 8, "usd_per_hr_node": 16.0}

plan_cluster {"weights_tb": 2.5, "vram_gb": 80, "gpus_per_node": 8,
              "layers": 92, "headroom_gb": 25}
  -> {"nodes": 8, "gpus": 64, "weights_gb_per_gpu": 39.1,
      "free_gb_per_gpu": 40.9,
      "legal_splits": [{"tp": 64, "pp": 1}, {"tp": 32, "pp": 2},
                       {"tp": 16, "pp": 4}],
      "rejected_smaller_clusters": [
        {"nodes": 6, "free_gb_per_gpu": 27.9,
         "why": "fits in memory but no tp dividing 64 heads"},
        {"nodes": 7, "free_gb_per_gpu": 35.4,
         "why": "fits in memory but no tp dividing 64 heads"}]}

cost         {"nodes": 8, "usd_per_hr_node": 16.0, "hours": 3}
  -> {"usd_per_hr_cluster": 128.0, "usd_total": 384.0}

The final answer then reproduced the non-obvious part without being prompted for it: six and seven node clusters fit the weights in memory and admit no legal placement, so they are infeasible. That is the same conclusion the architecture section reached, arrived at independently from the tool output.

It Gets the Arithmetic Right

Two of the prompts had ground truth we could check rather than eyeball. Asked for the annual cost of a 6.5 kW rack at $0.11 per kWh with cooling billed at 40 per cent of the power bill, it returned $6,263.40 for power, $2,505.36 for cooling and $8,768.76 in total, which matches an independent calculation exactly. Asked how 92 layers can be split evenly across pipeline stages, it returned the full set.

The possible numbers of pipeline stages are 2, 4, 23, 46 and 92.

Let s be the number of stages. For exactly equal layers per stage, 92 / s
must be an integer, so s must divide 92. The prime factorisation is
92 = 2^2 x 23, so its positive divisors are 1, 2, 4, 23, 46 and 92.
Excluding a single stage leaves 2, 4, 23, 46, 92.

Throughput, and What Sets It

We swept concurrency from one request to 32 against the live endpoint, with a 256 token cap on each, and measured what came back.

python3 q38_bench.py \
  --base http://[HEAD NODE IP]:30000/v1 \
  --model Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --levels 1,4,16,32 --max-tokens 256 --out ./bench

 conc   ok   wall_s  agg tok/s  per-req t/s     p50     p95
    1    1    22.42       10.9         10.9   22.42   22.42
    4    4    32.57       30.8          7.8   32.31   32.56
   16   16    87.45       45.5          2.9   86.82   87.44
   32   32    95.83       81.6          2.7   95.28   95.83

== peak aggregate 81.6 tok/s at concurrency 32

The Recurrent State Pool Is the Ceiling, Not the Context Length

On a conventional transformer, concurrency runs out when the key-value cache fills, so long prompts and many users compete for the same resource. Here the engine reported a limit of 541 concurrent requests set by the recurrent state pool, from 2,164 state slots at four slots per request. That limit does not move with prompt length, because a Gated DeltaNet layer carries the same fixed state whether the sequence is a thousand tokens or a hundred thousand.

Two flags move it. Raising --mamba-full-memory-ratio gives the pool more of the free memory, and --mamba-ssm-dtype bfloat16 halves the size of every slot against float32. Both are already set in the launch above.

Two Limits on These Figures

These numbers are a floor rather than a ceiling, for two reasons.

Using default MoE kernel config. Performance might be sub-optimal!
Config file not found at .../configs/triton_3_7_1/
  E=512,N=128,device_name=NVIDIA_H100_PCIe.json

Qwen3.8 Max Deployment Cost

A Qwen3.8 Max deployment cost is easy to reason about once the node count is settled, because the cluster bills as eight identical machines by the minute. The rate is the same whether the GPUs are loading weights, capturing graphs or serving traffic, so the only open question is how long each phase lasts.

What Each Phase Cost in This Deployment

These are the phases we timed, priced at the Spot cluster rate of $128.00 per hour, which is $2.13 a minute. Everything here is measured rather than modelled.

The Three Levers That Move the Bill

Only one of these is a tuning exercise. The other two are decisions taken before the cluster exists.

Serve a 2.4 trillion parameter model yourself

Deploy Qwen3.8 Max on 64 NVIDIA H100 GPUs

Eight nodes on Spot at $128.00 per hour. Our deployment loaded 2.50 TB across 64 GPUs, reached a live endpoint in under six minutes of engine start-up and served the full concurrency sweep without a single failed request.

64x NVIDIA H100 80GBTensor 16 x pipeline 4262,144 token context$128.00 per hour

Launch an NVIDIA GPU cluster on Hyperstack today.

FAQs

What hardware do you need to deploy Qwen3.8 Max?

The FP8 checkpoint is 2.50 TB, which is 39.1 GB per GPU across 64 cards. We ran it on eight Hyperstack nodes of 8x NVIDIA H100 80GB PCIe, leaving 17.34 GB free per GPU once the caches and CUDA graphs were resident.

How much does a Qwen3.8 Max deployment cost?

Eight nodes of 8x NVIDIA H100 80GB PCIe on Spot are $2.00 per GPU hour, so $128.00 an hour for the cluster, billed by the minute. A warm engine start costs $12.14 and the full concurrency sweep cost $8.46.

Can you turn thinking off on Qwen3.8 Max?

No. Every completion carries chain of thought, and reasoning shares the same max_tokens budget as the answer. The documented control is reasoning_effort, which takes low, medium or xhigh, and defaults to xhigh.

Why does Qwen3.8 Max need eight nodes and not six?

Six nodes hold the weights with 27.9 GB free per GPU, then fail on arithmetic. Tensor parallel 24 and 12 divide neither the 64 query heads nor the layer count, so no legal placement exists. Seven nodes fail the same test.

How long does Qwen3.8 Max take to start serving?

Five minutes 42 seconds from container start to a live endpoint with a warm page cache, or roughly ten minutes reading the 213 shards cold. Capturing CUDA graphs is 221.6 seconds of that and happens on every launch.

MiniMax H3 is a 33 billion parameter omni-modal generative system, open sourced on 3 August 2026, and it does something the text-to-video models before it did not: it writes the soundtrack at the same time as the picture. Video latents and audio latents are predicted by the same transformer, in the same forward pass, from one packed multimodal sequence. What lands on disk is a single MP4 carrying H.264 video at 24 frames per second and AAC stereo at 32 kHz, already in sync, with no separate audio model and no post-production step.

This guide serves it on one Hyperstack node: four NVIDIA H100 80GB PCIe GPUs on Spot at $8.00 per hour, through SGLang Diffusion. Every number below comes from a single session on 11 August 2026 that pulled 196 GB of checkpoints, reached a live endpoint, and produced eleven finished clips across all three task modes. Total generation time on the four GPUs was 51.5 minutes for 65.3 seconds of video, and every clip was verified with ffprobe to be carrying two channels of 32 kHz audio.

Serving it yourself means three decisions: how to place the model across four GPUs, how to configure the collectives for the fabric those GPUs sit on, and how to write a prompt without the hosted preprocessor that normally rewrites one. If you have read our technical deep dive into Kimi K3, this is the same shape of exercise on a much smaller cluster and a generative rather than a language workload. If you want a lighter video model on a single GPU first, our Wan 2.1 video generation tutorial covers that ground.

What MiniMax H3 Is, and Which Part of It You Can Run Yourself

H3 is not one model. The model card describes three modules, and only the middle one has open weights. Getting that distinction right before you book hardware saves a great deal of confusion later, because most of the published sample output was produced by all three working together.

That leaves a self-hosted deployment with two consequences and one advantage. The consequences: output is capped at a 768 pixel short edge, and nothing rewrites a rough prompt into something the model understands well. The advantage: the weights are the whole system you are running, so results are reproducible from a seed and a prompt, with no hidden service in the middle changing what you asked for.

The two output streams stay together because of how the sequence is built. Text goes through the H3-Encoder. Images and video go through the H3-Encoder and the visual VAE. Audio goes only through the audio VAE. All of it is then packed into one sequence with three-dimensional rotary position embeddings over time, height and width, and the transformer predicts video latents and audio latents jointly. Synchronisation is not a post-process, it is a property of the sequence.

What is open, and what is not

The Arithmetic That Decides the Hardware

Two numbers set the shape of the deployment. The first is what the checkpoints weigh. The repository hosts the original checkpoint and the diffusers format side by side, so scoping the download matters: pulling only model_index.json, FL2VA/* and Ref2VA/* landed 196 GB across 163 files on our node.

The second is what has to be resident while a generation runs. The official vLLM recipe puts the BF16 footprint of one task partition at roughly 135 GiB before any activations: two DiTs at 66.3 GB each, the Qwen3-VL-32B encoder at 51.5 GB, the video VAE at about 10 GB and the audio VAE at about 0.6 GB. No single 80 GB card holds that, which is why four GPUs is the floor rather than a preference.

How those four GPUs are arranged is a choice rather than a default. The SGLang MiniMax-H3 cookbook publishes three placements for 4x NVIDIA H100 80GB, all of which keep the whole pipeline resident, and they trade latency against memory.

The cookbook labels tensor parallel 2 with Ulysses degree 2 the fastest measured placement on this hardware, and it leads by 0.61 seconds. It also states plainly that pure Ulysses degree 4 cannot keep the pipeline resident on 80 GB NVIDIA H100 cards. We ran tensor parallel 4 with Ulysses degree 1, the lowest-memory row, because it leaves the most headroom for a long reference clip and asks the least of the interconnect on a PCIe fabric.

Matching the Configuration to Your Node

Two settings depend on how the four GPUs in your node are wired together. Each one is a single line in the launch command, and each one follows from a check that takes a second.

1. Read the fabric, then set NCCL to match it

Ask the driver how the four cards are connected, and ask CUDA what it will pass directly between them. The two answers agree, and together they tell you which transport NCCL should use.

# What the driver reports about how the four cards are wired together
nvidia-smi topo -m

# And what CUDA will actually permit between them
python3 -c "
import torch
for i in range(4):
    for j in range(4):
        if i != j:
            print(i, j, torch.cuda.can_device_access_peer(i, j))
"

        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity
GPU0     X      PHB     PHB     PHB     0-123           0-1
GPU1    PHB      X      PHB     PHB     0-123           0-1
GPU2    PHB     PHB      X      PHB     0-123           0-1
GPU3    PHB     PHB     PHB      X      0-123           0-1

0 1 False    0 2 False    0 3 False
1 0 False    1 2 False    1 3 False
2 0 False    2 1 False    2 3 False
3 0 False    3 1 False    3 2 False

PHB means every pair of GPUs communicates through a PCIe host bridge, which is the fabric on this flavour, and torch.cuda.can_device_access_peer reports False for all twelve ordered pairs. Collectives therefore travel through host memory, and NCCL is told so explicitly with two NCCL environment variables: NCCL_P2P_DISABLE=1 selects that path, and NCCL_CUMEM_ENABLE=0 keeps the allocator on the matching one. Neither costs anything measurable here, because the sequence is split four ways rather than thirty-two and the collectives are small next to the denoising loop they sit inside.

2. Choose the parallel placement that suits the fabric

All three documented placements fit on four NVIDIA H100 cards. Tensor parallel 2 with Ulysses degree 2 is the fastest of them by 0.61 seconds, and it is the one to reach for on a node with peer-to-peer. On a PCIe fabric, tensor parallel 4 with Ulysses degree 1 is the better fit: it is the lowest-memory row at 49.80 GB per GPU, and with Ulysses degree 1 there are no sequence-parallel groups to form, so nothing crosses the host bridge that does not have to.

Trading 0.61 seconds of pipeline latency for 16 GB of headroom per card is a comfortable trade when a single Ref2VA job peaks at 54.9 GB as it is, and it leaves room for a longer reference clip than the fastest row would.

How to Deploy MiniMax H3 on Hyperstack

Now, let us walk through the step-by-step process of standing up the node and serving MiniMax H3 across all four GPUs.

Four things decide the flavour, and n3-H100x4 supplies all four.

Step 1: Accessing Hyperstack

First, you will need an account on Hyperstack.

• Go to the Hyperstack website and log in.
• If you are new, create an account and set up your billing information. Our documentation can guide you through the initial setup.

Step 2: Deploying a 4x NVIDIA H100 Virtual Machine

From the Hyperstack dashboard, launch a single GPU virtual machine. The GPU count is fixed by the memory arithmetic above: one task partition is about 135 GiB in BF16, and no smaller shape holds it.

• Initiate Deployment: Click the "Deploy New Virtual Machine" button on the dashboard.

The button sits above the virtual machine list, under Cloud then Virtual Machines.

• Select Hardware Configuration: Choose a 4x NVIDIA H100 80GB PCIe flavour, listed in the flavour reference as n3-H100x4. Our run used the Spot variant, n3-H100x4-spot, at $2.00 per GPU per hour, which is $8.00 per hour for the node. The flavour also carries 124 vCPUs, 720 GB of RAM and a 3.2 TB ephemeral disk, all three of which this deployment uses.

Choose the NVIDIA H100-80G-PCIe card, then set the count to 4x in its dropdown.

• Choose the Operating System: Select the "Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker" image. The driver line has to match what the SGLang container expects, and this image also ships Docker and the NVIDIA container runtime ready to use, so there is no driver work at all.
• Environment: Choose an environment carrying NVIDIA H100 stock. Our run used CANADA-1.
• Select a Keypair: Choose an existing SSH keypair, or import one now. You will need it in step 4.
• Network Configuration: Assign a Public IP so you can reach the machine over SSH.
• Enable Ephemeral Storage: Make sure the ephemeral disk is attached. The checkpoints are 196 GB and the root disk is 96 GB, so this is not optional here.
• Review and Deploy: Check the settings and click "Deploy". The machine reaches ACTIVE in a couple of minutes.

Open the Ubuntu dropdown and take the R570 CUDA 12.8 with Docker build.

Step 3: Configure the Firewall

This deployment needs one inbound rule, and one port deliberately left shut. Add the first under firewall rules on the virtual machine:

Port 30010 stays closed deliberately. Every generation here runs from a script on the node, because reference images and clips are passed as file:// URIs that only resolve inside the server container. Driving the endpoint remotely would put those references out of reach and expose an unauthenticated API at the same time. If you do need it from your own machine, forward it over the SSH session you already have rather than opening a port:

# Optional. Forwards the endpoint to localhost:30010 on your own machine without
# opening anything on the node. Note that file:// conditioning still resolves only
# inside the container, so image and video references need the on-node client.
ssh -i /path/to/your-key -L 30010:127.0.0.1:30010 ubuntu@[PUBLIC IP]

Step 4: Accessing Your Node

Once the virtual machine is running, copy its Public IP from the dashboard and connect.

# The public IP is on the virtual machine's detail page in the dashboard
ssh -i /path/to/your-key ubuntu@[PUBLIC IP]

Confirm the two things this deployment depends on: four NVIDIA H100 cards, and a large disk mounted at /ephemeral.

nvidia-smi --query-gpu=index,name,memory.total --format=csv,noheader
df -h / /ephemeral | tail -2

0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
2, NVIDIA H100 PCIe, 81559 MiB
3, NVIDIA H100 PCIe, 81559 MiB

/dev/vda1        96G   13G   83G  14% /
/dev/vdb        3.2T   89M  3.0T   1% /ephemeral

The disk line is what shapes the rest of the guide. The machine has two disks and only one of them matters here: the root filesystem is roughly 96 GB, while the NVMe disk is mounted separately at /ephemeral and offers around 3.2 TB on this flavour. It is documented under ephemeral storage. The two checkpoint partitions are 196 GB together and the container image adds more on top, so the model cache has to live on the big disk.

Step 5: Prepare the Disks and Pull the Checkpoints

This script does four things: creates the working directories on /ephemeral, installs ffmpeg and jq, and then starts the container image pull and the 196 GB weight download at the same time rather than one after the other.

#!/bin/bash
# Per-node prep for MiniMax-H3 on a single 4x NVIDIA H100 box.
# The Hugging Face cache goes to /ephemeral: root is only 96 GB, the checkpoints are not.
set -u

sudo mkdir -p /ephemeral/hf /ephemeral/media /ephemeral/out
sudo chown -R ubuntu:ubuntu /ephemeral/hf /ephemeral/media /ephemeral/out

sudo apt-get update -qq
sudo apt-get install -y -qq ffmpeg jq
python3 -m pip install -q --break-system-packages huggingface_hub hf_transfer

# Pull the engine image and the weights at the same time: neither one waits for the other.
nohup docker pull lmsysorg/sglang:dev >/tmp/pull.log 2>&1 &

nohup env HF_HOME=/ephemeral/hf HF_HUB_ENABLE_HF_TRANSFER=1 HF_TOKEN="$(cat ~/hf_token.txt)" \
  python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('MiniMaxAI/MiniMax-H3',
                  allow_patterns=['model_index.json', 'FL2VA/*', 'Ref2VA/*'],
                  max_workers=16)
" >/tmp/dl.log 2>&1 &

echo "image pull and checkpoint download started"

The download is the long pole. With hf_transfer enabled and sixteen workers it is bounded by the network rather than the disk. Check it landed before you launch anything.

du -sh /ephemeral/hf/hub/models--MiniMaxAI--MiniMax-H3
find /ephemeral/hf -name "*.safetensors" | wc -l
df -h /ephemeral | tail -1

196G    /ephemeral/hf/hub/models--MiniMaxAI--MiniMax-H3
163
/dev/vdb        3.2T  197G  2.8T   7% /ephemeral

Step 6: Confirm the GPU Topology

Check the fabric before you serve. It takes one second, and it decides two environment variables in the next step.

# What the driver reports about how the four cards are wired together
nvidia-smi topo -m

# And what CUDA will actually permit between them
python3 -c "
import torch
for i in range(4):
    for j in range(4):
        if i != j:
            print(i, j, torch.cuda.can_device_access_peer(i, j))
"

If every pair reads PHB and can_device_access_peer reports False, as on ours, keep NCCL_P2P_DISABLE=1 and NCCL_CUMEM_ENABLE=0 in the launch below. If you see NV12 and True, you are on an NVLink flavour and can leave both out. The section above explains the difference.

Step 7: Launch MiniMax H3

The base SGLang image does not ship the diffusion extra, so it is installed from the bundled source at container start, which is exactly what the cookbook Docker form does. The two environment variables from step 6 go on the docker run line, and /ephemeral/media is mounted read-only so that file conditioning resolves later without any upload step.

#!/bin/bash
# Serve MiniMax-H3 on a single 4x NVIDIA H100 node.
# The two NCCL variables suit a PCIe fabric; on an NVLink flavour you can leave them out.
#
# Usage: serve_mmh3.sh [variant]    variant = fl2va (default, serves t2va + fl2va) | ref2va
VARIANT=${1:-fl2va}

docker rm -f mmh3 2>/dev/null || true
docker run -d --name mmh3 --gpus all --shm-size 32g --ipc=host --network host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  -v /ephemeral/media:/data/minimax-h3:ro \
  --env "HF_TOKEN=$(cat $HOME/hf_token.txt)" \
  --env HF_HUB_ENABLE_HF_TRANSFER=1 \
  --env NCCL_P2P_DISABLE=1 \
  --env NCCL_CUMEM_ENABLE=0 \
  lmsysorg/sglang:dev \
  bash -lc 'python -m pip install -e "/sgl-workspace/sglang/python[diffusion]" && exec sglang serve "$@"' -- \
    --model-path MiniMaxAI/MiniMax-H3 \
    --model-variant "$VARIANT" \
    --num-gpus 4 \
    --tp-size 4 \
    --ulysses-degree 1 \
    --performance-mode speed \
    --host 0.0.0.0 \
    --port 30010

echo "launched MiniMax-H3 variant=$VARIANT on 4x NVIDIA H100"

Three flags shape what the server does. --model-variant fl2va serves both t2va and fl2va from one checkpoint partition, so nine of our eleven clips came from a single server. ref2va is a separate partition and needs a restart, which took about seven minutes including the diffusion extra install. --performance-mode speed deliberately keeps the DiT eager: the cookbook does not recommend torch.compile here because it changes numerical output for a negligible gain.

Step 8: Verify the Deployment

Confirm the pipeline end to end before committing four minutes to a full generation. Eight steps at four seconds is enough to exercise every stage.

# Eight steps and four seconds, purely to prove the pipeline runs end to end
JOB=$(curl -s -X POST http://127.0.0.1:30010/v1/videos \
  -H "Content-Type: application/json" \
  -d '{"model": "MiniMaxAI/MiniMax-H3",
       "prompt": "integrated_multimodal_description: [Shot 1] A violet ink drop blooms in clear water. The camera holds a static shot throughout.\noverall_soundscape: A single soft plink, then a quiet liquid swirl.\nnon_diegetic_music: One sustained synthesiser pad, no percussion.",
       "seconds": 4, "task": "t2va", "conditions": [],
       "target": {"short_edge": 768, "aspect_ratio": "1:1", "duration_seconds": 4.0},
       "num_inference_steps": 8, "seed": 9233}' | jq -r .id)

until [ "$(curl -s http://127.0.0.1:30010/v1/videos/$JOB | jq -r .status)" = "completed" ]; do
  sleep 4
done

curl -s http://127.0.0.1:30010/v1/videos/$JOB/content -o /ephemeral/out/smoke.mp4
ffprobe -v error -show_entries stream=codec_name,width,height,sample_rate,channels,nb_frames \
  -of csv=p=0 /ephemeral/out/smoke.mp4

h264,768,768,107
aac,32000,2,141

Two codecs, one file, 32000 Hz, two channels, back in 16.1 seconds. The 107 frames matter too: H3 quantises duration to a legal frame count, so a request for four seconds becomes 4.46 seconds.

Driving the Asynchronous Video Endpoint

H3 does not stream. A generation takes minutes, so SGLang exposes it as an OpenAI-shaped asynchronous job: submit, poll, then fetch the bytes. Three endpoints cover the whole surface.

The request body carries the decisions that shape the output. This is the exact shape we sent for every text-to-video clip.

{
  "model": "MiniMaxAI/MiniMax-H3",
  "prompt": "<the structured prompt, in full>",
  "seconds": 6,
  "task": "t2va",
  "conditions": [],
  "target": {
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "duration_seconds": 6.0
  },
  "num_outputs_per_prompt": 1,
  "num_inference_steps": 50,
  "flow_shift": 12.0,
  "audio_flow_shift": 3.0,
  "seed": 9233
}

Conditioning is passed as a list. An empty list is text-to-video; an image with the keyframe role and frame_index: 0 anchors the first frame; a video or an image with the reference role drives Ref2VA. Every URI is a file:// path resolved inside the server container, which is why the media directory is bind mounted read-only at /data/minimax-h3.

def build_conditions(task, spec):
    """MEDIA is the read-only mount inside the server container, /data/minimax-h3."""
    if task == "t2va":
        return []
    if task == "fl2va":
        img = f"{MEDIA}/{spec['source']}_frame.png"
        return [{"type": "image", "uri": "file://" + img,
                 "role": "keyframe", "frame_index": 0}]
    if task == "ref2va":
        if spec.get("source_video"):
            return [{"type": "video", "uri": f"file://{MEDIA}/{spec['source_video']}.mp4",
                     "role": "reference", "start_time_seconds": 0.0}]
        return [{"type": "image", "uri": f"file://{MEDIA}/{spec['source']}_frame.png",
                 "role": "reference"}]
    raise ValueError(task)

The client is straightforward: submit, poll on a four second interval, download, then probe the result with ffprobe and write a JSON record holding the exact request, the job id, the wall clock and the measured media properties. Those records are where every timing and every media property quoted here comes from.

BASE = "http://127.0.0.1:30010"

def run_one(task, spec, steps):
    body = {
        "model": "MiniMaxAI/MiniMax-H3",
        "prompt": spec["prompt"],
        "seconds": spec["seconds"],
        "task": task,
        "conditions": build_conditions(task, spec),
        "target": {"short_edge": 768, "aspect_ratio": spec["ratio"],
                   "duration_seconds": float(spec["seconds"])},
        "num_outputs_per_prompt": 1,
        "num_inference_steps": steps,
        "flow_shift": 12.0,
        "audio_flow_shift": 3.0,
        "seed": spec["seed"],
    }
    t0 = time.time()
    job = post("/v1/videos", body)
    vid = job.get("id")

    while True:
        time.sleep(4)
        st = get(f"/v1/videos/{vid}")
        status = st.get("status")
        if status in ("completed", "succeeded"):
            break
        if status in ("failed", "cancelled", "error"):
            return {"id": spec["id"], "error": st}
    gen_s = time.time() - t0

    mp4 = os.path.join(OUT, f"{spec['id']}.mp4")
    with urllib.request.urlopen(f"{BASE}/v1/videos/{vid}/content", timeout=600) as r, \
         open(mp4, "wb") as f:
        f.write(r.read())

    # ffprobe every file, so the record holds measured properties
    return {"id": spec["id"], "task": task, "generation_seconds": round(gen_s, 1),
            "num_inference_steps": steps, "request": body, "job_id": vid,
            "media": probe(mp4), "mp4": mp4}

Prompting Is the Entire Quality Lever

Because H3-Context-IR is not part of the open release, nothing sits between what you type and what the transformer reads, so a vague prompt stays vague. MiniMax anticipated this and published the specification its own service writes to, as a set of prompt-writing skills in the MiniMax-H3 repository. Writing to that specification is not a style preference, it is the deployment.

The specification has two shapes. Text-to-video and frame conditioning take exactly three sections. Reference conditioning takes exactly six.

The four rules that carry the most weight

• Sound is a first-class section, not an afterthought. Splitting diegetic sound from non-diegetic music is what lets the model place a bell motif in the score and a fabric rustle in the room at the same time.
• Camera motion has a canonical phrasing: motion type, then amplitude, then speed. "The camera performs a tracking shot forward with medium amplitude at slow speed" is understood. "Slow dolly in" is a guess.
• First-frame conditioning has a mandatory opening sentence. "For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced." Without it the supplied frame is treated as loose inspiration rather than as frame zero.
• Dialogue is tagged, and speakers are stable. The speaker is described outside the tag and the words go inside it with a language label, so the same identifier keeps the same voice across the clip.

A composed male presenter in his thirties (S1) says, <d>[English] Every frame you are
watching, and every sound, came out of one model in a single pass.</d>

Writing a brand palette into the pixels

Every clip here is in the Hyperstack palette, and none of it was colour graded afterwards. The palette was written into every prompt in natural language, so H3 rendered those colours directly. Hex codes mean nothing to a video model. Colour names do.

The clause that does the most work is the negative one. Every prompt ends its visual section with the same sentence: a strict colour palette of electric violet, deep royal indigo and rich amethyst, with luminous orchid highlights and pale lilac-white bloom; no warm oranges, no greens and no reds anywhere in frame. Stating what must not appear is what holds the palette steady across eleven independent generations with eleven different seeds.

integrated_multimodal_description: [Shot 1] Abstract macro cinematography with a tight,
centred composition and shallow depth of field. A single sheet of weightless liquid silk
fills the frame against a seamless near-black background. Its surface carries a smooth
135-degree gradient running from electric violet at the upper left to deep royal indigo
at the lower right. The silk folds and unfurls continuously in slow motion; rich amethyst
shadows pool inside each fold while luminous orchid catch-lights trace the rolling edges,
and a pale lilac-white bloom haloes the brightest crests. The camera holds a static shot
throughout, letting the fabric move within the frame. Rendered with a strict colour
palette of electric violet, deep royal indigo and rich amethyst, with luminous orchid
highlights and pale lilac-white bloom; no warm oranges, no greens and no reds anywhere
in frame.
overall_soundscape: A soft, airy low-frequency swell of fabric moving through still air
in a large quiet room. A faint high shimmer rises each time the light catches a fold.
There are no voices and no mechanical noise.
non_diegetic_music: Minimal ambient electronic score: one slow sustained synthesiser pad
in a major key with a single soft bell motif every few seconds and gentle sub-bass, no
percussion.

Confirming the structure before you spend GPU time on it

The specification is easy to hold to once it is encoded. Two small helpers build the prompts in the official order, and a block of assertions checks every one of them at import, so the structure is confirmed in a second rather than inferred from the finished video four minutes later.

PALETTE = ("a strict colour palette of electric violet, deep royal indigo and rich amethyst, "
           "with luminous orchid highlights and pale lilac-white bloom; no warm oranges, "
           "no greens and no reds anywhere in frame")

# The official opener, required whenever a real first frame is supplied
FIRST_FRAME_ANCHOR = ("For the target video, at 0.00 seconds into the target video, "
                      "<Picture 1> (from [Shot 1]) is fully referenced.")


def P(desc, sound, music):
    """T2VA and FL2VA: exactly three sections, in the official order."""
    return (f"integrated_multimodal_description: {desc}\n"
            f"overall_soundscape: {sound}\n"
            f"non_diegetic_music: {music}")


def R(subjects, summary, retention, detail, sound, music):
    """Ref2VA: exactly six sections, in the official order."""
    return (f"subject_definitions:\n{subjects}\n\n"
            f"summary:\n{summary}\n\n"
            f"retention_analysis:\n{retention}\n\n"
            f"detailed_description:\n{detail}\n\n"
            f"overall_soundscape:\n{sound}\n\n"
            f"non_diegetic_music:\n{music}")

# Run at import, so the structure is confirmed before any GPU time is spent.
for p in T2VA + FL2VA:
    assert p["prompt"].startswith("integrated_multimodal_description:"), p["id"]
    assert "\noverall_soundscape:" in p["prompt"], p["id"]
    assert "\nnon_diegetic_music:" in p["prompt"], p["id"]

for p in REF2VA:
    for section in ["subject_definitions:", "summary:", "retention_analysis:",
                    "detailed_description:", "overall_soundscape:", "non_diegetic_music:"]:
        assert section in p["prompt"], (p["id"], section)

for p in FL2VA:
    assert "<Picture 1>" in p["prompt"] and "0.00 seconds" in p["prompt"], p["id"]

This is the cheapest quality control in the whole pipeline. On a node costing $8.00 per hour, confirming the structure at import rather than after generation is worth roughly sixty cents of GPU time each time it runs.

Eleven Clips Across Three Task Modes

Every reference image and every reference clip used below came out of this same run. Nothing here is stock footage. After the seven text-to-video clips finished, we pulled one frame out of four of them at roughly forty per cent in, past the opening ramp where the composition has settled, and used those frames to drive the conditioned modes.

#!/bin/bash
# Pull frames out of our own T2VA output, to use as reference material.
# Frames are taken ~40% into each clip, past the opening ramp, where the composition
# has settled. /ephemeral/media is mounted read-only in the server at /data/minimax-h3.
set -u
OUT=/ephemeral/out
MEDIA=/ephemeral/media
mkdir -p "$MEDIA"

extract () {
  local src="$OUT/$1.mp4"
  local dur at
  dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$src")
  at=$(python3 -c "print(round(float('$dur') * 0.4, 2))")
  ffmpeg -y -v error -ss "$at" -i "$src" -frames:v 1 "$MEDIA/$1_frame.png"
}

for id in "$@"; do extract "$id"; done

# Ref2VA video conditioning needs the clip itself visible inside the container too
for id in "$@"; do cp -f "$OUT/$id.mp4" "$MEDIA/$id.mp4"; done

# T2VA and FL2VA are both served by the fl2va checkpoint, so these run back to back
python3 mmh3_generate.py --task t2va
bash mmh3_extract_frames.sh 01_brand_hero 02_gpu_datacentre 03_data_viz 04_aurora
python3 mmh3_generate.py --task fl2va

# Ref2VA is a separate checkpoint partition, so the server has to be restarted
bash serve_mmh3.sh ref2va
python3 mmh3_generate.py --task ref2va

Text to video and audio

Seven clips, no conditioning of any kind, straight from the three-section prompt. Each one has sound generated in the same pass as the picture, so unmute them.

Liquid gradient silk

A sheet of weightless silk carrying the 135-degree brand gradient, folding in slow motion against near-black. The soundtrack is a fabric swell with a soft bell motif over a sustained pad.

GPU data-centre aisle

A forward tracking shot with medium amplitude at slow speed down a symmetrical aisle. The volumetric haze and the reflected floor came from the prompt, not from a render pass.

Data visualisation materialising

An arc shot around glass bar-chart columns that grow in sequence. This clip later became the style reference for clip 11.

Violet aurora over dark peaks

The widest geometry in the set at 21:9 and eight seconds, which makes it the most expensive clip we generated. Note the aurora reflected in the lake, which no part of the prompt describes twice.

Violet ink bloom, macro

Square, five seconds, and the cheapest clip in the set at 104.1 seconds. Fewer pixels and fewer frames mean a shorter sequence, and the denoising loop scales with it directly.

Refracting glass prisms

An arc shot with medium amplitude at slow speed around three prisms, with caustics thrown across the floor. The audio is glassy interface tones rather than music.

Speech, lip sync and a generated voice

The same three-section prompt, with one dialogue line tagged in the official form. H3 produced the speech, the lip movement and the voice timbre together with the picture, in the same pass, on the same 50 steps. Nothing here was dubbed.

A studio presenter delivering one tagged line. The voice, the lip sync and the rim lighting are all model output from a single prompt.

This clip cost 276.2 seconds against 272.2 for the three silent 16:9 clips of the same geometry. Adding speech, lip sync and a voice to a generation added four seconds, or about 1.5 per cent. The audio stream is cheap next to the video stream it is synchronised with.

First frame to video and audio

Both clips below start from a PNG pulled out of the text-to-video output above, passed as a keyframe condition at frame_index: 0, with the mandatory anchor sentence at the top of the prompt. The task is continuation: hold the composition, keep the motion going.

The hero clip continues from its own frame

Conditioned on a frame taken 40 per cent into clip 01. The gradient keeps sliding and the folds keep forming from exactly where the reference left them.

The aurora continues from its own frame

Conditioned on a frame from clip 04, at 21:9. The peaks hold as a static silhouette while the curtains keep moving, which is what the prompt asked for explicitly.

Reference to video and audio

Ref2VA is the most capable mode and the most expensive one. It is also a separate checkpoint partition, so the server has to be restarted with --model-variant ref2va before either of these will run. The prompt shape changes completely: six sections, labelled references, and a retention marker on every label saying how much of it should survive.

Video to video: clip 02, restyled

The data-centre aisle from clip 02 as a reference video, restyled into glowing monoliths. The forward camera motion and the aisle symmetry are preserved, the surfaces and the light are not.

The retention markers are doing the work here. <Video 1> is marked partially_preserved, which keeps the motion and the pacing. <Subject 1>, the racks, is marked attribute_transfer, which keeps their position and rhythm along the aisle while changing what they are made of. The markers are the dial: fully_preserved holds a reference exactly, weak_reference keeps only its atmosphere, and the two in between set how much of it carries across.

subject_definitions:
<Video 1> is the source video for the editing task: a forward tracking shot down the
centre of a modern GPU data-centre aisle lit by vertical strips of electric violet light.
<Subject 1> is the double row of tall black server racks receding toward a vanishing
point in <Video 1>.

summary:
[video editing + reference generation] The target video restyles <Video 1> into a
futuristic night-time control room. <Subject 1> is transformed from server racks into
tall glowing monoliths while the original forward camera motion, aisle symmetry and
timing are preserved.

retention_analysis:
<Video 1> (source video editing): partially_preserved - the forward tracking motion, the
symmetrical aisle layout and the overall pacing are retained, while the surfaces, haze
density and light intensity are restyled.
<Subject 1> (appears in [Shot 1]): attribute_transfer - the racks keep their position,
scale and rhythm along the aisle but take on the appearance of smooth glowing monoliths.

detailed_description:
The target video is in a sleek, high-contrast, photoreal science-fiction style with heavy
volumetric atmosphere.
[Shot 1] Cinematic wide symmetrical composition down the centre of a darkened control-room
corridor. The two facing rows from <Subject 1> now read as tall, smooth, edge-lit monoliths
whose vertical seams glow electric violet. Denser volumetric haze fills the corridor so each
seam casts a hard directional beam across the floor, and the polished surface below mirrors
the light into long deep royal indigo streaks. Scattered luminous orchid indicator points
pulse slowly along the monolith faces, and a pale lilac-white light source marks the far end
of the corridor. The camera performs a tracking shot forward with medium amplitude at slow
speed, matching the motion of <Video 1>. Rendered with a strict colour palette of electric
violet, deep royal indigo and rich amethyst, with luminous orchid highlights and pale
lilac-white bloom; no warm oranges, no greens and no reds anywhere in frame.

overall_soundscape:
A deep mechanical hum with a wide cavernous reverberation fills the corridor, layered with
a fine airy hiss of moving air and occasional low electronic pulses.

non_diegetic_music:
Low pulsing synth ostinato with sparse metallic percussion, darker and more spacious than
the source.

Image reference: a style carried onto a new subject

A frame from clip 03 supplied as a style reference only, marked weak_reference. The bar-chart geometry does not appear: the glass material, the colour system and the particle atmosphere carry across to a network graph instead.

weak_reference is the marker that makes this work. It tells the model to take the material treatment, the lighting and the colour system, and to leave the subject behind. The same image marked fully_preserved would have produced bar charts again.

Verifying all eleven

One loop over the output directory confirms what every file contains.

for f in /ephemeral/out/*.mp4; do
  echo "$(basename $f) -> $(ffprobe -v error \
    -show_entries stream=codec_name,width,height,r_frame_rate,nb_frames,sample_rate,channels \
    -of csv=p=0 "$f" | tr '\n' ' ')"
done

01_brand_hero.mp4      -> h264,1344,768,24/1,158  aac,32000,2
02_gpu_datacentre.mp4  -> h264,1344,768,24/1,158  aac,32000,2
03_data_viz.mp4        -> h264,1344,768,24/1,158  aac,32000,2
04_aurora.mp4          -> h264,1536,672,24/1,192  aac,32000,2
05_studio_dialogue.mp4 -> h264,1344,768,24/1,158  aac,32000,2
06_ink_bloom.mp4       -> h264,768,768,24/1,124   aac,32000,2
07_glass_prisms.mp4    -> h264,1344,768,24/1,124  aac,32000,2
08_hero_continue.mp4   -> h264,1344,768,24/1,124  aac,32000,2
09_aurora_continue.mp4 -> h264,1536,672,24/1,124  aac,32000,2
10_v2v_restyle.mp4     -> h264,1344,768,24/1,124  aac,32000,2
11_image_ref_style.mp4 -> h264,1344,768,24/1,124  aac,32000,2

Every file is H.264 and AAC, at 24 frames per second, 32000 Hz, two channels. Look at that column of frame counts: 158, 192 and 124 are all of the form 17n + 5, which is exactly the frame quantisation the vLLM recipe documents. A request for six seconds becomes 158 frames, which is 6.58 seconds, and the extra 0.58 seconds is the model rounding up to the nearest legal length.

Where the Generation Time Goes

Eleven clips, 3,090.1 seconds of generation, 65.3 seconds of finished video. That is about 47 seconds of four-GPU wall clock for every second you end up with. The server log shows exactly where that time goes.

[08-11 03:53:08] Running pipeline stages: ['InputValidationStage',
  'MiniMaxH3PartitionAdmissionStage', 'MiniMaxH3TextEncodingStage',
  'MiniMaxH3VisualEncodingStage', 'MiniMaxH3AudioEncodingStage',
  'MiniMaxH3LatentPreparationStage', 'MiniMaxH3TimestepPreparationStage',
  'MiniMaxH3DenoisingStage', 'MiniMaxH3DecodingStage']
[08-11 03:53:16] [MiniMaxH3VisualEncodingStage]     finished in 5.8935 seconds
[08-11 03:53:16] [MiniMaxH3AudioEncodingStage]      finished in 0.3196 seconds
[08-11 03:53:16] [MiniMaxH3LatentPreparationStage]  finished in 0.0214 seconds
[08-11 03:53:16] [MiniMaxH3TimestepPreparationStage] finished in 0.0003 seconds
minimax_h3 denoise: 100%|##########| 49/49 [09:35<00:00, 11.75s/it]
[08-11 04:02:52] [MiniMaxH3DenoisingStage]          finished in 575.9068 seconds
[08-11 04:03:06] [MiniMaxH3DecodingStage]           finished in 2.0632 seconds
[08-11 04:03:07] Peak memory usage: 54888.00 MB

Denoising is 98.6 per cent of server-side time in both jobs. Everything else together is under nine seconds. That leaves two settings worth tuning: the number of inference steps, and the length of the sequence being denoised. Nothing else moves the needle.

The two jobs also differ by exactly one thing, and it shows. The video-reference job ran at 11.75 seconds per step; the image-reference job, at the same resolution and the same step count, ran at 6.08. A reference video is decoded, encoded and then carried in the attention sequence for every one of the 50 steps, so it nearly doubles the per-step cost. It also lifts peak memory from 46,020 MB to 54,888 MB.

Sequence length is the price, not resolution

Two of the clips isolate frame count directly. A 1344x768 frame and a 1536x672 frame contain exactly the same 1,032,192 pixels, so the only difference between clip 01 and clip 04 is the frame count: 158 against 192. Twenty-two per cent more frames cost thirty-one per cent more time.

If cost were linear in sequence length that line would be flat. It climbs from 34.1 to 43.1 seconds per megapixel-second instead, because this release ships inference with full attention only: the initial open source drop is full attention, and the sparse attention implementation MiniMax trained with is published separately at a later date. Until it lands, doubling the length of a clip costs more than twice as much.

What each task mode costs

The four clips below are the cleanest comparison in the run: identical geometry at 1344x768, identical duration at 5.17 seconds, identical 50 steps. Only the conditioning path changes.

How four NVIDIA H100 GPUs compare

The vLLM recipe publishes measured end-to-end figures for the same model on other four-GPU configurations. Normalising all of them to megapixel-seconds of finished video makes them roughly comparable, with the caveat that the NVIDIA B300 figure is a first-frame job at 1248x768 while the other two are text to video at 1344x768.

NVIDIA H100 is roughly four times the wall clock of NVIDIA B300 on this workload, which is what you would expect from a diffusion transformer that is almost entirely attention and feed-forward compute in BF16. It is also a good deal cheaper per GPU hour, at $2.00 on Spot against $7.40 for NVIDIA B300 on demand, so the cost per finished clip is much closer than the latency suggests. If you want faster iteration on NVIDIA H100 rather than faster hardware, the lever is step count: there is a community Turbo LoRA that distils first-and-last-frame generation down to four steps instead of fifty, with a distillation recipe behind it. We ran the full 50-step path throughout, on the released weights as published.

What the Run Cost

Four NVIDIA H100 80GB PCIe cards on Spot are $2.00 per GPU per hour, so the node runs at $8.00 per hour, or 0.22 cents per second. That makes the cost of every clip a simple multiplication.

The session as a whole is a different number, because the 196 GB download and two server starts are on the clock as well. End to end, from an empty node to eleven verified clips, took about two hours.

The download is the single largest line, and it is the one that does not get cheaper on faster GPUs. Everything after it is minutes, which means the marginal cost of another clip is measured in cents once the node is warm.

Generate video with native audio

Run MiniMax H3 on four NVIDIA H100 GPUs

One node on Spot at $8.00 per hour. Our run pulled 196 GB, served both checkpoint partitions and produced eleven clips with synchronised stereo audio for under $16.

4x NVIDIA H100 80GBTP 4 x Ulysses 1768p with 32 kHz stereo$8.00 per hour

Launch an NVIDIA GPU node on Hyperstack today.

FAQs

What hardware do you need to run MiniMax H3?

One task partition is about 135 GiB in BF16, so four GPUs of 80 GB is the practical floor. We ran it on 4x NVIDIA H100 80GB PCIe on a single Hyperstack node, peaking at 46.0 to 54.9 GB per GPU with tensor parallel 4 and Ulysses degree 1.

Can you get 2K video out of the open MiniMax H3 weights?

No. 2K comes from H3-Regenerate-2K, which is not part of the open release, so a self-hosted deployment produces a 768 pixel short edge. Every published aspect ratio from 21:9 to 9:16 works, and sets the long edge from there.

How long does MiniMax H3 take to generate a video on NVIDIA H100?

On 4x NVIDIA H100 80GB at 50 inference steps we measured 104.1 seconds for a five second square clip, 272.2 seconds for six seconds at 1344x768, and 600.4 seconds with a reference video. Denoising is 98.6 per cent of that.

How much does it cost to run MiniMax H3 on Hyperstack?

The 4x NVIDIA H100 80GB PCIe Spot node is $8.00 per hour, or 0.22 cents per second. The eleven clips cost $6.87 of GPU time between them, and the whole session including the 196 GB download came to about $16.

How do you write a MiniMax H3 prompt without H3-Context-IR?

Follow the prompt-writing skill in the model repository. Text-to-video and frame conditioning take exactly three sections in order: integrated_multimodal_description, overall_soundscape and non_diegetic_music. Reference conditioning takes six, and every labelled reference needs a retention marker.

Kimi K3 is Moonshot AI's 2.8 trillion parameter flagship, and since the weights were published on 27 July 2026 anyone can serve it themselves. It is a sparse Mixture-of-Experts model built on Kimi Delta Attention and Attention Residuals, activating 104 billion parameters through 16 of 896 experts per token, with native vision through MoonViT-V2 and a one million token context window. The checkpoint ships in native MXFP4, and when we pulled it the 96 safetensors shards measured 1,560.94 GB. That one number decides everything about serving it. The official vLLM recipe and the vLLM launch blog both put the floor at Blackwell: at least one 8x NVIDIA B300 node, or an NVIDIA GB300 NVL72, with 16x NVIDIA B200 also supported. NVIDIA H100 is not on that list.

This guide serves Kimi K3 on NVIDIA H100 anyway: 32 NVIDIA H100 80GB GPUs across four Hyperstack nodes, tensor parallel 32, expert parallel 32, at $64 per hour. Every log line, memory figure and timing below comes from a single run on 28 July 2026 that reached a live endpoint eleven minutes after the containers started, then answered chat, called two tools in one turn, read an image and returned its chain of thought, with the full one million token context window intact. If you have read our technical deep dive into Kimi K3, this is the deployment half of that story: the arithmetic that fixes the cluster shape, the launch command, and what the model does once it answers.

Kimi K3 in the Numbers That Decide a Deployment

Before any hardware is booked, five figures from the model card set the shape of the deployment: the parameter count, the activated parameter count, the quantisation format, the layer composition and the context window. Everything else in this guide follows from them.

Two of those rows are worth pausing on. MXFP4 stores roughly four bits per parameter plus a shared block scale, so 2.8 trillion parameters land at about 1.4 TB before overheads, and at 1.56 TB in practice once every tensor and the vision tower are counted. 16 of 896 is an extreme sparsity ratio, and it is why a 2.8 trillion parameter model has a compute cost closer to a 104 billion parameter one. The memory cost, however, is the full 2.8 trillion, because every expert has to be resident somewhere.

For context on what that memory buys, here is Kimi K3 against the strongest proprietary models on a representative slice of the published results.

The architecture you are about to shard across 32 GPUs

Kimi K3 is built on two architectural updates. Kimi Delta Attention gives the model an efficient foundation for scaling attention along sequence length, and Attention Residuals retrieve representations selectively across depth rather than accumulating them uniformly. The Mixture-of-Experts block is a Stable LatentMoE, trained stable at this sparsity by four techniques: Quantile Balancing, which derives expert allocation from router-score quantiles instead of a heuristic; Per-Head Muon, which optimises attention heads independently; the Sigmoid Tanh Unit for activation control; and Gated MLA for attention selectivity.

Kimi K3 also produced a motion-graphics explainer of its own architecture, which is worth watching before you shard it across 32 GPUs.

A 3Blue1Brown-style explainer of the Kimi K3 architecture, made by Kimi K3. Video: Moonshot AI, Kimi K3 release blog.

Why 1.56 TB of MXFP4 Does Not Fit Where You Expect

Read this section before provisioning anything, because it decides the shape of the cluster and it is where most of the money goes.

Start with the weights

MXFP4 packs four bits per parameter plus a shared exponent per block, so 2.8 trillion parameters imply roughly 1.4 TB before any overhead. The download settles the question exactly. Our verification script walked the Hugging Face manifest, resolved every symlink and added up the real bytes on disk on each of the four nodes:

repo             : moonshotai/Kimi-K3
snapshot dir     : /ephemeral/hf/hub/models--moonshotai--Kimi-K3/snapshots/9f62e4e9fffbd0a83ddd60e1c209d828994b3569
safetensors      : 96/96 present
total bytes      : 1,560,936,091,448 (1560.94 GB)
config.json      : OK
.incomplete files: 0
RESULT: COMPLETE

That is 1,560,936,091,448 bytes across 96 shards, identical on all four nodes, and it is the number every later calculation uses. It is worth pausing on the second half of that sentence, because a tensor-parallel deployment does not split the download the way it splits the model.

Now count the GPUs

An NVIDIA H100 "80GB" card is a rounded label. On our nodes nvidia-smi reported 81,559 MiB, which is 79.65 GiB, and the PyTorch allocator counted a total capacity of 79.19 GiB per card. Thirty-two of them come to about 2,534 GiB, or roughly 2.72 TB.

Set 1.56 TB of weights against 2.72 TB of memory and the deployment fits, with about 1.16 TB left for the KV cache, the recurrent state that Kimi Delta Attention needs, the activations, the CUDA context and the communication buffers. That arithmetic is what makes 32 NVIDIA H100 GPUs the right shape for this model, and the measured per-GPU budget later in this guide lands almost exactly where it predicts.

Why adding more NVIDIA H100 GPUs does not help

The obvious response to a tight fit is to add nodes. On Kimi K3 that does not work, and the reason is arithmetic rather than budget. Tensor parallelism has to divide the model evenly, which means it has to divide both the 96 attention heads and the 7168 hidden size. The greatest common divisor of 96 and 7168 is 32, so the valid tensor-parallel sizes are 1, 2, 4, 8, 16 and 32, and no more. A tensor-parallel size of 64 would ask for 1.5 attention heads per rank.

So TP 32 is not the largest shape that happens to work. It is the only one that clears both tests, and four nodes of eight NVIDIA H100 GPUs is the only way to reach it on this generation. Adding a fifth node cannot lower the per-GPU weight load, because 40 does not divide 96 either, and data parallelism replicates the model rather than sharding it. If TP 32 does not fit, the answer is a larger card, not a longer invoice.

One Model, Four Machines: the 32-GPU Topology

Kimi K3 has no sanctioned pipeline-parallel strategy, so a multi-node deployment runs tensor parallelism across the node boundary, with expert parallelism laid over the top. That is a different shape from the one we used for Tencent Hy3, where pipeline parallelism kept the heavy collectives inside each node and put a single activation tensor on the wire. Here every rank participates in every layer, so the topology is flat: thirty-two equal ranks, one process group, one model.

Three things follow from that, and they shape every command in the deployment steps below. All four nodes have to sit in the same environment, because that is what puts them on a shared private network. Collectives are pinned to that private interface with NCCL_SOCKET_IFNAME and run over TCP sockets, since on-demand nodes are joined by ordinary Ethernet rather than InfiniBand. And every node needs its own full copy of the weights on local NVMe, because each rank loads its shard of every tensor from disk.

The Kernel Path on Hopper, and the Settings That Fit 1.56 TB

This deployment uses SGLang, which publishes a Kimi K3 cell for four NVIDIA H100 nodes in its deployment cookbook. Two things about that configuration are worth understanding before you run it, because they explain both the flags and the performance.

MXFP4 weights, a mixed-precision kernel

Kimi K3 stores its expert weights in MXFP4, and the MoE backends that execute 4-bit weights natively are built around Blackwell FP4 hardware, which is why the published guidance points at NVIDIA B300 and NVIDIA GB300. On Hopper the expert matrix multiplies run instead through Marlin, the mixed-precision W4A16 kernel that multiplies 16-bit activations by 4-bit weights and hides the dequantisation cost behind the matrix multiply itself. The engine announces the choice on the first line of weight loading:

[TP0 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled
          (moe_runner_backend=marlin, quant_method=Mxfp4MoEMethod).

That single line in the log explains the throughput figures later in this guide. It is also why --moe-runner-backend marlin appears in the launch command rather than being left to auto: on Hopper it is the path that runs, so it is worth stating.

Three settings that keep the load inside the budget

Loading 1.56 TB through 32 allocators at once is the part of this deployment that is sensitive to configuration. Three of the settings in the launch command exist for exactly that, and they are the ones to keep if you change anything else:

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True switches the PyTorch caching allocator onto CUDA virtual memory management, so segments grow and shrink in place instead of being pinned as separate cudaMalloc blocks that can never merge. On a load that streams 96 shards through the allocator, that is what keeps the resident footprint close to the size of the weights themselves.
• --ep-size 32 states expert parallelism explicitly rather than leaving it to be inferred, so each rank holds exactly its own slice of the 896 experts and no more.
• --mem-fraction-static 0.85 sizes the static pool, and it is the single lever to reach for if you want a larger KV cache or more headroom.

With those in place the load runs to completion, and the head node reports where every gigabyte went. The log arrives in four stages, and each one answers a different question.

First, the cluster forms. The tokeniser loads, then all 32 ranks find each other and agree on the process group. Forty-one seconds for a rendezvous across four machines is normal, and almost no memory has been touched yet:

[2026-07-28 07:39:14] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2026-07-28 07:39:14] Applying special tokens cache patch for Kimi tokenizer
[2026-07-28 07:39:14 TP0 EP0] Init torch distributed begin.
[2026-07-28 07:39:58 TP0 EP0] Init torch distributed ends. elapsed=41.49 s, mem usage=0.47 GB

Then the weights load. This is the long part, and the middle line is the kernel choice from the section above. Note the pair of numbers at either end: 78.23 GB available before, 18.60 GB after, so the weights themselves cost 59.63 GB per card:

[2026-07-28 07:39:58 TP0 EP0] Load weight begin. avail mem=78.23 GB
[2026-07-28 07:39:58 TP0 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled
     (moe_runner_backend=marlin, quant_method=Mxfp4MoEMethod).
[2026-07-28 07:45:16 TP0 EP0] Load weight end. elapsed=317.50 s,
     type=KimiK3ForConditionalGeneration, quant=compressed-tensors,
     avail mem=18.60 GB, mem usage=59.63 GB.

Next the memory pools. The KV cache is only 3.62 GB, but available memory falls by 7.03, and the difference is the recurrent state that Kimi Delta Attention keeps per sequence. The attention backends are chosen here too, one for decode and a different one for prefill:

[2026-07-28 07:45:18 TP0 EP0] KV Cache is allocated. dtype: torch.bfloat16,
     #tokens: 140352, KV size: 3.62 GB
[2026-07-28 07:45:18 TP0 EP0] Memory pool end. avail mem=11.57 GB
[2026-07-28 07:45:18 TP0 EP0] Using hybrid attention backend for decode and prefill:
     decode_backend=flashmla, prefill_backend=fa3.

Finally the server comes up. The line to read is the first one, because it is where the engine states what it will actually accept:

[2026-07-28 07:49:32 TP0 EP0] max_total_num_tokens=140352, chunked_prefill_size=8192,
     max_prefill_tokens=16384, max_running_requests=49, context_len=1048576,
     available_gpu_mem=10.34 GB
[2026-07-28 07:49:32] INFO:     Application startup complete.
[2026-07-28 07:49:32] INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2026-07-28 07:50:15] The server is fired up and ready to roll!

Put those four stages together and the per-GPU budget is settled: 59.63 GB of weights, 7.03 of pools, about 1.2 for CUDA graph capture, and 10.34 GB still free. Across all 32 cards nvidia-smi showed 70,960 MiB in use of 81,559 MiB, and the server advertised the full context_len=1048576.

How to Deploy Kimi K3 on Hyperstack

Now, let us walk through the step-by-step process of standing up the four-node cluster and serving Kimi K3 across all 32 GPUs.

Step 1: Accessing Hyperstack

First, you will need an account on Hyperstack.

• Go to the Hyperstack website and log in.
• If you are new, create an account and set up your billing information. Our documentation can guide you through the initial setup.

Step 2: Deploying Four 8x NVIDIA H100 Virtual Machines

From the Hyperstack dashboard, launch four identical GPU virtual machines. The node count is fixed by the tensor-parallel arithmetic above: tensor parallel 32 needs 32 GPUs, and nothing smaller divides the model.

• Initiate Deployment: Click the "Deploy New Virtual Machine" button on the dashboard.

• Select Hardware Configuration: Choose an 8x NVIDIA H100-80G-PCIe-NVLink flavour, listed in the flavour reference as n3-H100x8-NVLink. Our run used the Spot variant, n3-H100x8-NVLink-spot, at $2.00 per GPU per hour, which is $16.00/hr per node and $64.00/hr for the cluster. Take the NVLink variant rather than plain PCIe: per NVIDIA's H100 PCIe product brief it bridges each card to a single adjacent card at 600 GB/s, and nvidia-smi topo confirmed the NV12 pairs on our nodes.

• Choose the Operating System: Select the "Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker" image. The driver line has to match what the engine container expects, and this image also ships Docker and the NVIDIA container runtime ready to use.
• Environment: Choose an environment carrying NVIDIA H100 NVLink stock. Our run used CANADA-1. All four nodes must land in the same environment, because that is what puts them on a shared private network, and the entire deployment depends on it.
• Select a Keypair: Choose an existing SSH keypair. Use the same keypair for all four nodes.
• Network Configuration: Assign a Public IP to every node so you can reach them over SSH, and note each node's private IP once it is running.
• Review and Deploy: Check the settings, click "Deploy", and repeat until all four nodes exist with identical settings.

Step 3: Configure the Firewall

The four nodes have to talk to each other freely on the private network, and you need to reach the API from your own machine. Add these firewall rules:

Every node's private IP is shown on its detail page in the dashboard, so you can fill in that third rule without logging in anywhere. It is also the rule people forget. NCCL negotiates its ports dynamically, so pinning it to a single port does not work and the range has to be wide. The source does not have to be wide, though. Scope it to the peer nodes' private addresses, or at most to your environment's own subnet, rather than to the whole 10.0.0.0/8 range: a cloud private network is shared, and "not reachable from the internet" is not the same as "reachable only by my four machines".

Step 4: Accessing Your Nodes

Once all four virtual machines are running, copy their Public IP addresses from the dashboard and connect to each in a separate terminal.

# Connect to each node using your private key and the node's public IP
ssh -i [path_to_your_ssh_key] ubuntu@[your_node_public_ip]

You will also need three pieces of information on every node: its private IP, the name of its private network interface, and confirmation that the large disk is mounted. Run this on all four:

# Private IP and the name of the private interface
ip -o -4 addr show | awk '{print $2, $4}' | grep -E ' 10\.'

# All eight NVIDIA GPUs present, with their real memory size
nvidia-smi --query-gpu=index,name,memory.total --format=csv

# The two disks: a small root and a very large NVMe
df -h / /ephemeral

ens6 10.0.0.219/24

index, name, memory.total [MiB]
0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
...
7, NVIDIA H100 PCIe, 81559 MiB

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        96G   12G   84G  13% /
/dev/vdb1       6.3T   28K  6.0T   1% /ephemeral

On our NVIDIA H100 nodes the private interface was ens6 on the 10.0.0.0/24 subnet, and the head node's private IP was 10.0.0.219. Do not assume these values match yours. The interface name in particular varies by flavour, so read both from the commands above and substitute them wherever this guide shows ens6 or a 10.0.0.x address. Both values go into the launch command in Step 8, so it is worth writing them down now.

The disk check is what shapes the rest of the guide. The machine has two disks and only one of them matters here: the root filesystem is roughly 96 GB, while the multi-terabyte NVMe disk is mounted separately at /ephemeral and offers around 6.3 TB on this flavour. It is documented under ephemeral storage. Kimi K3 is 1.56 TB and the container image adds another 13.3 GB on top, so nothing about this deployment fits on the root disk.

Step 5: Point Docker and the Model Cache at /ephemeral

Two things have to move onto the big disk: the Hugging Face cache that will hold the weights, and Docker's own storage, because container images are pulled onto the root disk by default. Everything in this step runs on all four nodes.

Start with the directories. /ephemeral/hf becomes the Hugging Face cache, and it is the path that gets mounted into the container later so the engine finds the weights already present:

# The Hugging Face cache and Docker's storage both go on the big NVMe disk
sudo mkdir -p /ephemeral/hf /ephemeral/docker
sudo chown -R ubuntu:ubuntu /ephemeral/hf

Next, move Docker itself. The data-root key tells the daemon to store images and layers on the NVMe disk rather than the 96 GB root partition, and it only takes effect on a restart:

# Docker data-root -> /ephemeral (root is only ~96 GB; the image alone is 13.3 GB)
sudo systemctl stop docker docker.socket
echo '{"data-root":"/ephemeral/docker"}' | sudo tee /etc/docker/daemon.json
sudo systemctl start docker
sleep 4

Finally, clear anything Docker has already written to the root disk, then check both halves landed. On a fresh node the prune finds little to remove, and it earns its place when you are re-running on a node that has pulled images before. docker info should now report Docker Root Dir: /ephemeral/docker:

# Reclaim whatever the previous storage driver left on the root disk
docker system prune -af

# Then confirm both halves of the move
docker info | grep 'Docker Root'
df -h /

Step 6: Download Kimi K3 and Pull the Engine Image

Every node needs its own complete copy of the weights, because every rank loads its shard of every tensor from local disk. The image pull and the model download are independent, so start them together on all four nodes. First make sure the Hugging Face client is present:

# Ubuntu 24.04 marks its Python as externally managed, hence the flag
python3 -m pip install -q --break-system-packages huggingface_hub hf_transfer

Kick off the image pull first, because it is the quicker of the two and nothing about it depends on the weights:

# Start the image pull in the background. It is 13.3 GB and it does not
# depend on the weights, so there is no reason to wait for it.
nohup docker pull lmsysorg/sglang:kimi-k3 >/tmp/pull.log 2>&1 &

Then start the weights. Three details in this command are doing the work: HF_TOKEN authenticates the download, because Hugging Face applies tighter rate limits to anonymous traffic than to signed-in users, so a token is what sustains full speed across 1.56 TB; HF_HOME puts the cache on the NVMe disk rather than the root partition, which is the whole point of the previous step; and max_workers=16 is what fills the pipe.

# Authenticate, then pull 1.56 TB of weights onto the NVMe disk.
# max_workers=16 is what fills a multi-gigabit pipe on this flavour.
export HF_TOKEN="YOUR_HF_TOKEN_HERE"
nohup env HF_HOME=/ephemeral/hf HF_HUB_ENABLE_HF_TRANSFER=1 HF_TOKEN="$HF_TOKEN" \
  python3 -c "from huggingface_hub import snapshot_download; snapshot_download('moonshotai/Kimi-K3', max_workers=16)" \
  >/tmp/dl.log 2>&1 &

Both are detached, so you can watch the disk fill rather than the terminal. On our nodes the transfer peaked at 1.8 GB/s and finished in about 32 minutes, an average of roughly 0.8 GB/s across the whole checkpoint:

# Watch the disk fill. Expect roughly 1.5 TB over about half an hour.
watch -n 30 'df -h /ephemeral | tail -1'

Step 7: Verify All 96 Shards Before You Serve

A 32-rank cluster spends about eleven minutes loading before it is in a position to notice that one of the 96 shards is short. Confirming the download first takes seconds on an idle node and gives you a definite answer before the expensive part begins, so it is worth the keystrokes on a checkpoint this size.

The script reads your token from ~/hf_token.txt, so save it there first with echo "$HF_TOKEN" > ~/hf_token.txt. It then builds in four parts, starting with the arguments and the header that authenticates the API call:

#!/usr/bin/env python3
# verify_k3_download.py - confirm a node has the COMPLETE Kimi K3 checkpoint
# before we try to serve. Run ON the node.
import argparse, glob, json, os, urllib.request

ap = argparse.ArgumentParser()
ap.add_argument("--repo", default="moonshotai/Kimi-K3")
ap.add_argument("--hf-home", default="/ephemeral/hf")
ap.add_argument("--token-file", default=os.path.expanduser("~/hf_token.txt"))
a = ap.parse_args()

tok = ""
if os.path.exists(a.token_file):
    tok = open(a.token_file).read().strip()
h = {"User-Agent": "Mozilla/5.0"}
if tok:
    h["Authorization"] = "Bearer " + tok

Next it asks Hugging Face what should be there. The repository metadata lists every file in the repo, and filtering it to .safetensors gives the 96 names to look for:

# expected manifest from HF
req = urllib.request.Request(f"https://huggingface.co/api/models/{a.repo}", headers=h)
meta = json.loads(urllib.request.urlopen(req, timeout=60).read())

expected = [s["rfilename"] for s in (meta.get("siblings") or [])]
exp_st = sorted(f for f in expected if f.endswith(".safetensors"))

Then it finds the local copy. The downloader writes into a snapshot directory named after the commit hash and fills it with symlinks into the blob store, so the size of each link has to be read through os.path.realpath rather than from the link itself:

snap = os.path.join(a.hf_home, "hub", "models--" + a.repo.replace("/", "--"), "snapshots")
dirs = glob.glob(os.path.join(snap, "*"))
if not dirs:
    print(f"MISSING: no snapshot dir under {snap}")
    raise SystemExit(2)
d = dirs[0]

local, total = {}, 0
for f in exp_st:
    p = os.path.join(d, f)
    if os.path.exists(p):
        sz = os.path.getsize(os.path.realpath(p))
        local[f] = sz
        total += sz

Finally it compares the two and exits non-zero if anything is missing, so the whole thing can go in front of your launch command:

missing = [f for f in exp_st if f not in local]
incomplete = glob.glob(os.path.join(d, "*.incomplete"))
cfg_ok = os.path.exists(os.path.join(d, "config.json"))

print(f"repo             : {a.repo}")
print(f"snapshot dir     : {d}")
print(f"safetensors      : {len(local)}/{len(exp_st)} present")
print(f"total bytes      : {total:,} ({total/1e9:.2f} GB)")
print(f"config.json      : {'OK' if cfg_ok else 'MISSING'}")
print(f".incomplete files: {len(incomplete)}")

ok = (not missing) and (not incomplete) and cfg_ok
print("RESULT:", "COMPLETE" if ok else "INCOMPLETE")
raise SystemExit(0 if ok else 1)

Run it on every node. What you want to see, with the same snapshot hash on all four:

repo             : moonshotai/Kimi-K3
snapshot dir     : /ephemeral/hf/hub/models--moonshotai--Kimi-K3/snapshots/9f62e4e9fffbd0a83ddd60e1c209d828994b3569
safetensors      : 96/96 present
total bytes      : 1,560,936,091,448 (1560.94 GB)
config.json      : OK
.incomplete files: 0
RESULT: COMPLETE

Step 8: Launch Kimi K3 Across All Four Nodes

Unlike a pipeline-parallel deployment, there is no separate follower command here. All four nodes run the same command, and only two values change between them: --node-rank and SGLANG_HOST_IP. Rank 0 is the head, and it is the only node that will answer HTTP, but it holds no more of the model than the others.

Set the five values first. Only the first two differ between machines, so the safest way to work is to open four terminals, paste this block into each, and change the two lines:

# Run this on EVERY node. Only RANK and SELF_PRIV change between them.
export RANK=0                     # 0 on the head, then 1, 2 and 3
export SELF_PRIV="10.0.0.219"     # THIS node's private IP
export HEAD_PRIV="10.0.0.219"     # the head node's private IP, on all four
export NIC="ens6"                 # the private interface from Step 4
export HF_TOKEN="YOUR_HF_TOKEN_HERE"

# Clear any container left from an earlier launch
docker rm -f sgl 2>/dev/null || true

Next collect the engine arguments. Putting them in a variable keeps them separate from the container plumbing, and it makes the difference between the four nodes a single line rather than a diff across thirty:

# Everything the engine itself needs, collected in one variable so the
# docker command below stays readable.
K3_ARGS="--trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 32 \
  --ep-size 32 \
  --nnodes 4 \
  --node-rank $RANK \
  --dist-init-addr $HEAD_PRIV:20000 \
  --moe-runner-backend marlin \
  --decode-attention-backend flashmla \
  --mem-fraction-static 0.85 \
  --dist-timeout 3600 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 8000"

Then launch. Everything here is container-level: the GPUs, host networking and shared memory, the weights mounted where the engine looks for them, and the environment the engine reads at start-up:

docker run -d --name sgl --gpus all --network host --ipc=host --shm-size 32g \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env NCCL_CUMEM_ENABLE=1 \
  --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --env SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \
  --env SGLANG_K3_ATTN_RES_MODE=jit \
  --env SGLANG_MOE_FUSED_GATE_RADIX=1 \
  --env SGLANG_HOST_IP=$SELF_PRIV \
  --env NCCL_SOCKET_IFNAME=$NIC \
  --env GLOO_SOCKET_IFNAME=$NIC \
  --env NCCL_IB_DISABLE=1 \
  lmsysorg/sglang:kimi-k3 \
  sglang serve $K3_ARGS

Start rank 0 first, then the other three within a minute or so of each other. The rendezvous on port 20000 waits for all four, and --dist-timeout 3600 gives it an hour before it gives up, which is deliberate on a load this size.

Here is what each part of those three blocks is for:

Step 9: Verify the Deployment

Follow the head node while the cluster forms:

docker logs -f sgl

The timeline below is our run, from the first container log line to the readiness message. Four moments in it are the ones to watch for, and they carry a filled marker.

Now confirm the model is being served, from your own machine:

# Use the HEAD node's PUBLIC IP here, not the private one
export HEAD_IP="your.head.node.public.ip"

curl -s http://$HEAD_IP:8000/v1/models

{
  "object": "list",
  "data": [
    {
      "id": "moonshotai/Kimi-K3",
      "object": "model",
      "owned_by": "sglang",
      "root": "moonshotai/Kimi-K3",
      "max_model_len": 1048576
    }
  ]
}

The max_model_len of 1,048,576 is the line to look for. A 2.8 trillion parameter model is live on 32 GPUs with its full one million token context window intact, not a truncated one.

Finally, look at the memory across all four nodes. This is 2.8 trillion parameters resident on thirty-two NVIDIA H100 GPUs:

=== node 0 (head, rank 0) ===
0, NVIDIA H100 PCIe, 71790 MiB, 81559 MiB
1, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...
=== node 1 (rank 1) ===
0, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...
=== node 2 (rank 2) ===
0, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...
=== node 3 (rank 3) ===
0, NVIDIA H100 PCIe, 70960 MiB, 81559 MiB
...

Thirty-one of the thirty-two cards read 70,960 MiB of 81,559 MiB. One card reads 71,790 MiB, about 830 MiB more, which is consistent with the rank that also carries the HTTP server process. That leaves about 10.3 GB of headroom on every card, matching the engine's own available_gpu_mem=10.34 GB exactly. Kimi K3 is live.

Talking to Kimi K3

The endpoint is OpenAI compatible, so every example below is a standard chat completion. What is specific to Kimi K3 is not the transport but the behaviour: it always thinks, it returns that thinking in a separate field, and it expects you to hand that thinking back on the next turn.

The first request, and the field most clients ignore

curl -s http://$HEAD_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "moonshotai/Kimi-K3",
    "messages": [{"role": "user", "content": "Who are you? One sentence."}],
    "max_tokens": 256
  }'

The reply is a standard chat completion with one extra field. Here is the message it returned:

"message": {
  "role": "assistant",
  "content": "Hello! I'm Kimi, an AI assistant developed by Moonshot AI.",
  "reasoning_content": "They specify background identity: current assistant is Kimi,
                        developed by Moonshot AI. Need answer naturally, short
                        sentence. Final only.",
  "tool_calls": null
}

The answer is in content as usual, but reasoning_content is populated alongside it. That field exists because the server was launched with --reasoning-parser kimi_k3, and without it the thinking would arrive inline in the answer. Now look at what it cost:

"usage": {
  "prompt_tokens": 137,
  "completion_tokens": 135,
  "total_tokens": 272,
  "reasoning_tokens": 109
}

reasoning_tokens reports 109 of the 135 completion tokens. On a question this small, 81 per cent of the generated tokens were thinking, and a client that reads only content pays for all of them while seeing none of them.

Every Python example from here on shares one small helper. It is the standard library only, so there is nothing to install, and it is the same request shape the shell examples above send:

import json
import time
import urllib.request

BASE = "http://<HEAD_NODE_PUBLIC_IP>:8000/v1"
MODEL = "moonshotai/Kimi-K3"
KEY = "EMPTY"

def post(body, timeout=1800):
    req = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.load(r), time.time() - t0

With that in place a chat request is a dictionary. The only field that is not standard OpenAI is reasoning_effort, and it sits at the top level of the body rather than inside chat_template_kwargs as some other models expect:

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain what a mixture-of-experts model is, simply, in 2 sentences."},
]

resp, dt = post({"model": MODEL, "messages": messages, "max_tokens": 1024,
                 "temperature": 0.8, "top_p": 1.0,
                 "reasoning_effort": "high"})     # low, high or max

Reading the reply is where the difference shows. The two halves come back on separate keys, so you can log the thinking and show the answer:

m = resp["choices"][0]["message"]
content = m.get("content") or ""
reasoning = m.get("reasoning_content") or ""

u = resp.get("usage", {})
ct = u.get("completion_tokens", 0)

print("--- thinking ---")
print(reasoning)              # the chain of thought lands here
print("--- answer ---")
print(content)                # the final answer lands here
print(f"    [{ct} tok in {dt:.1f}s = {ct / dt:.1f} tok/s]")

[2026-07-28 07:57:13] Kimi K3 supports thinking_effort low/high/max;
                      ignoring reasoning_effort='no_think'.

Thinking effort: what the knob actually changes

The interesting question is not whether the effort setting works, but what it moves. We put the same word problem through at low and at high and compared the token accounting:

prompt = ("Three friends split a bill. Ana pays twice what Ben pays, and Ben pays 3 more "
          "than Cara. The total is 43. How much does each pay?")

for effort in ("low", "high"):
    resp, dt = post({"model": MODEL, "max_tokens": 2000, "reasoning_effort": effort,
                     "messages": [{"role": "user", "content": prompt}]})
    u = resp.get("usage", {})
    print(f"  [{effort:>4}] reasoning_tokens={u.get('reasoning_tokens')} "
          f"completion={u.get('completion_tokens')} {dt:.1f}s")

The token counts move sharply, and the wall clock with them:

  [ low] reasoning_tokens=54 completion=202 33.6s
  [high] reasoning_tokens=184 completion=344 56.6s

The thinking itself is where the difference is visible. At low it is one line. At high the model restates the problem, solves it, checks the total and then considers whether non-integer amounts are acceptable:

# the whole of the low-effort thinking:
Cara = c, Ben = c+3, Ana = 2c+6. Total 4c+9 = 43, c=8.5. Ben 11.5, Ana 23. Total 43

# the opening of the high-effort thinking:
The user wants me to solve a word problem. Let me work through it.
Let Cara pay x. Ben pays 3 more than Cara: Ben = x + 3.
Ana pays twice what Ben pays: Ana = 2(x + 3) = 2x + 6.
Total: x + (x + 3) + (2x + 6) = 43 ...
Non-integer amounts, but that's fine (money can be .50).

# both answers identical: Cara 8.50, Ben 11.50, Ana 23.00

That is a clean result for capacity planning. The effort setting is a latency and cost dial, not a verbosity dial. At low the model wrote a single compressed line of working. At high it restated the problem, solved it, checked the total and then explicitly considered whether non-integer amounts were acceptable. On a problem this size the extra thinking bought nothing, which is exactly the sort of thing worth measuring on your own workload before you default the whole fleet to max.

Preserved thinking history, which is a requirement rather than an option

Kimi K3 was trained in preserved thinking history mode. The model card is direct about what happens otherwise: if the harness fails to pass back the historical thinking content, or if a session running on another model is switched over to K3 mid-conversation, generation quality may become highly unstable. In practice that means the assistant message you append to messages has to carry reasoning_content as well as content and tool_calls.

Here is a probe that can only be answered correctly if the previous turn's thinking survived the round trip. Turn one is an ordinary request:

msgs = [{"role": "user", "content": "Tell me three random numbers."}]

resp1, dt1 = post({"model": MODEL, "messages": msgs, "max_tokens": 1024,
                   "temperature": 0.8, "top_p": 1.0, "reasoning_effort": "high"})
m1 = resp1["choices"][0]["message"]

This next block is the one that matters. The assistant message goes back onto messages with its reasoning attached, rather than being rebuilt from content alone:

# pass the assistant message back AS-IS, including reasoning_content
assistant_turn = {"role": "assistant", "content": m1.get("content") or ""}
if m1.get("reasoning_content"):
    assistant_turn["reasoning_content"] = m1["reasoning_content"]
msgs.append(assistant_turn)

Turn two then asks about something that only ever existed inside that thinking:

msgs.append({"role": "user",
             "content": "What other numbers did you have in mind while thinking? Name them."})

resp2, dt2 = post({"model": MODEL, "messages": msgs, "max_tokens": 1024,
                   "temperature": 0.8, "top_p": 1.0, "reasoning_effort": "high"})
m2 = resp2["choices"][0]["message"]

print(f"    turn1: {(m1.get('content') or '')[:90]}")
print(f"    turn2: {(m2.get('content') or '')[:120]}")

    turn1: 17, 482, 96
    turn2: I didn't have specific alternates in mind, I just picked three.
           Three more random ones could be: 8, 157, 642.

The model read its own prior reasoning, correctly reported that it had not held back any alternatives, and offered three new numbers instead of inventing a memory. That is the behaviour you want, and it is the behaviour you lose if the assistant turn is rebuilt from content alone.

Agentic tool calling with the kimi_k3 parser

Because the server was launched with --tool-call-parser kimi_k3, the model decides for itself when to reach for a tool and the parser converts its output into standard OpenAI tool_calls. The first of the two tools is a calculator, and it walks the expression as an abstract syntax tree rather than calling eval, so anything other than arithmetic raises a ValueError rather than reaching a shell:

import ast
import operator

ops = {
    ast.Add:  operator.add,
    ast.Sub:  operator.sub,
    ast.Mult: operator.mul,
    ast.Div:  operator.truediv,
    ast.Pow:  operator.pow,
    ast.USub: operator.neg,
}

The calculator itself recurses over that tree. Anything that is not a number or one of the five permitted operators raises rather than evaluating:

def calculate(expression: str):
    def ev(n):
        if isinstance(n, ast.Constant):
            return n.value
        if isinstance(n, ast.BinOp):
            return ops[type(n.op)](ev(n.left), ev(n.right))
        if isinstance(n, ast.UnaryOp):
            return ops[type(n.op)](ev(n.operand))
        raise ValueError("bad expr")

    return {"result": ev(ast.parse(expression, mode="eval").body)}

The second tool is a stub, and a dispatch map ties the two names back to the two functions:

def get_weather(city: str):
    fake = {"Tokyo": "22C, clear", "London": "14C, rain", "Cairo": "35C, sunny"}
    return {"city": city, "weather": fake.get(city, "18C, partly cloudy")}

IMPL = {"calculate": calculate, "get_weather": get_weather}

Then describe the same two functions to the model. This is the standard OpenAI tools schema, with nothing Kimi-specific in it:

CALCULATE_TOOL = {
    "type": "function",
    "function": {
        "name": "calculate",
        "description": "Evaluate a basic arithmetic expression.",
        "parameters": {
            "type": "object",
            "properties": {
                "expression": {"type": "string", "description": "e.g. '47*89'"},
            },
            "required": ["expression"],
        },
    },
}

The second entry has the same shape, and the two go into the list the request will carry:

WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}

TOOLS = [CALCULATE_TOOL, WEATHER_TOOL]

The loop itself is the usual pattern. Send the task, and stop as soon as a reply comes back without tool calls, because that reply is the answer:

task = ("What is 47 * 89? Also, what's the weather in Tokyo? "
        "Use your tools, then give me one combined answer.")
messages = [{"role": "system", "content": "You are a helpful assistant with tools."},
            {"role": "user", "content": task}]

for turn in range(6):
    resp, dt = post({"model": MODEL, "messages": messages, "tools": TOOLS,
                     "max_tokens": 1024, "temperature": 0.7, "reasoning_effort": "high"})
    m = resp["choices"][0]["message"]
    tcs = m.get("tool_calls") or []
    print(f"[turn {turn}] tool_calls={len(tcs)} "
          f"reasoning={len(m.get('reasoning_content') or '')} chars")

    if not tcs:
        print(m.get("content", ""))
        break

If there were tool calls, the assistant turn goes back onto messages first, and it carries the same three things every time:

    # echo the assistant message back AS-IS (content + reasoning_content + tool_calls)
    at = {"role": "assistant", "content": m.get("content") or "", "tool_calls": tcs}
    if m.get("reasoning_content"):
        at["reasoning_content"] = m["reasoning_content"]
    messages.append(at)

Then each requested tool runs and its result is appended as a tool message, which is what the model reads on the next pass:

    for tc in tcs:
        fn = tc["function"]["name"]
        try:
            args = json.loads(tc["function"].get("arguments") or "{}")
            result = IMPL[fn](**args)
        except Exception as e:
            args, result = {}, {"error": str(e)}

        print(f"    -> {fn}({args}) = {result}")
        messages.append({"role": "tool", "tool_call_id": tc.get("id", fn),
                         "name": fn, "content": json.dumps(result)})

Kimi K3 requested both tools in a single turn rather than serialising them into two round trips. Its reasoning for that turn, 116 characters of it, reads: "The user wants 47*89 and the weather in Tokyo. I should call both tools in the same block since they are independent."

[turn 0] tool_calls=2 reasoning=116 chars
    -> calculate({'expression': '47*89'}) = {'result': 4183}
    -> get_weather({'city': 'Tokyo'}) = {'city': 'Tokyo', 'weather': '22C, clear'}
[turn 1] tool_calls=0 reasoning=37 chars

Here are both results:

- 47 x 89 = 4,183
- Tokyo weather: 22C and clear

So, 47 times 89 equals 4,183, and it's currently a pleasant clear day at 22C in Tokyo!

The whole exchange took 36.25 seconds across two round trips. Parallel tool calls in one turn are the behaviour you want from an agentic model, and the kimi_k3 parser delivers it with no orchestration code of your own.

Native vision, tested against an image we generated ourselves

Kimi K3 is natively multimodal through MoonViT-V2, so images go into the same endpoint as text. Rather than describe a photograph and grade the answer by eye, we built the test image in the script: a 128 by 128 PNG whose left half is pure red and whose right half is pure blue, written byte by byte with nothing but the standard library. The correct answer is therefore known before the request is sent.

import base64
import struct
import zlib

def make_png(w=128, h=128):
    rows = b""
    for _ in range(h):
        row = b"\x00"                                    # filter byte
        for x in range(w):
            row += b"\xff\x00\x00" if x < w // 2 else b"\x00\x00\xff"
        rows += row

    def chunk(tag, data):
        c = struct.pack(">I", len(data)) + tag + data
        return c + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)

    ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)    # 8-bit truecolour
    return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
            + chunk(b"IDAT", zlib.compress(rows, 9)) + chunk(b"IEND", b""))

Sending it is the same call as any other, with the image as a second content part rather than a separate endpoint:

b64 = base64.b64encode(make_png()).decode()

resp, dt = post({"model": MODEL, "max_tokens": 400, "reasoning_effort": "low",
                 "messages": [{"role": "user", "content": [
                     {"type": "text", "text": "What do you see in this image? "
                                              "Name the colours and how they are arranged."},
                     {"type": "image_url",
                      "image_url": {"url": f"data:image/png;base64,{b64}"}}]}]})

ans = (resp["choices"][0]["message"].get("content") or "").strip()
print(f"  latency {dt:.1f}s | usage={resp.get('usage')}")
print(f"  ANSWER: {ans[:300]}")

  latency 18.4s | usage={'prompt_tokens': 174, 'total_tokens': 272,
                          'completion_tokens': 98, 'reasoning_tokens': 54,
                          'prompt_tokens_details': {'cached_tokens': 0, 'image_tokens': 25}}
  ANSWER: The image shows two solid color blocks arranged side by side: the left half
          is red and the right half is blue, meeting along a sharp vertical line down
          the center.

That answer is objectively correct, including the vertical boundary down the centre. The usage block is the interesting part for anyone budgeting a vision workload: the whole image cost 25 image tokens, and the request completed in 18.4 seconds. Vision on Kimi K3 is not a bolt-on adapter, and it is priced in tokens like everything else.

Throughput on 32 NVIDIA H100 GPUs

Single-stream decode across our three chat prompts averaged 5.8 tokens per second, with individual prompts at 5.9, 5.8 and 5.6. The more useful question for a sparse MoE with room for 49 concurrent requests is what happens when requests arrive together, so we fired four at once. The script is standard library only, so it runs anywhere:

import json
import threading
import time
import urllib.request

BASE = "http://<HEAD_NODE_PUBLIC_IP>:8000/v1"
MODEL = "moonshotai/Kimi-K3"
N = 4

One worker sends one request and records what it got back:

def one(i, res):
    body = {"model": MODEL, "max_tokens": 128, "temperature": 0.9,
            "messages": [{"role": "user",
                          "content": f"Write one interesting paragraph about topic #{i}: the deep ocean."}]}
    req = urllib.request.Request(
        BASE + "/chat/completions", data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json", "Authorization": "Bearer EMPTY"})

    t0 = time.time()
    with urllib.request.urlopen(req, timeout=600) as r:
        resp = json.load(r)

    res[i] = {"completion_tokens": resp["usage"]["completion_tokens"],
              "latency_s": time.time() - t0}

Then start all four at once and time the batch as a whole, which is the number that matters for a server rather than for a single user:

res = {}
ths = [threading.Thread(target=one, args=(i, res)) for i in range(N)]

t0 = time.time()
for t in ths:
    t.start()
for t in ths:
    t.join()
wall = time.time() - t0

toks = sum(v["completion_tokens"] for v in res.values())
agg = toks / wall
print(json.dumps({"concurrency": N, "wall_s": round(wall, 2),
                  "total_completion_tokens": toks,
                  "aggregate_tok_per_s": round(agg, 1),
                  "per_stream_tok_per_s": round(agg / len(res), 1)}, indent=2))

{
  "concurrency": 4,
  "wall_s": 36.55,
  "total_completion_tokens": 512,
  "aggregate_tok_per_s": 14.0,
  "per_stream_tok_per_s": 3.5
}

What sets that figure, and what would move it

Those numbers come from a bring-up configuration, and every choice in it is a deliberate trade for a fast, cheap start rather than for peak throughput:

• The MoE path is Marlin. Every expert matrix multiply dequantises MXFP4 into 16 bits on the way through, because the native MXFP4 backends target Blackwell.
• The cards are PCIe rather than SXM. NVLink bridges pairs of cards, and the remaining hops inside each node go through the PCIe host bridge.
• Collectives run over Ethernet. Twenty-four of the thirty-two ranks sit on a different machine from rank 0, so their all-reduces cross the private network.
• Speculative decoding is off. Kimi K3 ships a draft path that the published recipes pair with the model, and we ran without it to keep the launch minimal.
• Every request thinks. Kimi K3 has no no-think mode, so even a one-line answer pays for a reasoning pass. On our first request that was 109 of 135 tokens.

For scale at the other end of the range, the vLLM launch blog reports 111 tokens per second per user on TP 8 and 118 on TP 16 at batch size 1, rising to 331 and 370 with DSpark speculative decoding, measured on NVIDIA GB300 NVL72. That is a different hardware class, with native FP4 kernels and NVLink across the whole domain, and the distance between the two is a fair picture of what each of the trades above is worth.

So this is the shape of deployment to reach for when the job is evaluation, integration and correctness work: proving an agent harness against the model itself, measuring how your prompts behave at each thinking effort, testing vision inputs, and validating the tool-call plumbing end to end, on hardware you can book by the minute. When throughput becomes the requirement, the levers in order of effect are more memory per card, then native FP4, then speculative decoding.

Cost and Tearing Down the Cluster

At $2.00 per GPU per hour on Spot, thirty-two NVIDIA H100 GPUs cost $64.00 per hour. Here is where that hour goes on a clean run that follows this guide:

The download dominates, at about 62 per cent of the bill, and it is the one phase that does not get cheaper with faster GPUs. Everything after it is minutes. Add whatever time you intend to spend actually using the endpoint on top, at roughly $1.07 per minute.

When you are finished, stop the containers and delete the virtual machines. This is the step you must not skip:

# On each node, stop the container first so the NVIDIA GPUs are released cleanly
docker rm -f sgl

# Then delete all four virtual machines from the Hyperstack dashboard,
# or through the API, and confirm they have gone.
curl -s -X DELETE https://infrahub-api.nexgencloud.com/v1/core/virtual-machines/$VM_ID \
  -H "api_key: $HYPERSTACK_API_KEY"

Serve an open 3T-class model

Run Kimi K3 across thirty-two NVIDIA H100 GPUs

Four nodes on Spot at $64.00 per hour. Our run downloaded 1.56 TB, launched 32 ranks and reached a live one-million-token endpoint eleven minutes after the containers started.

32x NVIDIA H100TP 32 x EP 321M contextReady in 11 minutes

Launch an NVIDIA GPU cluster on Hyperstack today.

FAQs

What hardware do you need to run Kimi K3?

The checkpoint is 1,560.94 GB across 96 shards and every expert has to stay resident, so the constraint is total GPU memory rather than compute. The official vLLM recipe asks for at least one 8x NVIDIA B300 node. This guide serves it on 32x NVIDIA H100 80GB instead, across four Hyperstack nodes.

What is the maximum tensor-parallel size for Kimi K3?

Thirty-two. Tensor parallelism has to divide both the 96 attention heads and the 7168 hidden size, and their greatest common divisor is 32. Adding a fifth node therefore cannot lower the weights each GPU carries.

Why does the Kimi K3 endpoint always return reasoning_content?

Kimi K3 always thinks: it accepts an effort of low, high or max, and nothing turns thinking off. Launching with --reasoning-parser kimi_k3 puts that thinking in its own field, and you have to pass it back on the next turn or later replies become unstable.

How fast is Kimi K3 on 32x NVIDIA H100?

We measured 5.8 tokens per second single stream and 14.0 aggregate at four-way concurrency, on the Hopper Marlin path without speculative decoding. For contrast, the vLLM launch blog reports 111 per user on TP 8 at batch size 1, measured on NVIDIA GB300 NVL72.

How much does it cost to run Kimi K3 on Hyperstack?

Four 8x NVIDIA H100 Spot nodes are $2.00 per GPU per hour, so $64.00 per hour for the 32-GPU cluster. A clean run reaches a live endpoint in about 52 minutes, or roughly $55. Delete the machines afterwards, because the ephemeral disk holding the weights goes with them.