Deploy Kimi K3.1 Pro on Hyperstack: A Comprehensive Guide for Coders

Kimi K3.1 Pro is Moonshot AI's open-weight, natively multimodal agentic model, post-trained from Kimi K3. Its sparse Mixture-of-Experts architecture has 2.8 trillion parameters, with 104 billion active per token, and a 1-million-token context window. Moonshot AI reports improvements in coding, 3D creation, financial research and factual accuracy, with benchmark results competitive with leading proprietary models.

This tutorial serves Kimi K3.1 Pro on 32 NVIDIA H100 GPUs across four Hyperstack virtual machines with SGLang, the same cluster shape as our Kimi K3 deployment, and then puts the endpoint to work: reasoning effort, preserved thinking, tool calls, image input, a coding agent, and Claude Code, Kimi Code CLI and OpenClaw on top. For the model family behind it, see our technical deep dive into Kimi K3.

What Is Kimi K3.1 Pro?

Kimi K3.1 Pro is built on the Kimi K3 architecture, with one major change to attention. Here is how the architecture works:

1. Sparse MoE Routing: 896 experts in total, with 16 selected per token plus 2 shared experts, so only 104B of the 2.8T parameters are active for any single token.
2. Hybrid Attention Stack: 93 layers made up of 69 Kimi Delta Attention (KDA) layers and 24 Gated DSA layers, with one dense layer.
3. Gated DeepSeek Sparse Attention: Compared with Kimi K3, the full-attention Gated MLA layers are replaced with Gated DSA. A lightweight indexer with 32 heads scores every earlier token and keeps the top 2,048, and attention then runs exactly over only those tokens. The vLLM team reports that at long context a decode step reads about a quarter of the bytes per layer that dense MLA reads.
4. Native MXFP4 Quantisation: The routed-expert weights are stored in MXFP4 with MXFP8 activations, trained with quantisation-aware training rather than quantised after the fact. Attention, the shared experts and the vision tower stay in BF16.
5. Configurable Thinking Effort: Reasoning depth is set per request with reasoning_effort, which accepts "low", "high" and "max" (default "max").
6. Native Multimodal Fusion (MoonViT-V2): A 401M-parameter vision encoder is integrated directly, so the model accepts images alongside text in the same request.

Where the 24 Sparse Layers Sit

The layer pattern is regular. Three Kimi Delta Attention layers are followed by one Gated DSA layer, all the way down the stack, and the last two layers are both sparse. In the model configuration the DSA layers are 4, 8, 12 and every fourth layer up to 92, plus 93.

How Gated DSA Picks 2,048 Tokens

Each sparse layer runs a small indexer next to its MLA attention. The indexer keeps one FP8 key per token, 132 bytes with its scale, scores every earlier token with 32 lightweight heads and keeps the 2,048 highest-scoring ones. MLA then attends exactly, but only over those tokens, reading their full 576-byte latents.

The saving grows with the context. Dense MLA reads 576 bytes per earlier token per layer. DSA still scans every earlier token, but only its 132-byte index key, and then reads full latents for a fixed 2,048 tokens. The chart works that arithmetic through from 4K to 1M tokens.

Kimi K3.1 Pro Features

Kimi K3.1 Pro is designed for long-running, multi-step work rather than single-turn chat:

• Agentic Coding: Scores 65.7 on Terminal-Bench 4.0, 51.7 on SWE Marathon v1.1 and 75.5 on DeepSWE v1.1, with the coding results evaluated through an agent harness.
• Agentic Search and Tool Use: Scores 92.1 on BrowseComp and 70.0 on HLE-Full with tools, and 73.3 on Automation Bench-AA.
• Office and Knowledge Work: Moonshot AI highlights financial research and document work, with 64.4 on OfficeQA Pro v2 and 85.7 on GDP.pdf with Python.
• Factual Accuracy: On Moonshot AI's internal hallucination benchmark with tools, Kimi K3.1 Pro records a claim-level hallucination rate of 8.3 per cent (lower is better), down from 23.4 per cent for Kimi K3.
• 3D Creation: Scores 72.8 on BenchCAD without tools and 92.7 with tools.
• Multimodal and Long Context: A 1,048,576-token context window with native image understanding through MoonViT-V2.

Against Kimi K3 itself, the Kimi K3.1 Pro release reports gains on every internal benchmark it lists:

Kimi K3.1 Pro vs Leading Models on Coding and Agentic Benchmarks

The deepest purple marks the best score in each row; on the hallucination row that is the lowest. Kimi K3.1 Pro and DeepSeek-V4.1-Flash-1017 run at max effort; the proprietary models at max effort with fallback, as the model card reports them.

What Kimi K3.1 Pro Builds

Moonshot AI published a set of projects that Kimi K3.1 Pro built end to end, across game engines, web pages and 3D tools. The coding examples run in Unreal Engine and Godot, and mix in web search and 3D generation. Pick a project under the player to switch the clip.

Celestial Palace

Sword Flight

Godot Adventure

Videos: Moonshot AI, Kimi K3.1 Pro release blog and Kimi K3.1 Pro quickstart.

One of the coding projects is still live. Kimi K3.1 Pro generated a browser-based 3D robotic-arm reveal with exploded views, transparent shells and cross-sections: 2,317 parts and 268 dynamic wire routes, all produced in code and checked by 55 passing browser tests. It runs below.

The live robotic-arm page. Open it full screen at ayhiidmm4ui46.ok.kimi.link. Built by Kimi K3.1 Pro, published by Moonshot AI.

A Long-Horizon Run: Teaching a Body to Walk

Kimi K3.1 Pro's long-horizon coding is strengthened through large-scale reinforcement learning. In one internal test it built a human walking dynamics simulator, with training and evaluation pipelines, on top of an open-source anatomical model of the human body. When reward shaping failed to improve arm motion, it ran open-loop probes of the simulator's dynamics, corrected errors in the shoulder-muscle moment-arm calculations and retrained the controller.

The walking simulator, from falling to a natural gait. Video: Moonshot AI, Kimi K3.1 Pro release blog.

3D Modelling and Design

Without MCP connectors, Kimi K3.1 Pro writes and runs scripts for tools such as Blender and CAD software to build and refine assets, edit components, and adjust geometry, materials and scene layout. On BenchCAD, which tests 3D reconstruction from multi-view renderings through CAD code generation, it scores 92.7 with tools. These are four of the assets it built:

Turbofan

Cathedral

Rainforest Valley

Modular Lamp

Videos: Moonshot AI, Kimi K3.1 Pro release blog and Kimi K3.1 Pro quickstart.

Kimi K3.1 Pro Deployment Architecture on 32 NVIDIA H100 GPUs

Kimi K3.1 Pro keeps the MXFP4 expert format of Kimi K3, and every expert has to stay resident in GPU memory. The checkpoint is 1,561,293,160,568 bytes across 96 shards in the repository file list, about 357 MB more than Kimi K3, in line with the small indexer added to each of the 24 sparse layers. That number decides the shape of the cluster.

An NVIDIA H100 "80GB" card is a rounded label. On this flavour nvidia-smi reports 81,559 MiB per card and PyTorch counts 79.19 GiB of usable capacity, so thirty-two cards come to about 2,534 GiB. Set the 1,454 GiB of weights against that and the deployment fits, with about 1,080 GiB left for the KV cache, the recurrent state that Kimi Delta Attention keeps for every request, the activations and the communication buffers.

Why Tensor Parallel Has to Be 32

Tensor parallelism has to divide both the 96 attention heads and the 7168 hidden size evenly. Their greatest common divisor is 32, so the valid tensor-parallel sizes are 1, 2, 4, 8, 16 and 32, and only the largest leaves the weights under the capacity of an 80 GB card.

So four nodes of eight NVIDIA H100 GPUs is the shape for this model on this generation. A fifth node cannot lower the per-GPU weight load, because 40 does not divide 96, and data parallelism replicates the model rather than sharding it.

One Model, Four Machines

A multi-node deployment runs tensor parallelism across the node boundary, with expert parallelism laid over the top: thirty-two equal ranks in one process group. Three things follow, and they shape every command below. All four nodes sit in the same environment, which puts them on a shared private network. Collectives are pinned to that private interface with NCCL_SOCKET_IFNAME and run over TCP. And every node keeps its own full copy of the weights on its ephemeral disk.

The Kernel Path on NVIDIA H100

Kimi K3.1 Pro's routed experts are MXFP4, and SGLang resolves its native MXFP4 runner automatically only on NVIDIA Blackwell. On NVIDIA H100 the launch selects the Marlin W4A16 kernel with --moe-runner-backend marlin: weights stay four-bit in memory and are expanded inside the kernel for each matrix multiply, the same path that serves Kimi K3 on this cluster.

The attention side changes with Kimi K3.1 Pro. The SGLang Kimi K3.1 Pro cookbook notes that SGLang serves the model with an FP8 KV cache and a page size of 64, that the MLA latent and indexer keys take about 18.9 KB per token, and that tensor parallelism replicates them on every GPU. Its tuning notes add that the recurrent KDA state pool, held per request, sets the concurrency ceiling, and its NVIDIA H100 layout caps concurrency with --max-running-requests 64.

Run Kimi K3.1 Pro on Hyperstack

Serve a 3T-Class Open Model on NVIDIA GPUs You Control

Four 8x NVIDIA H100 nodes, one SGLang command, and an endpoint that Claude Code, Kimi Code CLI and OpenClaw can drive.

32x NVIDIA H100SGLangOpenAI and Anthropic APIsPer-minute billing

Get started with Hyperstack in minutes.

How to Deploy Kimi K3.1 Pro on Hyperstack

Now, let us walk through the step-by-step process of deploying the necessary infrastructure, from four empty virtual machines to a live endpoint.

Step 1: Accessing Hyperstack

First, you will need an account on Hyperstack.

• Go to the Hyperstack console and log in.
• If you are new, create an account and set up your billing information. Our documentation can guide you through the initial setup.

Step 2: Deploying Four 8x NVIDIA H100 Virtual Machines

From the Hyperstack dashboard, launch four identical GPU virtual machines. Tensor parallel 32 needs 32 GPUs, so the node count follows from the arithmetic above.

• Create a Network-Optimised Environment: On the Environments page, click Create New Environment, name it and select CANADA-1. Hyperstack marks network-optimised environments with the ⚡ icon, and inside one, compatible GPU virtual machines reach inter-VM bandwidth of up to 350 Gbps, against up to 125 Gbps in legacy environments. NVIDIA H100 PCIe with NVLink is on the compatible list, and at tensor parallel 32 every layer exchanges data across all four nodes.
• Initiate Deployment: Look for the "Deploy New Virtual Machine" button on the dashboard and click it.

The Deploy New Virtual Machine button.

• Select Hardware Configuration: Choose the 8x NVIDIA H100-80G-PCIe-NVLink flavour, listed in the flavour reference as n3-H100x8-NVLink with 252 CPUs, 1,440 GB of RAM, a 100 GB root disk and 6,500 GB of ephemeral storage. Take the NVLink variant rather than plain PCIe: per the NVIDIA H100 PCIe product brief it bridges each card to an adjacent card at 600 GB/s.

The flavour card for 8x NVIDIA H100-80G-PCIe-NVLink.

• Choose the Operating System: Select the "Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker" image. The image reference lists it with the NVIDIA driver, the CUDA toolkit and Docker installed, so the node is ready for containers as soon as you connect.
• Environment: Choose the network-optimised environment you created. All four nodes must land in the same environment, because that is what puts them on a shared private network.
• Select a Keypair: Choose an existing SSH keypair from your account to securely access the VMs. Use the same keypair for all four nodes.
• Network Configuration: Ensure you assign a Public IP to every node. This is crucial for remote management and connecting your local development tools.
• Review and Deploy: Double-check your settings, click the "Deploy" button, and repeat until four identical nodes are running.

The OS image list. Choose Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker.

Step 3: Configure the Firewall

The four nodes have to talk to each other freely on the private network, and you need to reach the API from your own machine. On each node's Firewall tab, add these firewall rules:

Adding an inbound rule on the Firewall tab. Use your own address followed by /32 rather than 0.0.0.0/0.

Every node's private IP is shown on its detail page in the dashboard, so you can fill in the third rule before logging in anywhere. NCCL negotiates its ports dynamically, so that range has to be wide, but its source does not: scope it to the peer nodes' private addresses.

Step 4: Accessing Your Nodes

Once your VMs are running, find each node's Public IP address in the Hyperstack dashboard and connect to it in a separate terminal.

# Connect to each node using your private key and the node's public IP
ssh -i [path_to_your_ssh_key] ubuntu@[your_node_public_ip]

Replace [path_to_your_ssh_key] with the path to your private SSH key file and [your_node_public_ip] with the IP address of the node. Then, on every node, read its private IP and interface name, and confirm that all eight GPUs are visible and the large disk is mounted:

# Private IP and the name of the private interface
ip -o -4 addr show | awk '{print $2, $4}' | grep -E ' 10\.'

# All eight NVIDIA GPUs present, with their memory size
nvidia-smi --query-gpu=index,name,memory.total --format=csv

# The two disks: a small root and a very large ephemeral disk
df -h / /ephemeral

ens6 10.0.0.219/24

index, name, memory.total [MiB]
0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
...
7, NVIDIA H100 PCIe, 81559 MiB

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        96G   12G   84G  13% /
/dev/vdb1       6.3T   28K  6.0T   1% /ephemeral

On these nodes the private interface is ens6 on a 10.0.0.0/24 subnet. Read both values from your own output and substitute them wherever this tutorial shows ens6 or a 10.0.0.x address; they go into the launch command in Step 8. The disk check shapes the rest: the root filesystem is about 96 GB, while the ephemeral disk mounted at /ephemeral offers about 6.3 TB.

Step 5: Prepare Storage on the Ephemeral Disk

The Kimi K3.1 Pro checkpoint is far too large for the root partition, so both the model weights and Docker's own storage go on the ephemeral disk mounted at /ephemeral. Everything in this step runs on all four nodes.

# The Hugging Face cache and Docker's storage both go on the ephemeral disk
sudo mkdir -p /ephemeral/hf /ephemeral/docker
sudo chown -R ubuntu:ubuntu /ephemeral/hf

Next, move Docker itself. The data-root key tells the daemon to store images and layers on the ephemeral disk rather than the root partition, and it takes effect on a restart:

# Docker data-root -> /ephemeral (root is only ~96 GB; the image is about 16 GB)
sudo systemctl stop docker docker.socket
echo '{"data-root":"/ephemeral/docker"}' | sudo tee /etc/docker/daemon.json
sudo systemctl start docker
sleep 4

Finally, clear anything Docker has already written to the root disk, then confirm the move. docker info should report Docker Root Dir: /ephemeral/docker:

# Reclaim whatever the previous storage driver left on the root disk
docker system prune -af

# Then confirm both halves of the move
docker info | grep 'Docker Root'
df -h /

Step 6: Pre-Download the Model Weights

The download is large enough that it is worth running once outside Docker, so you can monitor progress and recover cleanly from interruptions. Every node needs its own complete copy, so run this step on all four nodes at the same time. First make sure the Hugging Face client is present:

# Ubuntu 24.04 marks its Python as externally managed, hence the flag
python3 -m pip install -q --break-system-packages -U huggingface_hub

Start the image pull first. It does not depend on the weights, so it can run while they download. The tag is the one the SGLang Kimi K3.1 Pro cookbook names for NVIDIA H100:

# Start the image pull in the background. It does not depend on the
# weights, so there is no reason to wait for it.
nohup docker pull lmsysorg/sglang:dev-kimi-k31-pro >/tmp/pull.log 2>&1 &

Then start the weights. HF_TOKEN authenticates the download, because Hugging Face applies tighter rate limits to anonymous traffic than to signed-in users, and HF_HOME puts the cache on the ephemeral disk:

# Authenticate, then pull 1.56 TB of weights onto the ephemeral disk.
# max_workers=16 runs sixteen file downloads in parallel.
export HF_TOKEN="YOUR_HF_TOKEN_HERE"
nohup env HF_HOME=/ephemeral/hf HF_TOKEN="$HF_TOKEN" \
  python3 -c "from huggingface_hub import snapshot_download; snapshot_download('moonshotai/Kimi-K3.1-Pro', max_workers=16)" \
  >/tmp/dl.log 2>&1 &

# Watch the disk fill towards 1.56 TB
watch -n 30 'df -h /ephemeral | tail -1'

Important: keep HF_HOME=/ephemeral/hf on the download command. Without it, the cache defaults to ~/.cache/huggingface on the 96 GB root disk, which cannot hold a 1.56 TB checkpoint.

Step 7: Check All 96 Shards Before You Serve

A 32-rank cluster loads for several minutes before it is in a position to notice that one of the 96 shards is short. Checking the download first takes seconds and gives you a definite answer before the expensive part begins. The script reads your token from ~/hf_token.txt, so save it there first with echo "$HF_TOKEN" > ~/hf_token.txt. It starts with the arguments and the header that authenticates the API call:

#!/usr/bin/env python3
# check_k31_download.py - confirm a node has the COMPLETE Kimi K3.1 Pro
# checkpoint before we try to serve. Run ON the node.
import argparse, glob, json, os, urllib.request

ap = argparse.ArgumentParser()
ap.add_argument("--repo", default="moonshotai/Kimi-K3.1-Pro")
ap.add_argument("--hf-home", default="/ephemeral/hf")
ap.add_argument("--token-file", default=os.path.expanduser("~/hf_token.txt"))
a = ap.parse_args()

tok = ""
if os.path.exists(a.token_file):
    tok = open(a.token_file).read().strip()
h = {"User-Agent": "Mozilla/5.0"}
if tok:
    h["Authorization"] = "Bearer " + tok

Next it asks Hugging Face what should be there. The repository metadata lists every file, and keeping only the safetensors files gives the 96 names to look for:

# expected manifest from HF
req = urllib.request.Request(f"https://huggingface.co/api/models/{a.repo}", headers=h)
meta = json.loads(urllib.request.urlopen(req, timeout=60).read())

expected = [s["rfilename"] for s in (meta.get("siblings") or [])]
exp_st = sorted(f for f in expected if f.endswith(".safetensors"))

Then it finds the local copy. The downloader writes into a directory named after the commit and fills it with links into the blob store, so the size of each file is read through os.path.realpath:

snap = os.path.join(a.hf_home, "hub", "models--" + a.repo.replace("/", "--"), "snapshots")
dirs = glob.glob(os.path.join(snap, "*"))
if not dirs:
    print(f"MISSING: no snapshot dir under {snap}")
    raise SystemExit(2)
d = dirs[0]

local, total = {}, 0
for f in exp_st:
    p = os.path.join(d, f)
    if os.path.exists(p):
        sz = os.path.getsize(os.path.realpath(p))
        local[f] = sz
        total += sz

Finally it compares the two and exits non-zero if anything is missing, so it can sit in front of your launch command:

missing = [f for f in exp_st if f not in local]
incomplete = glob.glob(os.path.join(d, "*.incomplete"))
cfg_ok = os.path.exists(os.path.join(d, "config.json"))

print(f"repo             : {a.repo}")
print(f"snapshot dir     : {d}")
print(f"safetensors      : {len(local)}/{len(exp_st)} present")
print(f"total bytes      : {total:,} ({total/1e9:.2f} GB)")
print(f"config.json      : {'OK' if cfg_ok else 'MISSING'}")
print(f".incomplete files: {len(incomplete)}")

ok = (not missing) and (not incomplete) and cfg_ok
print("RESULT:", "COMPLETE" if ok else "INCOMPLETE")
raise SystemExit(0 if ok else 1)

Run it on every node. This is the result to look for, with the same commit hash on all four:

repo             : moonshotai/Kimi-K3.1-Pro
snapshot dir     : /ephemeral/hf/hub/models--moonshotai--Kimi-K3.1-Pro/snapshots/6d0a0202956796543364d666ab19c6ad1a494ee1
safetensors      : 96/96 present
total bytes      : 1,561,293,160,568 (1561.29 GB)
config.json      : OK
.incomplete files: 0
RESULT: COMPLETE

Step 8: Launch Kimi K3.1 Pro Across All Four Nodes

All four nodes run the same command, and only two values change between them: --node-rank and SGLANG_HOST_IP. Rank 0 is the head, and it is the only node that answers HTTP. Set the values first; paste this block into each node's terminal and change the first two lines:

# Run this on EVERY node. Only RANK and SELF_PRIV change between them.
export RANK=0                     # 0 on the head, then 1, 2 and 3
export SELF_PRIV="10.0.0.219"     # THIS node's private IP
export HEAD_PRIV="10.0.0.219"     # the head node's private IP, on all four
export NIC="ens6"                 # the private interface from Step 4
export HF_TOKEN="YOUR_HF_TOKEN_HERE"

# Clear any container left from an earlier launch
docker rm -f sgl 2>/dev/null || true

Next collect the engine arguments, which follow the SGLang cookbook's NVIDIA H100 layout for Kimi K3.1 Pro: tensor parallel 32, expert parallel 32 and a 0.85 static memory share across four nodes.

# Everything the engine itself needs, collected in one variable so the
# docker command below stays readable.
K31_ARGS="--trust-remote-code \
  --model-path moonshotai/Kimi-K3.1-Pro \
  --tp-size 32 \
  --ep-size 32 \
  --nnodes 4 \
  --node-rank $RANK \
  --dist-init-addr $HEAD_PRIV:20000 \
  --moe-runner-backend marlin \
  --mem-fraction-static 0.85 \
  --max-running-requests 64 \
  --dist-timeout 3600 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 8000"

Then launch. Everything here is container-level: the GPUs, host networking and shared memory, the weights mounted where the engine looks for them, and the environment it reads at start-up:

docker run -d --name sgl --gpus all --network host --ipc=host --shm-size 32g \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env NCCL_CUMEM_ENABLE=1 \
  --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --env SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \
  --env SGLANG_HOST_IP=$SELF_PRIV \
  --env NCCL_SOCKET_IFNAME=$NIC \
  --env GLOO_SOCKET_IFNAME=$NIC \
  --env NCCL_IB_DISABLE=1 \
  lmsysorg/sglang:dev-kimi-k31-pro \
  sglang serve $K31_ARGS

Start rank 0 first, then the other three. The rendezvous on port 20000 waits for all four, and --dist-timeout 3600 gives it an hour. This is what each setting does:

docker logs -f sgl

The process is complete when the head node's log shows:

The server is fired up and ready to roll!

Step 9: Test the Deployment

Run these from your local machine (not the VM), against the head node's public IP, once the port 8000 rule from Step 3 is in place. First list the served model:

# Use the HEAD node's PUBLIC IP here, not the private one
export HEAD_IP="your.head.node.public.ip"

curl -s http://$HEAD_IP:8000/v1/models

{
  "object": "list",
  "data": [
    {
      "id": "moonshotai/Kimi-K3.1-Pro",
      "object": "model",
      "owned_by": "sglang",
      "root": "moonshotai/Kimi-K3.1-Pro",
      "max_model_len": 1048576
    }
  ]
}

Then send a first chat request. reasoning_effort is set to "low" to keep the check quick:

curl -s http://$HEAD_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "moonshotai/Kimi-K3.1-Pro",
    "messages": [{"role": "user", "content": "Who are you? One sentence."}],
    "reasoning_effort": "low",
    "max_tokens": 256
  }'

"message": {
  "role": "assistant",
  "content": "Hello! I am Kimi, an AI assistant developed by Moonshot AI.",
  "reasoning_content": "The user asks who I am, in one sentence. I am Kimi,
                        developed by Moonshot AI. Answer briefly.",
  "tool_calls": null
}

A successful response returns the answer in content and the model's thinking in a separate reasoning_content field. The Kimi K3.1 Pro model is now deployed on Hyperstack.

Moonshot AI's published evaluation results use the following sampling parameters:

{"reasoning_effort": "max", "temperature": 1.0, "top_p": 0.95}

Controlling Thinking Effort

Now that the SGLang server is running, we can interact with it from Python. The helper below uses only the standard library, so there is nothing to install; any OpenAI-compatible client works the same way against /v1/chat/completions.

import json
import time
import urllib.request

BASE = "http://<HEAD_NODE_PUBLIC_IP>:8000/v1"
MODEL = "moonshotai/Kimi-K3.1-Pro"
KEY = "EMPTY"

def post(body, timeout=1800):
    req = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.load(r), time.time() - t0

A first request. The thinking and the answer come back in separate fields:

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain what a mixture-of-experts model is, simply, in 2 sentences."},
]

resp, dt = post({"model": MODEL, "messages": messages, "max_tokens": 4096,
                 "reasoning_effort": "high"})     # low, high or max

m = resp["choices"][0]["message"]
content = m.get("content") or ""
reasoning = m.get("reasoning_content") or ""

print("--- thinking ---")
print(reasoning)              # the chain of thought lands here
print("--- answer ---")
print(content)                # the final answer lands here

--- thinking ---
The user wants a simple, two-sentence explanation of a mixture-of-experts model.
Key idea: many expert sub-networks, a router picks a few per token, so only part
of the model runs each time. Keep it plain.
--- answer ---
A mixture-of-experts model is a large neural network split into many smaller
"expert" sub-networks, plus a router that sends each token to only a few of them.
Because only those few experts run for any given token, the model can hold far
more knowledge than it has to compute with at each step.

Kimi K3.1 Pro always thinks. Unlike Kimi K2.6, there is no instant mode to switch thinking off. Instead, the depth of thinking is set with the top-level reasoning_effort field, which accepts "low", "high" and "max" (default "max"). Moonshot AI's reasoning effort guide presents the three levels as a trade between reasoning depth, latency and token usage. Use "low" for direct questions and keep "max" for debugging, planning and maths.

prompt = ("Three friends split a bill. Ana pays twice what Ben pays, and Ben pays 3 more "
          "than Cara. The total is 43. How much does each pay?")

for effort in ("low", "max"):
    resp, dt = post({"model": MODEL, "max_tokens": 16384, "reasoning_effort": effort,
                     "messages": [{"role": "user", "content": prompt}]})
    m = resp["choices"][0]["message"]
    print(f"  [{effort:>3}] {m.get('content')}")

  [low] Cara pays 8.50, Ben pays 11.50 and Ana pays 23.00.
  [max] Cara pays 8.50, Ben pays 11.50 and Ana pays 23.00, which adds up to 43.

Both efforts reach the same answer. What changes is the thinking in front of it:

# the low-effort thinking:
Cara = c, Ben = c+3, Ana = 2c+6. Total 4c+9 = 43, c = 8.5. Ben 11.5, Ana 23.

# the opening of the max-effort thinking:
The user wants me to solve a word problem. Let me work through it.
Let Cara pay x. Ben pays 3 more than Cara: Ben = x + 3.
Ana pays twice what Ben pays: Ana = 2(x + 3) = 2x + 6.
Total: x + (x + 3) + (2x + 6) = 43, so 4x + 9 = 43 and x = 8.5 ...
Check: 8.5 + 11.5 + 23 = 43.

Every request pays for a reasoning pass, even at "low". Read usage in the response when budgeting token costs, as a client that reads only content still pays for the thinking tokens.

Preserved Thinking for Multi-Turn Coding Agents

For multi-turn conversations and tool calls, Kimi K3.1 Pro requires the complete assistant message returned by the API to be passed back into messages as-is, including reasoning_content and tool_calls, not just content. This is a requirement rather than an option: if the reasoning is dropped between turns, the model loses the thinking it built on the previous turn.

The example below, adapted from the Moonshot AI model card, shows the pattern. The prior assistant turn carries reasoning that mentions five numbers, but its visible answer lists only three:

# the example from the Kimi K3.1 Pro model card: the earlier reasoning
# named five numbers, the visible answer gave only three
msgs = [
    {"role": "user", "content": "Tell me three random numbers."},
    {"role": "assistant",
     "reasoning_content": "I will start by listing five numbers: 473, 921, 235, 215, 222, "
                          "and I will tell you the first three.",
     "content": "473, 921, 235"},
]

# pass the assistant message back AS-IS, including reasoning_content.
# For a turn you received from the server, rebuild it the same way:
def assistant_turn(m):
    turn = {"role": "assistant", "content": m.get("content") or ""}
    if m.get("reasoning_content"):
        turn["reasoning_content"] = m["reasoning_content"]
    if m.get("tool_calls"):
        turn["tool_calls"] = m["tool_calls"]
    return turn

msgs.append({"role": "user", "content": "What are the other two numbers you have in mind?"})

resp2, dt2 = post({"model": MODEL, "messages": msgs, "max_tokens": 16384,
                   "reasoning_effort": "max"})
print(resp2["choices"][0]["message"].get("content"))

The other two numbers were 215 and 222.

Because the prior reasoning trace mentioned 215 and 222, the model can answer with those exact numbers. If you rebuild the assistant turn from content alone, that reasoning is silently lost, so carry reasoning_content back explicitly, as assistant_turn() does.

Multimodal Capabilities with Kimi K3.1 Pro

Kimi K3.1 Pro is natively multimodal through its integrated MoonViT-V2 encoder, so it can analyse images alongside text in the same request. The image input format is identical to the OpenAI vision API. To make the check objective, the example builds its own test image, left half red and right half blue, with nothing but the standard library:

import base64
import struct
import zlib

def make_png(w=128, h=128):
    rows = b""
    for _ in range(h):
        row = b"\x00"                                    # filter byte
        for x in range(w):
            row += b"\xff\x00\x00" if x < w // 2 else b"\x00\x00\xff"
        rows += row

    def chunk(tag, data):
        c = struct.pack(">I", len(data)) + tag + data
        return c + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)

    ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)    # 8-bit truecolour
    return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
            + chunk(b"IDAT", zlib.compress(rows, 9)) + chunk(b"IEND", b""))

b64 = base64.b64encode(make_png()).decode()

resp, dt = post({"model": MODEL, "max_tokens": 2048, "reasoning_effort": "low",
                 "messages": [{"role": "user", "content": [
                     {"type": "text", "text": "What do you see in this image? "
                                              "Name the colours and how they are arranged."},
                     {"type": "image_url",
                      "image_url": {"url": f"data:image/png;base64,{b64}"}}]}]})

ans = (resp["choices"][0]["message"].get("content") or "").strip()
print(f"  ANSWER: {ans[:300]}")

  ANSWER: The image shows two solid colour blocks side by side: the left half is
          red and the right half is blue, meeting along a sharp vertical line down
          the middle.

The image travels as a base64 data URI in the "url" field, so the server needs no outbound access to fetch it.

Agentic Use Case with Kimi K3.1 Pro

One of the defining features of moonshotai/Kimi-K3.1-Pro is long-horizon agentic tool calling. Unlike standard chat where the model just generates text, an agentic workflow lets it:

• Decide when external tools are needed
• Call tools automatically
• Receive tool outputs and continue reasoning
• Complete multi-step tasks autonomously

Because we launched SGLang with --tool-call-parser kimi_k3, the server emits structured tool calls in the standard OpenAI format. Start with a round trip that is easy to check: an arithmetic tool and a small weather lookup.

import ast
import operator

ops = {
    ast.Add:  operator.add,
    ast.Sub:  operator.sub,
    ast.Mult: operator.mul,
    ast.Div:  operator.truediv,
    ast.Pow:  operator.pow,
    ast.USub: operator.neg,
}

def calculate(expression: str):
    def ev(n):
        if isinstance(n, ast.Constant):
            return n.value
        if isinstance(n, ast.BinOp):
            return ops[type(n.op)](ev(n.left), ev(n.right))
        if isinstance(n, ast.UnaryOp):
            return ops[type(n.op)](ev(n.operand))
        raise ValueError("bad expr")

    return {"result": ev(ast.parse(expression, mode="eval").body)}

def get_weather(city: str):
    forecasts = {"Tokyo": "22C, clear", "London": "14C, rain", "Cairo": "35C, sunny"}
    return {"city": city, "weather": forecasts.get(city, "18C, partly cloudy")}

IMPL = {"calculate": calculate, "get_weather": get_weather}

CALCULATE_TOOL = {
    "type": "function",
    "function": {
        "name": "calculate",
        "description": "Evaluate a basic arithmetic expression.",
        "parameters": {
            "type": "object",
            "properties": {
                "expression": {"type": "string", "description": "e.g. '47*89'"},
            },
            "required": ["expression"],
        },
    },
}

WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}

TOOLS = [CALCULATE_TOOL, WEATHER_TOOL]

The loop sends the task, runs every tool call the model returns, appends the results and calls the model again until it stops asking for tools. Each assistant turn goes back whole:

task = ("What is 47 * 89? Also, what is the weather in Tokyo? "
        "Use your tools, then give me one combined answer.")
messages = [{"role": "system", "content": "You are a helpful assistant with tools."},
            {"role": "user", "content": task}]

for turn in range(6):
    resp, dt = post({"model": MODEL, "messages": messages, "tools": TOOLS,
                     "max_tokens": 4096, "reasoning_effort": "high"})
    m = resp["choices"][0]["message"]
    tcs = m.get("tool_calls") or []
    print(f"[turn {turn}] tool_calls={len(tcs)}")

    if not tcs:
        print(m.get("content", ""))
        break

    # echo the assistant message back AS-IS (content + reasoning_content + tool_calls)
    messages.append(assistant_turn(m))

    for tc in tcs:
        fn = tc["function"]["name"]
        try:
            args = json.loads(tc["function"].get("arguments") or "{}")
            result = IMPL[fn](**args)
        except Exception as e:
            args, result = {}, {"error": str(e)}

        print(f"    -> {fn}({args}) = {result}")
        messages.append({"role": "tool", "tool_call_id": tc.get("id", fn),
                         "name": fn, "content": json.dumps(result)})

[turn 0] tool_calls=2
    -> calculate({'expression': '47*89'}) = {'result': 4183}
    -> get_weather({'city': 'Tokyo'}) = {'city': 'Tokyo', 'weather': '22C, clear'}
[turn 1] tool_calls=0

Here are both results:

- 47 x 89 = 4,183
- Tokyo weather: 22C and clear

So 47 times 89 is 4,183, and it is a clear day at 22C in Tokyo.

Both calls arrive in a single turn, and the second turn combines the two results into one answer.

A Coding Agent with a Shell Tool

For sustained engineering work, the same harness gets a single shell tool scoped to a workspace directory. Run it on a machine you are happy for the model to change, since the tool executes whatever command the model writes:

import os
import subprocess

WORKSPACE = os.path.expanduser("~/agent_workspace")
os.makedirs(WORKSPACE, exist_ok=True)

SHELL_TOOL = {
    "type": "function",
    "function": {
        "name": "run_shell",
        "description": "Run a shell command inside the workspace and return its output.",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string"}},
            "required": ["command"],
        },
    },
}

def run_shell(command):
    try:
        proc = subprocess.run(command, shell=True, cwd=WORKSPACE,
                              capture_output=True, text=True, timeout=120)
    except subprocess.TimeoutExpired:
        return "command timed out after 120 seconds"
    return (proc.stdout + proc.stderr)[-8000:]

def agent_loop(task, max_steps=30):
    messages = [{"role": "user", "content": task}]
    for _ in range(max_steps):
        resp, dt = post({"model": MODEL, "messages": messages, "tools": [SHELL_TOOL],
                         "max_tokens": 16384, "reasoning_effort": "high"})
        m = resp["choices"][0]["message"]
        messages.append(assistant_turn(m))      # the whole turn goes back each time
        if not m.get("tool_calls"):
            return m.get("content")
        for tc in m["tool_calls"]:
            command = json.loads(tc["function"]["arguments"])["command"]
            messages.append({"role": "tool", "tool_call_id": tc["id"],
                             "name": "run_shell", "content": run_shell(command)})
    return "step limit reached"

Example 1: Long-Horizon Refactor

This is the kind of sustained, multi-step engineering work Kimi K3.1 Pro is built for: read existing code, plan a change, apply it and check it without losing track. We seed the workspace with a small file and ask the model to refactor it:

# Seed the workspace with a deliberately rough file
with open(os.path.join(WORKSPACE, "calc.py"), "w") as f:
    f.write("def calc(a, b, op):\n"
            "    if op == 'add': return a + b\n"
            "    if op == 'sub': return a - b\n"
            "    if op == 'mul': return a * b\n"
            "    if op == 'div': return a / b\n")

print(agent_loop(
    "Refactor calc.py to use a dict-based dispatch table instead of if chains, "
    "add type hints, handle division by zero, and add three pytest tests in "
    "test_calc.py. Run the tests and fix any failures."
))

What happens internally over the tool calls:

1. The model interprets the request and decides it needs to read calc.py first.
2. It calls run_shell with cat calc.py and reads the existing structure.
3. It writes a refactored version of the file.
4. It writes test_calc.py with three test cases.
5. It runs pytest test_calc.py -q and reads the output.
6. If a test fails, it reads the failure, edits the file and re-runs. This is where preserved thinking matters most, as the model continues from its earlier reasoning instead of re-deriving it.
7. It returns a summary of changes.

Example 2: Coding-Driven Design (Build a Website)

Using the same agent harness, Kimi K3.1 Pro can turn a single prompt into a self-contained website:

print(agent_loop(
    "Build a single-page cat-themed website at index.html. Include a hero section, "
    "a description of three cat breeds in cards, a scroll-triggered fade-in "
    "animation, and inline CSS so the file works standalone. Then confirm the "
    "file exists."
))

The model writes a complete index.html in ~/agent_workspace. Open the file in a browser to review the page it generated.

Integration with Third-Party Coding Assistants

Because SGLang exposes the OpenAI-compatible /v1/chat/completions endpoint alongside an Anthropic-compatible /v1/messages route, and we enabled the Kimi tool-call parser, the self-hosted Kimi K3.1 Pro plugs into most modern coding assistants. Below are three of the most useful integrations.

Integrating with Kimi Code CLI

Moonshot AI's own agent framework is the Kimi Code CLI, available at kimi.com/code, and Moonshot AI uses it as the harness for several of the published Kimi K3.1 Pro benchmark results. Install it with the official script:

# Install Kimi Code CLI (macOS and Linux)
curl -fsSL https://code.kimi.com/kimi-code/install.sh | bash

By default Kimi Code talks to Moonshot AI's hosted API. To point it at your endpoint, add a provider and a model to ~/.kimi-code/config.toml. The providers reference documents the openai type for any OpenAI-compatible service, and it handles the reasoning_content field and effort settings itself:

# ~/.kimi-code/config.toml
default_model = "hyperstack/kimi-k3.1-pro"

[providers.hyperstack]
type = "openai"
base_url = "http://<HEAD_NODE_PUBLIC_IP>:8000/v1"
api_key = "EMPTY"

[models."hyperstack/kimi-k3.1-pro"]
provider = "hyperstack"
model = "moonshotai/Kimi-K3.1-Pro"
max_context_size = 131072
capabilities = ["thinking", "always_thinking", "image_in", "tool_use"]
support_efforts = ["low", "high", "max"]
default_effort = "high"

max_context_size is required, and it tells the CLI when to compact a long session; keep it at or below the KV cache size SGLang prints at start-up as max_total_num_tokens. The field names follow the configuration file reference. Then start the CLI in a project directory:

# Start in your project directory, then check the model with /status
kimi

Integration with Claude Code

Moonshot AI provides an Anthropic-compatible API for Kimi K3.1 Pro, and SGLang serves an Anthropic-compatible /v1/messages route on every server, so you can point Claude Code at your self-hosted Kimi K3.1 Pro server. Moonshot AI also evaluated many of the coding benchmarks above using the Claude Code harness.

Set the following environment variables before launching Claude Code:

# Point Claude Code at the head node's Anthropic-compatible route
export ANTHROPIC_BASE_URL="http://<HEAD_NODE_PUBLIC_IP>:8000"
export ANTHROPIC_AUTH_TOKEN="EMPTY"
export API_TIMEOUT_MS="3000000"
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1

# Use Kimi K3.1 Pro for every Claude Code model tier
export ANTHROPIC_DEFAULT_OPUS_MODEL="moonshotai/Kimi-K3.1-Pro"
export ANTHROPIC_DEFAULT_SONNET_MODEL="moonshotai/Kimi-K3.1-Pro"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="moonshotai/Kimi-K3.1-Pro"

claude

Environment Variables Overview

Efficiency Tip: Add these environment variables to your shell profile (for example ~/.bashrc or ~/.zshrc) or define them inside ~/.claude/settings.json under the "env" key for a persistent setup.

Once Claude Code launches, check the connection with a simple prompt. If the model responds correctly, your agentic coding environment is running on Kimi K3.1 Pro.

Deployment with OpenClaw

Kimi K3.1 Pro's long-horizon reasoning also makes it a strong engine for OpenClaw, the open-source, self-hosted agent platform. Pointing OpenClaw at your SGLang server gives you an autonomous coding environment in your terminal. Install it with the official installer:

# Install OpenClaw (macOS, Linux and WSL2)
curl -fsSL https://openclaw.ai/install.sh | bash

# Any non-empty value works while the server runs without a key
export SGLANG_API_KEY="EMPTY"

Configuring the Local Provider

OpenClaw ships an SGLang provider for OpenAI-compatible SGLang servers. To bridge it to your cluster, merge the following provider block into ~/.openclaw/openclaw.json:

{
  "models": {
    "mode": "merge",
    "providers": {
      "sglang": {
        "baseUrl": "http://<HEAD_NODE_PUBLIC_IP>:8000/v1",
        "apiKey": "${SGLANG_API_KEY}",
        "api": "openai-completions",
        "models": [
          {
            "id": "moonshotai/Kimi-K3.1-Pro",
            "name": "Kimi K3.1 Pro on Hyperstack",
            "reasoning": true,
            "input": ["text", "image"],
            "contextWindow": 131072,
            "maxTokens": 8192
          }
        ]
      }
    }
  }
}

Configuration Tip: The id field must match the model id the server reports at /v1/models (here, moonshotai/Kimi-K3.1-Pro). contextWindow tells OpenClaw how much history to send, so keep it within the KV cache size in the SGLang start-up log, as with Kimi Code.

Once configured, set the model and launch the OpenClaw TUI:

openclaw models set sglang/moonshotai/Kimi-K3.1-Pro
openclaw tui

From here you can issue high-level coding instructions and the Kimi K3.1 Pro-backed agent will plan, edit and check changes on your 32-GPU NVIDIA H100 cluster.

Running Kimi K3.1 Pro on One NVIDIA B300 Node

If you would rather serve Kimi K3.1 Pro from a single machine, the vLLM Kimi K3.1 Pro recipe gives the simplest working setup on NVIDIA Blackwell: one tensor-parallel group of eight GPUs, on one NVIDIA B300 node. Hyperstack offers it as the NVIDIA HGX B300 flavour n3-B300-SXM6x8, with 224 CPUs, 1,920 GB of RAM, a 1,000 GB root disk, 40,000 GB of ephemeral storage and 2.3 TB of HBM3e across its eight GPUs, so the whole checkpoint fits on one node and NVIDIA Blackwell runs the MXFP4 experts natively.

• The flavour reference requires the "Ubuntu Server 24.04 LTS R595 CUDA 13.2 with Docker" image for this flavour, which suits the CUDA 13 build of the vLLM image.
• The checkpoint goes on /ephemeral as on the NVIDIA H100 nodes, using Steps 5 to 7 on the one machine.
• The firewall needs only the SSH and port 8000 rules, since there are no peers to reach.

Then start the recipe's command for one NVIDIA B300 node, with the cache mounted from the ephemeral disk:

# One 8x NVIDIA B300 node: the vLLM recipe's command, cache on /ephemeral
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

The vLLM team reports that tensor parallel 8 on eight NVIDIA B300 GPUs holds about 2.6 million tokens of FP8 KV cache, and that decode context parallelism raises that to about 18.5 million when long sessions need more. One client-side difference from the SGLang endpoint: vLLM returns the thinking in a reasoning field rather than reasoning_content, and accepts either name on the way back in.

Kimi K3.1 Pro Deployment Cost

The four-node cluster bills as four 8x NVIDIA H100 NVLink virtual machines. At the GPU pricing rate of $2.60 per GPU per hour on demand, and from $1.82 per GPU per hour reserved, the cluster comes to:

Hyperstack bills per minute, with your balance updating as resources run, so an evaluation run costs only the minutes the nodes are active. The download and the start-up are part of that time, which is one more reason to start the four downloads in parallel.

Run Kimi K3.1 Pro on Hyperstack

Serve a 3T-Class Open Model on NVIDIA GPUs You Control

Four 8x NVIDIA H100 nodes, one SGLang command, and an endpoint that Claude Code, Kimi Code CLI and OpenClaw can drive.

32x NVIDIA H100SGLangOpenAI and Anthropic APIsPer-minute billing

Get started with Hyperstack in minutes.

FAQs

What is Kimi K3.1 Pro?

Kimi K3.1 Pro is Moonshot AI's open-weight, natively multimodal agentic model and its most capable 3T-class model to date. It uses a Mixture-of-Experts architecture with 2.8T total parameters and 104B activated, and is post-trained from Kimi K3 with large-scale joint reinforcement learning.

What is the difference between Kimi K3 and Kimi K3.1 Pro?

Kimi K3.1 Pro keeps Kimi K3's parameter count, layer count, 896-expert MoE and MXFP4 expert format. The main change is attention: the 24 full-attention Gated MLA layers become Gated DeepSeek Sparse Attention (DSA). Moonshot AI also reports gains in coding, 3D creation, financial research and factual accuracy.

What is Kimi K3.1 Pro's context window?

Kimi K3.1 Pro supports a native 1,048,576-token context window. How much a deployment can use at once depends on the KV cache left after the weights, which SGLang prints at start-up as max_total_num_tokens.

Can I turn off thinking in Kimi K3.1 Pro?

No. Kimi K3.1 Pro always reasons before answering. You control how much it reasons with the reasoning_effort field, which accepts "low", "high" and "max" (default "max"). For multi-turn conversations and tool calls, pass the full assistant message back, including reasoning_content.

What hardware is needed for Kimi K3.1 Pro?

Every one of the 896 experts has to stay resident in GPU memory, so total GPU memory is the constraint. This tutorial serves the 1.56 TB checkpoint on 32 NVIDIA H100 GPUs across four Hyperstack nodes with SGLang; one 8x NVIDIA B300 node with vLLM also holds it.

Which inference engines support Kimi K3.1 Pro?

Moonshot AI recommends vLLM, SGLang and TokenSpeed. vLLM and SGLang both use the kimi_k3 reasoning and tool-call parsers for Kimi K3.1 Pro.