Run DeepSeek OCR on Hyperstack with your Own UI

Jev is not a language model in the usual sense. TypeSafe AI released the Jev AI model on 15 September 2026 as the first of what it calls a System One model: an engine that takes a state and one or more questions and returns typed answers with probabilities, never a sentence. The company reports 70 to 500 millisecond end-to-end latency, together with headline claims of up to 200 times faster and 400 times cheaper than comparable large language model workflows on the same class of decision.

This piece looks past the announcement. It explains the mechanism that lets Jev skip the sentence entirely, reproduces that mechanism against an open model running on a single Hyperstack NVIDIA GPU, and checks TypeSafe’s own numbers against an independent benchmark that LangChain published within days of launch.

This piece draws on four sources throughout: TypeSafe’s own announcement and documentation, the SGLang documentation behind the scoring mechanism reproduced further down, and LangChain’s own evaluation of Jev against three large language model judges, published as Jev-as-a-Judge for Agent Evals. Figures that come from TypeSafe’s own comparisons are marked as such wherever they appear, separate from the independent measurements.

The Decision Your Code Keeps Paying Full Price For

Most of what software asks a language model to do is not writing. It is a judgement with a small number of possible answers: is this support ticket urgent, which team should own it, is a shell command safe to run, does a retrieved passage answer the question that was asked. None of that needs a paragraph. It needs one of a handful of known labels.

The usual approach reaches for the same tool regardless of the question: a full generative call, a sentence or a JSON object to wait for, and code that parses that answer back into the label it wanted in the first place. Inside an agent loop, that pattern repeats on almost every iteration.

while not task_done:
    action = llm(context)          # full generation, just to pick a tool
    result = run_tool(action)
    context += result
    task_done = llm(context)       # full generation again, just to check "done or not"

Every pass through that loop calls a generative model twice for a judgement with only a handful of possible answers: which tool to call next, and whether the task is finished. Both are single-token decisions, priced and timed as though they were prose.

The gap between the two ways of answering comes down to how many times the model has to run. Writing an answer means running the model once per output token, because each token depends on the one written immediately before it. A bounded decision needs exactly one pass, because the set of possible answers is already known before the model is ever called.

Jev’s premise, as TypeSafe AI frames it, is that language generation is the wrong interface once code already knows the set of possible answers. If every outcome can be enumerated ahead of time, the system needs a model that scores those outcomes, not one that writes about them.

What Jev Is

TypeSafe’s own documentation defines System One models as a class of AI models built to make fast, structured decisions that software can use directly. A System One model evaluates a state and returns typed answers and probabilities, and it does not write replies, produce code, or explain its own reasoning.

A request to Jev carries two things: a state, which is the text or JSON describing the situation, and one or more questions about that state. Every question declares its answer shape in advance, using one of three primitives.

Choice selects one option from a list defined by the caller and returns a probability for every option, not only the winner. Score places the input on an ordered scale defined by the caller and returns a probability-weighted position on that scale rather than a single label. Noul, TypeSafe’s own name for the yes-or-no primitive, returns the probability that a stated question is true. Every answer comes back as a number between 0 and 1 that application code can act on directly, with no option outside the ones the caller declared.

A Score answer such as 1.74 on a zero-to-two scale is not a rounding artefact. It is the probability-weighted average across every rung on the scale, so a result close to 2.0 signals near certainty on the top label, while 1.74 signals a strong lean towards the top label with real weight left on the one below it, information a single winning label would discard.

A single request can carry several questions about the same state, and every one of them is answered from the same evaluation, not a separate call for each.

{
  "model": "jev-latest",
  "state": "I was charged twice for the same subscription.",
  "questions": {
    "urgent": {
      "type": "noul",
      "instructions": "Does this need attention right now?"
    },
    "team": {
      "type": "choice",
      "instructions": "Which team should own this ticket?",
      "criteria": {
        "billing": "Charges, invoices, refunds",
        "technical": "Product bugs and errors",
        "account": "Login and access problems"
      }
    }
  }
}

{
  "urgent": {"probability": 0.61},
  "team": {
    "choice": "billing",
    "probabilities": {"billing": 0.91, "technical": 0.06, "account": 0.03}
  }
}

Application code reads that response directly. There is no prose to parse and no answer outside the declared options, because team can only ever be one of the three names supplied in the request.

How This Differs From Structured Output

It is easy to confuse Jev’s mechanism with structured output, because both restrict what an application receives. The two do different work inside the serving engine.

A structured-output call constrains a model’s generation to a schema, so a request for a ticket’s team might return {"team": "billing"}. The schema prevents an invalid object, but it does not guarantee a correct one. The model still writes that object one token at a time: the opening brace, the field name, the value and the closing brace, decoded exactly as a sentence would be. A schema narrows the shape of the output. It does not remove the decode loop underneath it.

Jev’s fixed-answer scoring exits before that loop starts. Given the same three allowed teams, it reads one score per team from a single forward pass and normalises them into a distribution, with no token of output ever generated.

Where That Probability Comes From

Seeing why Jev can skip the decode loop starts with what a normal generation step does. A tokenizer converts a prompt into token IDs. The model processes that sequence and produces one vector for the next position, holding a raw score, a logit, for every token in the vocabulary, tens of thousands of numbers for a typical open model.

Turning a logit into a probability means running softmax across every other logit in that whole vocabulary.

Ordinary generation does not even run that plain version. It divides every logit by a temperature first, sharpening the distribution below 1 or flattening it above 1, before a token is sampled.

That whole cycle, a forward pass, decoding rules, one appended token, repeats until a stop token or a length limit ends it.

Reading Three Logits Instead of Tens of Thousands

A bounded decision only needs the very first vector, and only a handful of its entries. Labelling three answers A, B and C inside the prompt, and ending that prompt exactly where the model would normally start writing the label, puts the logits for all three at a single, known position. The model still produces its usual vocabulary-sized vector for that position. It holds the logit for A, the logits for B and C, and every other token in the vocabulary besides.

Every logit outside the three declared labels is discarded, and softmax runs only across the three that remain.

Why the Labels Are Single Letters, Not Whole Words

A visible word is not necessarily one token. billing might be one token for one tokenizer and several for another, and technical support spans more than one position under any tokenizer. Comparing multi-token phrases would need sequence scoring: score the first token, append it, score the next, and combine the values for the whole phrase, with length becoming part of the comparison. A single-letter label avoids that problem entirely, because every option sits at exactly one output position while the meaning behind it still lives in the prompt next to the letter.

A second trap sits underneath the first. Tokenizers often fold a leading space into a token, so A and  A can resolve to two different token IDs, and a chat template can insert that whitespace without it being obvious from the prompt text alone. The way around this is to render the full prompt through the model’s own chat template, check the exact continuation it expects at the answer position, and confirm through a tokenizer call that each label resolves to exactly one token before trusting a scoring request against it.

Reproducing the Mechanism on a Hyperstack NVIDIA GPU

Jev itself is closed. Its weights, its Reinforcement Learning for Calibrated Decisions training run and its evaluation stack are not published. The restricted-softmax mechanism behind it, however, already sits inside open serving engines, which means the trick can be built and measured against a model under full control rather than taken on trust. The steps below reproduce that mechanism using SGLang, run here on a single Hyperstack NVIDIA H100 virtual machine rather than a laptop.

SGLang is the same serving engine behind the Kimi K3, the Qwen3.8 Max and the MiniMax H3 tutorials, so a virtual machine already configured for one of those deployments needs no extra setup for this exercise. A single NVIDIA H100 is more GPU than a 0.5 billion parameter model needs, which is the point: the model here is small on purpose, chosen so the mechanism, not the hardware, is what gets measured.

Starting the Model Server

A clean environment needs two packages: SGLang itself, and requests for the client that follows.

python3 -m venv .venv
source .venv/bin/activate
pip install "sglang[all]==0.5.10.post1" "requests==2.34.2"

The server is the only process doing GPU work. Everything after this is a client script talking to it over HTTP.

python -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-0.5B-Instruct \
    --host 127.0.0.1 \
    --port 30000

The first launch downloads Qwen2.5-0.5B-Instruct from Hugging Face. Later launches reuse the local cache, and the model stays resident in GPU memory on port 30000 for every request that follows.

Defining the Choices and Building the Prompt

The client, decide.py, starts with the labels and the ticket being routed. The dictionary keeps both representations of every answer together, the letter that gets scored and the meaning returned to the application, so their order cannot drift apart between tokenizing, scoring and mapping the result back.

# decide.py
import json
import requests

BASE_URL = "http://127.0.0.1:30000"
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"

# A is the token that actually gets scored. "billing and payments" is what
# the application does with the answer once it comes back.
choices = {
    "A": "billing and payments",
    "B": "technical support",
    "C": "account access",
}
ticket = "I was charged twice for the same subscription."

# decide.py (continued)
choice_lines = "\n".join(
    f"{label} = {meaning}" for label, meaning in choices.items()
)
prompt = f"""Ticket:
{ticket}
Question:
Which category matches the ticket?
Allowed labels:
{choice_lines}
Return only the label.
Label:
"""
print(prompt)

Ticket:
I was charged twice for the same subscription.
Question:
Which category matches the ticket?
Allowed labels:
A = billing and payments
B = technical support
C = account access
Return only the label.
Label:

The prompt ends at Label:, the exact position whose vocabulary vector gets read. SGLang is never asked to generate the token that would normally follow it.

Resolving the Label Token IDs

Before anything can be scored, the real token ID behind each of A, B and C has to be confirmed against this specific tokenizer, not assumed from any other model.

# decide.py (continued)
label_token_ids = []

for label in choices:
    response = requests.post(
        f"{BASE_URL}/tokenize",
        json={"model": MODEL, "prompt": label, "add_special_tokens": False},
        timeout=30,
    )
    response.raise_for_status()
    token_ids = response.json()["tokens"]
    if len(token_ids) != 1:
        raise ValueError(f"{label!r} is not a single token: {token_ids}")
    print(f"{label!r} -> {token_ids}")
    label_token_ids.append(token_ids[0])

'A' -> [32]
'B' -> [33]
'C' -> [34]

Each label returns a single integer, so every option occupies exactly one vocabulary position. Had any label produced two or more IDs, the raise ValueError above would have stopped the script there rather than let a silent mismatch reach the scoring call.

Scoring the Three Positions

With the token IDs confirmed, the decision itself is a single HTTP call to SGLang's /v1/score endpoint, documented as computing token probabilities for specified tokens given a query, intended for classification tasks, scoring responses and log-probability computation.

# decide.py (continued)
response = requests.post(
    f"{BASE_URL}/v1/score",
    json={
        "model": MODEL,
        "query": prompt,
        "items": [""],
        "label_token_ids": label_token_ids,
        "apply_softmax": True,
    },
    timeout=120,
)
response.raise_for_status()
score_response = response.json()
print(json.dumps(score_response, indent=2))
scores = score_response["scores"][0]

{
  "scores": [
    [
      0.6777623295783997,
      0.31087860465049744,
      0.011359035037457943
    ]
  ]
}

query carries the full prompt. The empty string inside items tells SGLang to score the position immediately after it. label_token_ids names the three positions to read, and apply_softmax normalises those three readings into a proper probability distribution. SGLang tokenized the prompt, ran the model once, and returned three numbers in the same order the token IDs were sent, with no token generated to produce them.

Mapping the Scores Back to a Decision

The last step is ordinary code, turning three floats back into the labels the application understands.

# decide.py (continued)
probabilities = {
    choices[label]: float(score)
    for label, score in zip(choices, scores, strict=True)
}
decision = max(probabilities, key=probabilities.get)
print(json.dumps({"decision": decision, "probabilities": probabilities}, indent=2))

{
  "decision": "billing and payments",
  "probabilities": {
    "billing and payments": 0.6777623295783997,
    "technical support": 0.31087860465049744,
    "account access": 0.011359035037457943
  }
}

Billing wins, but only at 0.678, well short of the 0.9-plus a reader might expect. A threshold of 0.70 would catch exactly this case and send it for a second look rather than route it without review, which is the kind of decision that belongs in application code rather than inside the model.

Measuring the Difference Against Standard Generation

The mechanism explains why scoring should be faster. Measuring it is more convincing than assuming it. The scoring path above was run against a standard chat-completion call on the same SGLang server, the same model, and the same 100 support-ticket cases, both starting at the same instant. Standard generation decoded up to 32 tokens per answer, an explanation alongside the label.

That gap is also a cost figure. Every millisecond a request holds a GPU is a millisecond another request cannot use it, so the same measurement that explains Jev’s latency claim also explains why a workload built on this pattern needs less GPU time per decision than one built on ordinary chat completions, whether the model behind it is Jev or an open one served from a Hyperstack GPU billed by the minute.

Why the Number Next to the Answer Matters

A typed answer solves half the problem. The label tells application code what won. The probability tells it how close the race was, and that second number is what the code should branch on.

{
  "choice": "billing",
  "probabilities": {"billing": 0.52, "technical": 0.46, "account": 0.02},
  "confidence": 0.06
}

Routing that ticket without a second look would be reckless. Billing won, but barely, and a low-confidence win should never take the same branch as a decisive one. One workable definition of confidence is the margin between the top answer and its closest rival, rather than the winning probability taken alone.

if probabilities["billing"] > 0.9:
    route_automatically(ticket, team="billing")
elif confidence < 0.3:
    send_to_human_review(ticket)
else:
    escalate_to_larger_model(ticket)

TypeSafe trains Jev with what it calls Reinforcement Learning for Calibrated Decisions, RLCD for short. The target is calibration, not only accuracy: a batch of answers each carrying 90 per cent confidence should turn out correct on roughly 90 per cent of those cases. Measuring that means binning every prediction by its stated confidence and comparing each bin’s average confidence against how often that bin was right.

The Hallucination Claim, Examined Closely

TypeSafe says Jev cannot hallucinate. That statement holds under a narrow definition, and it is worth separating from a broader one that it does not support.

Declaring billing, technical and account as the only valid answers means the response can never come back as legal, an option that does not exist in the schema. It also cannot return malformed prose where code expected a label. That part of the hallucination problem does not apply to it.

It can still confidently choose the wrong valid option. A login bug scored as billing instead of technical is schema-valid and still wrong, and now the wrong customer gets refunded over an issue that was never about a charge. A schema rules out an answer that does not exist. It says nothing about whether the answer that does exist is the right one.

Where Jev Earns Its Place Inside an Agent

As a System One model, Jev works best alongside a generative model, not instead of one. The generative model plans, writes, explains and calls tools, the work that needs language. Jev handles the frequent small decisions that surround that work.

Model Routing

A simple lookup does not need the same model as a demanding architecture review. Jev can score the incoming request and select the least expensive model likely to complete it. The router decides which model should answer, not the answer itself.

Tool Risk Gating

Before an agent runs a shell command, Jev can classify it as read-only, reversible or destructive, alongside separate questions about whether it touches production or leaves an approved directory. A high-confidence read-only command can continue unattended. A destructive or uncertain one pauses for a person.

{
  "model": "jev-latest",
  "state": "rm -rf /data/tmp",
  "questions": {
    "risk": {
      "type": "choice",
      "instructions": "Classify this shell command.",
      "criteria": {
        "read_only": "Inspects state, changes nothing",
        "reversible": "Changes state, but the change can be undone",
        "destructive": "Deletes data or is otherwise hard to undo"
      }
    }
  }
}

Verification

An agent can report that a task is finished while its tests are still failing. Jev can inspect the state and answer bounded questions: did the tests pass, is the agent repeating the same failed action, does the output follow policy. A hard test still catches what it was built to catch. Jev adds a semantic check for what that hard rule was never designed to see.

Problems Jev Can Solve Today

Jev sits within a broader set of AI models for automation, but it fits a specific niche inside that set. The workloads where it earns its keep share three properties: every possible answer can be named ahead of time, a careful person could judge the input quickly by hand, and the decision happens often enough that latency or cost start to matter.

Does This Hold Up? An Independent Benchmark

TypeSafe’s own numbers for the Jev AI model deserve a careful reading. A company benchmarking its own product tends to choose the comparison that flatters it, and TypeSafe AI reports 70 to 500 millisecond latency, $0.042 per million input tokens with output free, and headline claims of up to 200 times faster and 400 times cheaper than comparable large language model workflows. Those figures are a ceiling worth treating with care, not a promise for every workload, even where the underlying mechanism behind them is real and measured earlier in this piece.

A comparison run by a party with nothing to gain from Jev looking good is worth more. LangChain ran exactly that kind of test, comparing Jev against three large language model judges, GPT-5.6 Luna, GPT-5.6 Terra and Claude Sonnet 4.6, using its own Deep Agents harness and LangSmith for the dataset. A weather agent answered five fixed requests, each response was captured once, and every judge scored the same frozen traces across 100 repetitions, so differences in the results measure the judge rather than the agent.

Across 500 repeated pass or fail judgements against a human oracle, Jev matched the human on every single one. GPT-5.6 Terra matched on 99.8 per cent, GPT-5.6 Luna on 96.4 per cent, and Claude Sonnet 4.6 on 80.0 per cent.

Accuracy alone tells half the story, since a judge can be right on average while swinging between different answers on the exact same input from one run to the next. Measuring variance across 100 repetitions on a single frozen case is where that gap opens further.

Cost and latency followed the same pattern. Jev averaged $0.00035 and 0.44 seconds per call. GPT-5.6 Luna averaged $0.00039 and 2.50 seconds, GPT-5.6 Terra $0.00289 and 2.83 seconds, and Claude Sonnet 4.6 $0.02811 and 2.16 seconds. The full project cost $0.34 for Jev against $28.17 for Claude Sonnet 4.6.

LangChain combined accuracy and consistency into a single signal value: oracle agreement multiplied by repeatability, the chance that two independent calls on the same frozen trace reach the same verdict.

One agent, five test cases and a narrow evaluation slice is not the final word on every workload. It is, however, an independent measurement rather than a vendor’s own comparison, and it points in the same direction the mechanism itself predicts.

Where Jev Falls Apart

Jev becomes less useful the moment the answer space stops being known ahead of time.

• It cannot write a response, summarise a document, generate code, or explain its own reasoning.
• It is unreliable at arithmetic, counting, date comparison or exact string manipulation. Those belong in ordinary code.
• It struggles when a decision needs several hidden reasoning steps chained together. Splitting the judgement into smaller questions, or using a reasoning model, works better.
• It cannot extract an unknown value directly out of text. Finding candidate values first with something else, then letting Jev choose among them, is the workable pattern.
• Irrelevant context can reduce its accuracy, so only the state a decision needs should be sent.
• Closed weights, an early access window, text-only input and still-thin independent calibration data outside the one benchmark cited above mean it is early to place full trust in a decision with real consequences.

A simpler rule sits underneath all of this. If deterministic code already solves the problem correctly, the code should stay. A plain if statement is faster, cheaper and easier to test than any model.

A Sane Way to Roll This Out

A cheap model can still turn out expensive if its mistakes generate retries, manual review or an incident, so a rollout has to measure the whole workflow, not the token price on its own.

1. Choose one bounded, low-risk decision. A small, clearly named set of possible answers, not the riskiest step in the pipeline.
2. Write the rubric before calling the model. Decide in words what belongs in every option. The rubric is part of the system, not an afterthought added later.
3. Collect representative examples first. Include the ambiguous and adversarial cases on purpose.
4. Run it in shadow mode. Let it answer every real case without changing behaviour, and log its probabilities beside the current outcome.
5. Plot accuracy against confidence and set thresholds from that data. The cut-off should come from the shadow-mode numbers, never a guess.
6. Automate the safest branch first. Keep a person, or a stronger model, as the fallback for anything the thresholds flag as uncertain.
7. Version everything. Pin the model version, the questions, the criteria and the thresholds together, so a change can be replayed against the same evaluation set.

Raising the bar too far means the system never makes a mistake because it never decides anything, which is not automation. It is expensive logging. The coverage constraint above keeps that from happening.

Build the scoring pattern on your own NVIDIA GPU

Try a Jev-Style Decision Engine on Hyperstack

One NVIDIA H100, SGLang, and an open model are enough to measure the pattern behind Jev for yourself, on the same platform already serving Kimi K3, Qwen3.8 Max and MiniMax H3.

One NVIDIA H100 GPUSGLang /v1/score463 ms measuredBilled by the minute

Launch an NVIDIA GPU virtual machine on Hyperstack today.

FAQs

What is Jev?

The Jev AI model is TypeSafe AI’s first System One model, released 15 September 2026. As a probabilistic AI model, it takes a state and one or more typed questions and returns typed answers with probabilities in a single forward pass, rather than generating text, and cannot write prose, code or explanations.

Can Jev’s scoring mechanism be reproduced without TypeSafe’s API?

Yes. SGLang’s /v1/score endpoint exposes the same restricted-softmax mechanism used by Jev against any open model, so a Choice or Noul-style answer can be computed from one forward pass on a Hyperstack NVIDIA GPU, with no access to Jev’s own weights required.

How is Jev different from structured output?

Structured output constrains a class of AI models for structured outputs to a schema, but generation still happens token by token underneath it. Jev’s fixed-answer scoring reads three or more logits from a single forward pass instead, and never enters the decode loop, so no token is ever generated.

How much GPU do you need to reproduce this pattern?

A single NVIDIA GPU is enough for a small model such as Qwen2.5-0.5B-Instruct, the model used in the walkthrough above. Production workloads with larger models still need only one forward pass per decision, so the sizing question is model size, not decision volume.

Does Jev replace LLM-as-judge evaluation?

Not entirely. Jev belongs to a small set of LLM alternatives for automation built specifically for bounded decisions, and LangChain’s benchmark found it more accurate and consistent than three LLM judges on a narrow agent-evaluation task, at a fraction of the cost. Open-ended judgements that need written reasoning still belong to a generative model, not a typed decision engine.

Qwen3.8 Max is a 2.4 trillion parameter mixture-of-experts model from the Qwen team, and 95 billion of those parameters are awake for any one token. Alibaba put the hosted version in front of developers on 3 August 2026 and published the weights shortly afterwards, which is the first time a Qwen Max class model has been released for anyone to run on their own hardware. The FP8 checkpoint is 2.50 TB across 213 safetensors shards, so a Qwen3.8 Max deployment is a multi-node exercise before it is anything else.

This guide deploys it on Hyperstack across eight nodes of eight NVIDIA H100 80GB PCIe GPUs: eight virtual machines, 64 GPUs in total, running SGLang with tensor parallel 16 by pipeline parallel 4. Every figure below comes from one session on 12 August 2026 that pulled 2.50 TB of weights, reached a live OpenAI-compatible endpoint in under six minutes of engine start-up, and then answered chat, reasoning, tool-calling and concurrency workloads with zero failed requests across the sweep.

There is one thing worth settling before the hardware. The hosted Qwen3.8 Max API does not publish an activated parameter count, which leaves anyone modelling cost to guess at the compute behind a token. The open checkpoint states it in config.json: 95 billion active, ten routed experts of 512 plus one shared expert, 92 layers. That is what makes a Qwen3.8 Max deployment something you can size and price rather than estimate. If you have followed our Kimi K3 multi-node deployment, this is the same class of exercise with a different bottleneck, and the reasons are arithmetic rather than memory.

Inside Qwen3.8 Max: 2.4 Trillion Parameters, 95 Billion Awake

Qwen3.8 Max is a sparse model with a hybrid attention stack. Both halves of that sentence change how you deploy it. Take them one at a time before booking any hardware.

Sparsity Is What Makes 2.4 Trillion Parameters Servable

Every parameter has to be resident in GPU memory, because the router picks its experts per token and per layer, and there is no useful way to predict which ones. What sparsity buys is compute rather than capacity. The forward pass touches 95 billion parameters, so the cluster is sized by the weight of the model and the speed is set by a far smaller number.

The Hybrid Attention Stack Sets the Concurrency Limit

The 92 layers are not uniform. Three out of every four use Gated DeltaNet, a linear attention layer that carries a fixed-size recurrent state per sequence. The fourth is gated full attention with a conventional key-value cache. The configuration expresses this as full_attention_interval: 4, which gives 69 linear layers and 23 full attention layers.

The practical consequence arrives at serving time. A conventional transformer runs out of concurrency when the key-value cache fills, so the limit moves with context length. Here the recurrent state is a fixed cost per sequence regardless of how long that sequence becomes, and it is the larger of the two pools. When the engine reported its capacity, the ceiling on concurrent requests came from the recurrent state pool rather than from the key-value cache, and that shapes the whole throughput picture further down.

What the Published Scores Say

Qwen positions this release around long-horizon agentic work, coding and research rather than raw single-turn question answering, and the published headline scores follow that emphasis.

The Qwen3.8 Max Deployment Architecture: Why 64 NVIDIA H100 GPUs

Sizing a cluster for a 2.50 TB checkpoint looks like a division problem and is not one. Memory rules out the small clusters, and then a second constraint rules out two more that memory was perfectly happy with. Getting to the right answer takes both filters, in that order.

The First Filter Is Memory, and It Is the Forgiving One

Weights are only part of what a GPU has to hold. The recurrent state pool, the key-value cache and the captured CUDA graphs all come out of the same 80 GB, and whatever is left is what serves the running batch. We sized this deployment against a floor of roughly 25 GB free per GPU after weights, which turns the question from a theoretical lower bound into a usable one.

Four nodes fail this outright. The weights alone would need 78.1 GB of an 80 GB card, which leaves nothing for anything else. Five nodes clear the weights and then fall under the free-memory floor. Six and seven nodes pass comfortably.

The Second Filter Is Divisibility, and It Is the Strict One

A parallel placement is only legal if the numbers divide. The pipeline degree has to divide the layer count evenly, and the tensor degree has to divide the attention head count and the expert count. Applying that to six and seven nodes removes both, even though both fit the weights with room to spare.

The arithmetic behind that is short enough to check by hand. Qwen3.8 Max has 92 layers, and 92 factorises as 2 squared times 23, so a pipeline degree has to be one of 1, 2, 4, 23, 46 or 92. It also has to divide the GPU count. On 48 GPUs that leaves pipeline degrees of 1, 2 and 4, giving tensor degrees of 48, 24 and 12. None of those divides 64 query heads. On 56 GPUs the tensor degrees are 56, 28 and 14, and again none divides 64. Six and seven node clusters are not tight, they are impossible.

Why Pipeline Parallel 4 and Tensor Parallel 16

Three placements are legal on 64 GPUs: tensor 64 by pipeline 1, tensor 32 by pipeline 2, and tensor 16 by pipeline 4. The choice between them is about where the traffic goes.

A tensor-parallel group communicates on every layer, so its members want to be as close together as possible. At tensor 64 the group spans all eight machines, and every collective in every layer crosses the network across 64 ranks. At tensor 16 the group spans two machines, and the four pipeline stages exchange activations only at the three stage boundaries. Twenty-three layers per stage divides evenly. The SGLang cookbook lists a verified four-node NVIDIA H200 placement for this same FP8 checkpoint, which is the same weights on cards large enough to need half as many GPUs. Our guide to tensor and pipeline parallelism covers the general trade-off between the two axes if you want the background.

What the Fabric Decides About Your Launch Flags

The launch command carries four settings that exist because of how the GPUs are wired, not because of anything about Qwen3.8 Max. Reading the topology first makes all four obvious.

      GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0
GPU0   X    NV12  PHB   PHB   PHB   PHB   PHB   PHB   PHB
GPU1  NV12   X    PHB   PHB   PHB   PHB   PHB   PHB   PHB
GPU2  PHB   PHB    X    NV12  PHB   PHB   PHB   PHB   PHB
GPU3  PHB   PHB   NV12   X    PHB   PHB   PHB   PHB   PHB
...
NIC0  PHB   PHB   PHB   PHB   PHB   PHB   PHB   PHB    X

  NV#  = Connection traversing a bonded set of # NVLinks
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge

A tensor-parallel group of 16 covers eight cards on one machine and eight on another, so the group is a multi-node group by definition. Two single-node optimisations then switch themselves off, and the engine says so during startup.

[18:15:14 PP0 TP0] Init torch distributed begin.
[18:15:40 PP0 TP0] CustomAllreduce is disabled because this process group
                   spans across nodes.
[18:15:40 PP0 TP0] Init torch distributed ends. elapsed=26.60 s, mem usage=0.47 GB
[18:15:46 PP0 TP0] multimem all-gather disabled because the TP group spans
                   across nodes.

Neither line calls for action. Both are the engine selecting the portable path because the fast path is only correct within one machine. What does call for action is telling the collectives which interface to use, because a multi-homed node will otherwise pick the wrong one and the rendezvous will not complete.

How to Deploy Qwen3.8 Max on Hyperstack, Step by Step

Nine steps take a Qwen3.8 Max deployment from an empty account to an endpoint answering requests. Everything up to Step 5 is ordinary virtual machine work repeated eight times. The launch itself is Steps 6 and 7, and Step 7 is where the flags specific to this model appear.

Step 1: Create Eight Virtual Machines

From the Hyperstack dashboard, open the virtual machines page and start a new deployment. The getting started guide covers the account setup if this is a first deployment.

Start from the virtual machines page and select deploy a new virtual machine.

Four choices matter, and one of them matters more than the rest.

The image picker. Choosing the CUDA and Docker variant removes every driver step from this deployment.

Step 2: Open the Three Ports That Matter

A multi-node deployment needs the nodes to talk to each other freely and needs you to talk to exactly one of them. Attach these firewall rules to every machine.

Step 3: Point the Model Cache at the Ephemeral Disk

Once the machines are active, connect to each one and take stock. The root disk is 96 GB, which is not where 2.50 TB of weights is going. This flavour ships an ephemeral NVMe disk mounted at /ephemeral, and that is the destination.

# From your workstation, once the eight virtual machines are ACTIVE
ssh -i ~/.ssh/[YOUR KEY] ubuntu@[NODE PUBLIC IP]

# Confirm what you are standing on before anything else
nvidia-smi --query-gpu=index,name,memory.total --format=csv
ip -o -4 addr show | awk '{print $2, $4}' | grep ' 10\.'
df -h / /ephemeral

index, name, memory.total [MiB]
0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
...
7, NVIDIA H100 PCIe, 81559 MiB

ens6 10.0.0.210/24

Every card reports 81,559 MiB and the private interface is ens6, holding a 10.x address. The third command is the one that decides where the weights go: the root disk is 96 GB, so /ephemeral has to take the 2.50 TB checkpoint. Confirm it has the room before starting the download.

# The root disk is 96 GB and the checkpoint is 2.50 TB, so the cache
# has to live on the ephemeral NVMe disk. Do this on all eight nodes.
sudo mkdir -p /ephemeral/hf
sudo chown -R ubuntu:ubuntu /ephemeral/hf

python3 -m pip install -q --break-system-packages huggingface_hub hf_transfer

Step 4: Pull the Serving Image and the Weights

The container image and the checkpoint can be fetched at the same time, and on eight nodes both should be started in the background so a dropped SSH session does not take the download with it. The image comes from the SGLang project and carries the kernels this model needs.

# Pull the serving image in the background while the weights download.
# Images stay on the root disk; only the checkpoint goes to /ephemeral.
nohup docker pull lmsysorg/sglang:qwen38 > /tmp/pull.log 2>&1 &

The weights need a Hugging Face token with access to the Qwen3.8-2.4T-A95B-FP8 repository. Create one from your account settings and keep it out of your shell history.

# 2.50 TB across 213 safetensors shards. hf_transfer keeps the NVMe disk fed;
# 32 workers saturated the link on our nodes without stalling.
export HF_HOME=/ephemeral/hf
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_TOKEN=[YOUR HUGGING FACE TOKEN]

nohup python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('Qwen/Qwen3.8-2.4T-A95B-FP8', max_workers=32)
" > /tmp/dl.log 2>&1 &

Step 5: Confirm Every Node Before You Commit to a Launch

A 64-rank job starts by having all 64 ranks meet each other. Every rank waits for the others, so the cluster is only as ready as its least ready node. Checking takes about twenty seconds per machine, and it belongs immediately before the launch rather than earlier in the session.

# Run this on every node and read the number, not the vibe. A shard that is
# still a .incomplete temporary file fails the launch twenty minutes in,
# after the rendezvous has already committed the other 63 ranks.
SHARDS=$(find /ephemeral/hf -name '*.safetensors' | wc -l)
PARTIAL=$(find /ephemeral/hf -name '*.incomplete' | wc -l)
echo "$(hostname): $SHARDS/213 shards, $PARTIAL incomplete"

q38-0: 213/213 shards, 0 incomplete
q38-1: 213/213 shards, 0 incomplete
q38-2: 213/213 shards, 0 incomplete
q38-3: 213/213 shards, 0 incomplete
q38-4: 213/213 shards, 0 incomplete
q38-5: 213/213 shards, 0 incomplete
q38-6: 213/213 shards, 0 incomplete
q38-7: 213/213 shards, 0 incomplete

The shard count is the check that pays for itself most often, because a download that stopped short looks completely normal until the engine reaches the missing file twenty minutes into loading. Three more checks round it out.

# One last look at every node, immediately before the launch rather than
# minutes before it. Twenty seconds here protects a 64-rank rendezvous.
timeout 25 nvidia-smi -L | grep -c '^GPU'          # expect 8
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader | sort -u
ip -o -4 addr show | awk '$4 ~ /^10\./ {print $2; exit}'
find /ephemeral/hf -name '*.safetensors' | wc -l   # expect 213

Step 6: Set the Launch Environment

Four values differ per node. Everything else in the launch is identical across the cluster, which is what makes this straightforward to script.

# Run on every node. RANK is 0 on the head node, then 1 through 7.
RANK=[THIS NODE RANK]
SELF=[THIS NODE PRIVATE IP]
HEAD=10.0.0.210          # the private address of rank 0
NIC=ens6                 # the interface holding the 10.x address

The head address is the private address of rank 0 and is the same on every node. The rank is the one value that has to be unique across the cluster, so set it from the machine name rather than by hand if you are scripting this.

Step 7: Launch All Eight Ranks

The arguments split into three groups: the topology, the backends the hardware calls for, and the parsers that shape the responses. Building them into one variable keeps the launch readable and keeps every node running the same string.

# The topology and the two backend choices that follow from the hardware
Q38_ARGS="--trust-remote-code \
  --model-path Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tp-size 16 --pp-size 4 --nnodes 8 \
  --node-rank $RANK \
  --dist-init-addr $HEAD:20000 --dist-timeout 1800 \
  --linear-attn-prefill-backend flashinfer \
  --linear-attn-decode-backend flashinfer \
  --mamba-full-memory-ratio 0.95 \
  --mamba-ssm-dtype bfloat16 \
  --mamba-radix-cache-strategy extra_buffer \
  --max-prefill-tokens 8192 --page-size 64 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --host 0.0.0.0 --port 30000"

The container itself needs host networking so the ranks can reach each other on the private addresses, the checkpoint mounted from the ephemeral disk, and enough shared memory for the loader. Run this on all eight nodes.

docker rm -f q38 2>/dev/null || true

docker run -d --name q38 --gpus all --network host --ipc=host \
  --shm-size 32g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$(cat $HOME/hf_token.txt)" \
  --env SGLANG_HOST_IP=$SELF \
  --env NCCL_SOCKET_IFNAME=$NIC \
  --env GLOO_SOCKET_IFNAME=$NIC \
  --env NCCL_IB_DISABLE=1 \
  --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  lmsysorg/sglang:qwen38 \
  sglang serve $Q38_ARGS

Step 8: Watch the Start-up

The engine reports its memory at five points, and those five numbers are the clearest signal that a launch is going well. Follow the head node.

# On the head node. The interesting lines are the memory checkpoints.
docker logs -f q38 2>&1 | grep -E "Init torch|Load weight|Cache is allocated|\
Memory pool|CUDA graph (begin|end)|max_total_num_tokens|Uvicorn"

The ranks meet first. Twenty-six seconds for 64 processes to join one group is normal, and the two lines about collectives are the engine adapting to a group that spans machines rather than anything going wrong.

[18:15:44 PP0 TP0] Load weight begin. avail mem=78.05 GB
Multi-thread loading shards: 100% | 213/213 [00:41<00:00, 16.52it/s]
[18:16:29 PP0 TP0] Load weight end. elapsed=45.02 s, type=Qwen3_5MoeForCausalLM,
                   quant=fp8, avail mem=41.73 GB, mem usage=36.32 GB.

Free memory falling from 78.05 GB to 41.73 GB is the moment the deployment becomes real. That 36.32 GB per GPU is the FP8 checkpoint spread across 64 cards. Our first load took 5 minutes 38 seconds reading cold from disk. On a relaunch with the page cache still warm the first rank finished in 45 seconds and the last in 55, and it is the slowest rank that decides when the cluster moves on.

[18:16:45 PP0 TP0] max_running_requests is capped to 541 by the mamba state
                   cache (max_mamba_cache_size=2164, 4 state slots per request).
[18:16:46 PP0 TP0] Mamba Cache is allocated. max_mamba_cache_size: 2164,
                   conv_state size: 0.28GB, ssm_state size: 9.51GB
[18:16:46 PP0 TP0] KV Cache is allocated. dtype: torch.bfloat16,
                   #tokens: 1803072, K size: 4.30 GB, V size: 4.30 GB
[18:16:46 PP0 TP0] Memory pool end. avail mem=22.78 GB

This is where the hybrid attention stack shows up in the accounting. The recurrent state pool takes 9.79 GB and the key-value cache takes 8.60 GB, and it is the recurrent pool that sets the ceiling on concurrent requests. The engine states the limit and the arithmetic behind it in one line: 2,164 state slots, four slots per request, so 541 requests can run at once.

[18:19:48 PP0 TP0] Capture target prefill CUDA graph end. elapsed=182.25 s,
                   mem usage=3.87 GB, avail mem=18.76 GB.
[18:20:27 PP0 TP0] Capture target decode CUDA graph end. elapsed=38.97 s,
                   mem usage=1.42 GB, avail mem=17.34 GB.
[18:20:27 PP0 TP0] max_total_num_tokens=1803072, chunked_prefill_size=8192,
                   max_prefill_tokens=8192, max_running_requests=541,
                   context_len=262144, available_gpu_mem=17.34 GB
[18:20:28] INFO:     Uvicorn running on http://0.0.0.0:30000

Capturing CUDA graphs is the longest phase of a warm start. The prefill capture walks 58 different token counts and the decode capture walks 52 batch sizes, and together they account for more than half the time between launching the container and answering a request.

Engine startup timings (s): load_weight=54.98, kv_cache_allocation=1.63,
  scheduler_e2e=317.47,
  cuda_graph={prefill=182.60, decode=38.97, target_verify=0.00},
  tokenizer_e2e=328.19

Step 9: Send the First Request

The endpoint speaks the OpenAI chat completions API on port 30000 of the head node. No other rank needs to be reachable from outside the environment.

# From your workstation, against the head node
curl -s http://[HEAD NODE IP]:30000/v1/models | head -c 200

curl -s http://[HEAD NODE IP]:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3.8-2.4T-A95B-FP8",
    "messages": [{"role": "user", "content": "Reply with exactly: OK"}],
    "max_tokens": 512
  }'

While that runs, the cluster looks like this. Roughly 63,400 MiB in use of the 81,559 MiB on each card, at low temperature, because a mixture-of-experts model at low concurrency waits on memory traffic rather than saturating the compute.

index, name, memory.used [MiB], memory.total [MiB], temperature.gpu, power.draw [W]
0, NVIDIA H100 PCIe, 63552 MiB, 81559 MiB, 26, 76.01 W
1, NVIDIA H100 PCIe, 63396 MiB, 81559 MiB, 24, 78.40 W
2, NVIDIA H100 PCIe, 63396 MiB, 81559 MiB, 36, 84.20 W
...
7, NVIDIA H100 PCIe, 63490 MiB, 81559 MiB, 25, 78.86 W

Qwen3.8 Max Inference: Reading What Comes Back

Qwen3.8 Max reasons before it answers, on every request. That single fact changes how a client has to be written, and it is the part of a Qwen3.8 Max implementation most likely to catch a team that has ported code from a model without chain of thought.

The Chain of Thought Arrives in Its Own Field

Because the server was launched with --reasoning-parser qwen3, the reasoning does not appear inline in the answer wrapped in tags. It comes back in message.reasoning_content, and the answer in message.content is clean. A client can log the reasoning, show it, or discard it, without parsing anything.

import json
import urllib.request

BASE = "http://[HEAD NODE IP]:30000/v1"
MODEL = "Qwen/Qwen3.8-2.4T-A95B-FP8"


def ask(prompt, max_tokens=4096):
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.7,
        "top_p": 0.8,
        "max_tokens": max_tokens,
    }
    request = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request, timeout=1800) as response:
        return json.load(response)

def show(prompt, max_tokens=4096):
    payload = ask(prompt, max_tokens)
    choice = payload["choices"][0]
    message = choice["message"]
    usage = payload["usage"]

    # The qwen3 reasoning parser puts the chain of thought in its own field,
    # so the answer arrives clean and the reasoning can be logged or dropped.
    thinking = message.get("reasoning_content") or ""
    answer = message.get("content") or ""

    print("finish_reason  :", choice["finish_reason"])
    print("reasoning      :", usage["reasoning_tokens"], "tokens")
    print("completion     :", usage["completion_tokens"], "tokens")
    print("answer         :", len(answer), "characters")
    return answer

The usage block carries a reasoning_tokens field alongside completion_tokens, which is the number to watch. Reasoning is not billed separately in a self-hosted deployment, but it does consume the same budget as the answer.

Reasoning and the Answer Share One Token Budget

This is the behaviour to design around. max_tokens covers the reasoning and the answer together, not the answer alone. We sent the same code prompt at two budgets.

{
  "prompt_tokens": 112,
  "completion_tokens": 2048,
  "reasoning_tokens": 1871,
  "total_tokens": 2160
}

{
  "prompt_tokens": 111,
  "completion_tokens": 6144,
  "reasoning_tokens": 6144,
  "total_tokens": 6255
}

At 6,144 the model spent the entire allowance thinking and emitted no answer tokens at all. A budget sized for a model without chain of thought does more than clip the tail here. It can return an empty response, or code that looks complete and is not.

# A budget sized for a model without chain of thought can return nothing at
# all here, because reasoning and the answer share one allowance. Treat a
# length stop as a retry signal rather than as a finished response.
answer = show("Write a Python function chunked(iterable, n) that yields "
              "lists of length n, lazily, raising ValueError if n < 1.",
              max_tokens=4096)

if not answer.strip():
    print("empty answer: raise max_tokens or lower reasoning_effort")

Reasoning Effort Is the Control Worth Wiring Up

The Qwen3.8-2.4T-A95B model card documents three reasoning levels, low, medium and xhigh, with xhigh as the default. Every measurement below was taken at the default, which is the most expensive setting and the right one to measure against. It is also the first lever to reach for when a workload does not need deep deliberation on every turn.

# reasoning_effort is the documented control. The default is xhigh, which is
# what every measurement above was taken at.
{
  "model": "Qwen/Qwen3.8-2.4T-A95B-FP8",
  "messages": [{"role": "user", "content": "..."}],
  "reasoning_effort": "low",
  "max_tokens": 4096
}

The shape of that chart is the useful part. On a two-sentence factual question the model spent 105 tokens thinking and 65 answering. On a code task it spent 1,871 thinking and 177 answering. Reasoning grows with difficulty far faster than the answer does, so a fixed budget that suits simple traffic will fail on hard traffic rather than degrade on it.

Tool Calling Works End to End

The qwen3_coder parser turns the native tool syntax of this model into OpenAI-shaped tool_calls. We tested it with a loop that executes the tools and feeds each result back, because a broken parser still produces plausible-looking first-turn output.

TOOLS = [{
    "type": "function",
    "function": {
        "name": "plan_cluster",
        "description": "Smallest cluster that leaves more than headroom_gb "
                       "free per GPU and admits a legal (tp, pp) split.",
        "parameters": {
            "type": "object",
            "properties": {
                "weights_tb": {"type": "number"},
                "vram_gb": {"type": "number"},
                "gpus_per_node": {"type": "integer"},
                "layers": {"type": "integer"},
                "headroom_gb": {"type": "number"},
            },
            "required": ["weights_tb", "vram_gb", "gpus_per_node", "layers"],
        },
    },
}]

messages = [
    {"role": "system", "content": "You are a deployment engineer. Call a tool "
                                  "for every factual number."},
    {"role": "user", "content": TASK},
]

for turn in range(8):
    payload = chat(messages, TOOLS)
    message = payload["choices"][0]["message"]
    calls = message.get("tool_calls") or []

    # Echo the assistant turn back so the model sees its own calls
    messages.append({k: v for k, v in message.items()
                     if k != "reasoning_content"})
    if not calls:
        print(message["content"])
        break

    for call in calls:
        name = call["function"]["name"]
        args = json.loads(call["function"]["arguments"] or "{}")
        result = IMPLEMENTATIONS[name](**args)
        messages.append({"role": "tool", "tool_call_id": call["id"],
                         "name": name, "content": json.dumps(result)})

The task was to size a cluster for this exact model using only tool results. It made four dependent calls in the right order, taking each argument from the previous result, across eight turns and 54.6 seconds.

lookup_model {"name": "qwen3.8-fp8"}
  -> {"weights_tb": 2.5, "layers": 92, "total_params_b": 2400}

lookup_gpu   {"flavor": "h100-80g"}
  -> {"vram_gb": 80, "gpus_per_node": 8, "usd_per_hr_node": 16.0}

plan_cluster {"weights_tb": 2.5, "vram_gb": 80, "gpus_per_node": 8,
              "layers": 92, "headroom_gb": 25}
  -> {"nodes": 8, "gpus": 64, "weights_gb_per_gpu": 39.1,
      "free_gb_per_gpu": 40.9,
      "legal_splits": [{"tp": 64, "pp": 1}, {"tp": 32, "pp": 2},
                       {"tp": 16, "pp": 4}],
      "rejected_smaller_clusters": [
        {"nodes": 6, "free_gb_per_gpu": 27.9,
         "why": "fits in memory but no tp dividing 64 heads"},
        {"nodes": 7, "free_gb_per_gpu": 35.4,
         "why": "fits in memory but no tp dividing 64 heads"}]}

cost         {"nodes": 8, "usd_per_hr_node": 16.0, "hours": 3}
  -> {"usd_per_hr_cluster": 128.0, "usd_total": 384.0}

The final answer then reproduced the non-obvious part without being prompted for it: six and seven node clusters fit the weights in memory and admit no legal placement, so they are infeasible. That is the same conclusion the architecture section reached, arrived at independently from the tool output.

It Gets the Arithmetic Right

Two of the prompts had ground truth we could check rather than eyeball. Asked for the annual cost of a 6.5 kW rack at $0.11 per kWh with cooling billed at 40 per cent of the power bill, it returned $6,263.40 for power, $2,505.36 for cooling and $8,768.76 in total, which matches an independent calculation exactly. Asked how 92 layers can be split evenly across pipeline stages, it returned the full set.

The possible numbers of pipeline stages are 2, 4, 23, 46 and 92.

Let s be the number of stages. For exactly equal layers per stage, 92 / s
must be an integer, so s must divide 92. The prime factorisation is
92 = 2^2 x 23, so its positive divisors are 1, 2, 4, 23, 46 and 92.
Excluding a single stage leaves 2, 4, 23, 46, 92.

Throughput, and What Sets It

We swept concurrency from one request to 32 against the live endpoint, with a 256 token cap on each, and measured what came back.

python3 q38_bench.py \
  --base http://[HEAD NODE IP]:30000/v1 \
  --model Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --levels 1,4,16,32 --max-tokens 256 --out ./bench

 conc   ok   wall_s  agg tok/s  per-req t/s     p50     p95
    1    1    22.42       10.9         10.9   22.42   22.42
    4    4    32.57       30.8          7.8   32.31   32.56
   16   16    87.45       45.5          2.9   86.82   87.44
   32   32    95.83       81.6          2.7   95.28   95.83

== peak aggregate 81.6 tok/s at concurrency 32

The Recurrent State Pool Is the Ceiling, Not the Context Length

On a conventional transformer, concurrency runs out when the key-value cache fills, so long prompts and many users compete for the same resource. Here the engine reported a limit of 541 concurrent requests set by the recurrent state pool, from 2,164 state slots at four slots per request. That limit does not move with prompt length, because a Gated DeltaNet layer carries the same fixed state whether the sequence is a thousand tokens or a hundred thousand.

Two flags move it. Raising --mamba-full-memory-ratio gives the pool more of the free memory, and --mamba-ssm-dtype bfloat16 halves the size of every slot against float32. Both are already set in the launch above.

Two Limits on These Figures

These numbers are a floor rather than a ceiling, for two reasons.

Using default MoE kernel config. Performance might be sub-optimal!
Config file not found at .../configs/triton_3_7_1/
  E=512,N=128,device_name=NVIDIA_H100_PCIe.json

Qwen3.8 Max Deployment Cost

A Qwen3.8 Max deployment cost is easy to reason about once the node count is settled, because the cluster bills as eight identical machines by the minute. The rate is the same whether the GPUs are loading weights, capturing graphs or serving traffic, so the only open question is how long each phase lasts.

What Each Phase Cost in This Deployment

These are the phases we timed, priced at the Spot cluster rate of $128.00 per hour, which is $2.13 a minute. Everything here is measured rather than modelled.

The Three Levers That Move the Bill

Only one of these is a tuning exercise. The other two are decisions taken before the cluster exists.

Serve a 2.4 trillion parameter model yourself

Deploy Qwen3.8 Max on 64 NVIDIA H100 GPUs

Eight nodes on Spot at $128.00 per hour. Our deployment loaded 2.50 TB across 64 GPUs, reached a live endpoint in under six minutes of engine start-up and served the full concurrency sweep without a single failed request.

64x NVIDIA H100 80GBTensor 16 x pipeline 4262,144 token context$128.00 per hour

Launch an NVIDIA GPU cluster on Hyperstack today.

FAQs

What hardware do you need to deploy Qwen3.8 Max?

The FP8 checkpoint is 2.50 TB, which is 39.1 GB per GPU across 64 cards. We ran it on eight Hyperstack nodes of 8x NVIDIA H100 80GB PCIe, leaving 17.34 GB free per GPU once the caches and CUDA graphs were resident.

How much does a Qwen3.8 Max deployment cost?

Eight nodes of 8x NVIDIA H100 80GB PCIe on Spot are $2.00 per GPU hour, so $128.00 an hour for the cluster, billed by the minute. A warm engine start costs $12.14 and the full concurrency sweep cost $8.46.

Can you turn thinking off on Qwen3.8 Max?

No. Every completion carries chain of thought, and reasoning shares the same max_tokens budget as the answer. The documented control is reasoning_effort, which takes low, medium or xhigh, and defaults to xhigh.

Why does Qwen3.8 Max need eight nodes and not six?

Six nodes hold the weights with 27.9 GB free per GPU, then fail on arithmetic. Tensor parallel 24 and 12 divide neither the 64 query heads nor the layer count, so no legal placement exists. Seven nodes fail the same test.

How long does Qwen3.8 Max take to start serving?

Five minutes 42 seconds from container start to a live endpoint with a warm page cache, or roughly ten minutes reading the 213 shards cold. Capturing CUDA graphs is 221.6 seconds of that and happens on every launch.

MiniMax H3 is a 33 billion parameter omni-modal generative system, open sourced on 3 August 2026, and it does something the text-to-video models before it did not: it writes the soundtrack at the same time as the picture. Video latents and audio latents are predicted by the same transformer, in the same forward pass, from one packed multimodal sequence. What lands on disk is a single MP4 carrying H.264 video at 24 frames per second and AAC stereo at 32 kHz, already in sync, with no separate audio model and no post-production step.

This guide serves it on one Hyperstack node: four NVIDIA H100 80GB PCIe GPUs on Spot at $8.00 per hour, through SGLang Diffusion. Every number below comes from a single session on 11 August 2026 that pulled 196 GB of checkpoints, reached a live endpoint, and produced eleven finished clips across all three task modes. Total generation time on the four GPUs was 51.5 minutes for 65.3 seconds of video, and every clip was verified with ffprobe to be carrying two channels of 32 kHz audio.

Serving it yourself means three decisions: how to place the model across four GPUs, how to configure the collectives for the fabric those GPUs sit on, and how to write a prompt without the hosted preprocessor that normally rewrites one. If you have read our technical deep dive into Kimi K3, this is the same shape of exercise on a much smaller cluster and a generative rather than a language workload. If you want a lighter video model on a single GPU first, our Wan 2.1 video generation tutorial covers that ground.

What MiniMax H3 Is, and Which Part of It You Can Run Yourself

H3 is not one model. The model card describes three modules, and only the middle one has open weights. Getting that distinction right before you book hardware saves a great deal of confusion later, because most of the published sample output was produced by all three working together.

That leaves a self-hosted deployment with two consequences and one advantage. The consequences: output is capped at a 768 pixel short edge, and nothing rewrites a rough prompt into something the model understands well. The advantage: the weights are the whole system you are running, so results are reproducible from a seed and a prompt, with no hidden service in the middle changing what you asked for.

The two output streams stay together because of how the sequence is built. Text goes through the H3-Encoder. Images and video go through the H3-Encoder and the visual VAE. Audio goes only through the audio VAE. All of it is then packed into one sequence with three-dimensional rotary position embeddings over time, height and width, and the transformer predicts video latents and audio latents jointly. Synchronisation is not a post-process, it is a property of the sequence.

What is open, and what is not

The Arithmetic That Decides the Hardware

Two numbers set the shape of the deployment. The first is what the checkpoints weigh. The repository hosts the original checkpoint and the diffusers format side by side, so scoping the download matters: pulling only model_index.json, FL2VA/* and Ref2VA/* landed 196 GB across 163 files on our node.

The second is what has to be resident while a generation runs. The official vLLM recipe puts the BF16 footprint of one task partition at roughly 135 GiB before any activations: two DiTs at 66.3 GB each, the Qwen3-VL-32B encoder at 51.5 GB, the video VAE at about 10 GB and the audio VAE at about 0.6 GB. No single 80 GB card holds that, which is why four GPUs is the floor rather than a preference.

How those four GPUs are arranged is a choice rather than a default. The SGLang MiniMax-H3 cookbook publishes three placements for 4x NVIDIA H100 80GB, all of which keep the whole pipeline resident, and they trade latency against memory.

The cookbook labels tensor parallel 2 with Ulysses degree 2 the fastest measured placement on this hardware, and it leads by 0.61 seconds. It also states plainly that pure Ulysses degree 4 cannot keep the pipeline resident on 80 GB NVIDIA H100 cards. We ran tensor parallel 4 with Ulysses degree 1, the lowest-memory row, because it leaves the most headroom for a long reference clip and asks the least of the interconnect on a PCIe fabric.

Matching the Configuration to Your Node

Two settings depend on how the four GPUs in your node are wired together. Each one is a single line in the launch command, and each one follows from a check that takes a second.

1. Read the fabric, then set NCCL to match it

Ask the driver how the four cards are connected, and ask CUDA what it will pass directly between them. The two answers agree, and together they tell you which transport NCCL should use.

# What the driver reports about how the four cards are wired together
nvidia-smi topo -m

# And what CUDA will actually permit between them
python3 -c "
import torch
for i in range(4):
    for j in range(4):
        if i != j:
            print(i, j, torch.cuda.can_device_access_peer(i, j))
"

        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity
GPU0     X      PHB     PHB     PHB     0-123           0-1
GPU1    PHB      X      PHB     PHB     0-123           0-1
GPU2    PHB     PHB      X      PHB     0-123           0-1
GPU3    PHB     PHB     PHB      X      0-123           0-1

0 1 False    0 2 False    0 3 False
1 0 False    1 2 False    1 3 False
2 0 False    2 1 False    2 3 False
3 0 False    3 1 False    3 2 False

PHB means every pair of GPUs communicates through a PCIe host bridge, which is the fabric on this flavour, and torch.cuda.can_device_access_peer reports False for all twelve ordered pairs. Collectives therefore travel through host memory, and NCCL is told so explicitly with two NCCL environment variables: NCCL_P2P_DISABLE=1 selects that path, and NCCL_CUMEM_ENABLE=0 keeps the allocator on the matching one. Neither costs anything measurable here, because the sequence is split four ways rather than thirty-two and the collectives are small next to the denoising loop they sit inside.

2. Choose the parallel placement that suits the fabric

All three documented placements fit on four NVIDIA H100 cards. Tensor parallel 2 with Ulysses degree 2 is the fastest of them by 0.61 seconds, and it is the one to reach for on a node with peer-to-peer. On a PCIe fabric, tensor parallel 4 with Ulysses degree 1 is the better fit: it is the lowest-memory row at 49.80 GB per GPU, and with Ulysses degree 1 there are no sequence-parallel groups to form, so nothing crosses the host bridge that does not have to.

Trading 0.61 seconds of pipeline latency for 16 GB of headroom per card is a comfortable trade when a single Ref2VA job peaks at 54.9 GB as it is, and it leaves room for a longer reference clip than the fastest row would.

How to Deploy MiniMax H3 on Hyperstack

Now, let us walk through the step-by-step process of standing up the node and serving MiniMax H3 across all four GPUs.

Four things decide the flavour, and n3-H100x4 supplies all four.

Step 1: Accessing Hyperstack

First, you will need an account on Hyperstack.

• Go to the Hyperstack website and log in.
• If you are new, create an account and set up your billing information. Our documentation can guide you through the initial setup.

Step 2: Deploying a 4x NVIDIA H100 Virtual Machine

From the Hyperstack dashboard, launch a single GPU virtual machine. The GPU count is fixed by the memory arithmetic above: one task partition is about 135 GiB in BF16, and no smaller shape holds it.

• Initiate Deployment: Click the "Deploy New Virtual Machine" button on the dashboard.

The button sits above the virtual machine list, under Cloud then Virtual Machines.

• Select Hardware Configuration: Choose a 4x NVIDIA H100 80GB PCIe flavour, listed in the flavour reference as n3-H100x4. Our run used the Spot variant, n3-H100x4-spot, at $2.00 per GPU per hour, which is $8.00 per hour for the node. The flavour also carries 124 vCPUs, 720 GB of RAM and a 3.2 TB ephemeral disk, all three of which this deployment uses.

Choose the NVIDIA H100-80G-PCIe card, then set the count to 4x in its dropdown.

• Choose the Operating System: Select the "Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker" image. The driver line has to match what the SGLang container expects, and this image also ships Docker and the NVIDIA container runtime ready to use, so there is no driver work at all.
• Environment: Choose an environment carrying NVIDIA H100 stock. Our run used CANADA-1.
• Select a Keypair: Choose an existing SSH keypair, or import one now. You will need it in step 4.
• Network Configuration: Assign a Public IP so you can reach the machine over SSH.
• Enable Ephemeral Storage: Make sure the ephemeral disk is attached. The checkpoints are 196 GB and the root disk is 96 GB, so this is not optional here.
• Review and Deploy: Check the settings and click "Deploy". The machine reaches ACTIVE in a couple of minutes.

Open the Ubuntu dropdown and take the R570 CUDA 12.8 with Docker build.

Step 3: Configure the Firewall

This deployment needs one inbound rule, and one port deliberately left shut. Add the first under firewall rules on the virtual machine:

Port 30010 stays closed deliberately. Every generation here runs from a script on the node, because reference images and clips are passed as file:// URIs that only resolve inside the server container. Driving the endpoint remotely would put those references out of reach and expose an unauthenticated API at the same time. If you do need it from your own machine, forward it over the SSH session you already have rather than opening a port:

# Optional. Forwards the endpoint to localhost:30010 on your own machine without
# opening anything on the node. Note that file:// conditioning still resolves only
# inside the container, so image and video references need the on-node client.
ssh -i /path/to/your-key -L 30010:127.0.0.1:30010 ubuntu@[PUBLIC IP]

Step 4: Accessing Your Node

Once the virtual machine is running, copy its Public IP from the dashboard and connect.

# The public IP is on the virtual machine's detail page in the dashboard
ssh -i /path/to/your-key ubuntu@[PUBLIC IP]

Confirm the two things this deployment depends on: four NVIDIA H100 cards, and a large disk mounted at /ephemeral.

nvidia-smi --query-gpu=index,name,memory.total --format=csv,noheader
df -h / /ephemeral | tail -2

0, NVIDIA H100 PCIe, 81559 MiB
1, NVIDIA H100 PCIe, 81559 MiB
2, NVIDIA H100 PCIe, 81559 MiB
3, NVIDIA H100 PCIe, 81559 MiB

/dev/vda1        96G   13G   83G  14% /
/dev/vdb        3.2T   89M  3.0T   1% /ephemeral

The disk line is what shapes the rest of the guide. The machine has two disks and only one of them matters here: the root filesystem is roughly 96 GB, while the NVMe disk is mounted separately at /ephemeral and offers around 3.2 TB on this flavour. It is documented under ephemeral storage. The two checkpoint partitions are 196 GB together and the container image adds more on top, so the model cache has to live on the big disk.

Step 5: Prepare the Disks and Pull the Checkpoints

This script does four things: creates the working directories on /ephemeral, installs ffmpeg and jq, and then starts the container image pull and the 196 GB weight download at the same time rather than one after the other.

#!/bin/bash
# Per-node prep for MiniMax-H3 on a single 4x NVIDIA H100 box.
# The Hugging Face cache goes to /ephemeral: root is only 96 GB, the checkpoints are not.
set -u

sudo mkdir -p /ephemeral/hf /ephemeral/media /ephemeral/out
sudo chown -R ubuntu:ubuntu /ephemeral/hf /ephemeral/media /ephemeral/out

sudo apt-get update -qq
sudo apt-get install -y -qq ffmpeg jq
python3 -m pip install -q --break-system-packages huggingface_hub hf_transfer

# Pull the engine image and the weights at the same time: neither one waits for the other.
nohup docker pull lmsysorg/sglang:dev >/tmp/pull.log 2>&1 &

nohup env HF_HOME=/ephemeral/hf HF_HUB_ENABLE_HF_TRANSFER=1 HF_TOKEN="$(cat ~/hf_token.txt)" \
  python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('MiniMaxAI/MiniMax-H3',
                  allow_patterns=['model_index.json', 'FL2VA/*', 'Ref2VA/*'],
                  max_workers=16)
" >/tmp/dl.log 2>&1 &

echo "image pull and checkpoint download started"

The download is the long pole. With hf_transfer enabled and sixteen workers it is bounded by the network rather than the disk. Check it landed before you launch anything.

du -sh /ephemeral/hf/hub/models--MiniMaxAI--MiniMax-H3
find /ephemeral/hf -name "*.safetensors" | wc -l
df -h /ephemeral | tail -1

196G    /ephemeral/hf/hub/models--MiniMaxAI--MiniMax-H3
163
/dev/vdb        3.2T  197G  2.8T   7% /ephemeral

Step 6: Confirm the GPU Topology

Check the fabric before you serve. It takes one second, and it decides two environment variables in the next step.

# What the driver reports about how the four cards are wired together
nvidia-smi topo -m

# And what CUDA will actually permit between them
python3 -c "
import torch
for i in range(4):
    for j in range(4):
        if i != j:
            print(i, j, torch.cuda.can_device_access_peer(i, j))
"

If every pair reads PHB and can_device_access_peer reports False, as on ours, keep NCCL_P2P_DISABLE=1 and NCCL_CUMEM_ENABLE=0 in the launch below. If you see NV12 and True, you are on an NVLink flavour and can leave both out. The section above explains the difference.

Step 7: Launch MiniMax H3

The base SGLang image does not ship the diffusion extra, so it is installed from the bundled source at container start, which is exactly what the cookbook Docker form does. The two environment variables from step 6 go on the docker run line, and /ephemeral/media is mounted read-only so that file conditioning resolves later without any upload step.

#!/bin/bash
# Serve MiniMax-H3 on a single 4x NVIDIA H100 node.
# The two NCCL variables suit a PCIe fabric; on an NVLink flavour you can leave them out.
#
# Usage: serve_mmh3.sh [variant]    variant = fl2va (default, serves t2va + fl2va) | ref2va
VARIANT=${1:-fl2va}

docker rm -f mmh3 2>/dev/null || true
docker run -d --name mmh3 --gpus all --shm-size 32g --ipc=host --network host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  -v /ephemeral/media:/data/minimax-h3:ro \
  --env "HF_TOKEN=$(cat $HOME/hf_token.txt)" \
  --env HF_HUB_ENABLE_HF_TRANSFER=1 \
  --env NCCL_P2P_DISABLE=1 \
  --env NCCL_CUMEM_ENABLE=0 \
  lmsysorg/sglang:dev \
  bash -lc 'python -m pip install -e "/sgl-workspace/sglang/python[diffusion]" && exec sglang serve "$@"' -- \
    --model-path MiniMaxAI/MiniMax-H3 \
    --model-variant "$VARIANT" \
    --num-gpus 4 \
    --tp-size 4 \
    --ulysses-degree 1 \
    --performance-mode speed \
    --host 0.0.0.0 \
    --port 30010

echo "launched MiniMax-H3 variant=$VARIANT on 4x NVIDIA H100"

Three flags shape what the server does. --model-variant fl2va serves both t2va and fl2va from one checkpoint partition, so nine of our eleven clips came from a single server. ref2va is a separate partition and needs a restart, which took about seven minutes including the diffusion extra install. --performance-mode speed deliberately keeps the DiT eager: the cookbook does not recommend torch.compile here because it changes numerical output for a negligible gain.

Step 8: Verify the Deployment

Confirm the pipeline end to end before committing four minutes to a full generation. Eight steps at four seconds is enough to exercise every stage.

# Eight steps and four seconds, purely to prove the pipeline runs end to end
JOB=$(curl -s -X POST http://127.0.0.1:30010/v1/videos \
  -H "Content-Type: application/json" \
  -d '{"model": "MiniMaxAI/MiniMax-H3",
       "prompt": "integrated_multimodal_description: [Shot 1] A violet ink drop blooms in clear water. The camera holds a static shot throughout.\noverall_soundscape: A single soft plink, then a quiet liquid swirl.\nnon_diegetic_music: One sustained synthesiser pad, no percussion.",
       "seconds": 4, "task": "t2va", "conditions": [],
       "target": {"short_edge": 768, "aspect_ratio": "1:1", "duration_seconds": 4.0},
       "num_inference_steps": 8, "seed": 9233}' | jq -r .id)

until [ "$(curl -s http://127.0.0.1:30010/v1/videos/$JOB | jq -r .status)" = "completed" ]; do
  sleep 4
done

curl -s http://127.0.0.1:30010/v1/videos/$JOB/content -o /ephemeral/out/smoke.mp4
ffprobe -v error -show_entries stream=codec_name,width,height,sample_rate,channels,nb_frames \
  -of csv=p=0 /ephemeral/out/smoke.mp4

h264,768,768,107
aac,32000,2,141

Two codecs, one file, 32000 Hz, two channels, back in 16.1 seconds. The 107 frames matter too: H3 quantises duration to a legal frame count, so a request for four seconds becomes 4.46 seconds.

Driving the Asynchronous Video Endpoint

H3 does not stream. A generation takes minutes, so SGLang exposes it as an OpenAI-shaped asynchronous job: submit, poll, then fetch the bytes. Three endpoints cover the whole surface.

The request body carries the decisions that shape the output. This is the exact shape we sent for every text-to-video clip.

{
  "model": "MiniMaxAI/MiniMax-H3",
  "prompt": "<the structured prompt, in full>",
  "seconds": 6,
  "task": "t2va",
  "conditions": [],
  "target": {
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "duration_seconds": 6.0
  },
  "num_outputs_per_prompt": 1,
  "num_inference_steps": 50,
  "flow_shift": 12.0,
  "audio_flow_shift": 3.0,
  "seed": 9233
}

Conditioning is passed as a list. An empty list is text-to-video; an image with the keyframe role and frame_index: 0 anchors the first frame; a video or an image with the reference role drives Ref2VA. Every URI is a file:// path resolved inside the server container, which is why the media directory is bind mounted read-only at /data/minimax-h3.

def build_conditions(task, spec):
    """MEDIA is the read-only mount inside the server container, /data/minimax-h3."""
    if task == "t2va":
        return []
    if task == "fl2va":
        img = f"{MEDIA}/{spec['source']}_frame.png"
        return [{"type": "image", "uri": "file://" + img,
                 "role": "keyframe", "frame_index": 0}]
    if task == "ref2va":
        if spec.get("source_video"):
            return [{"type": "video", "uri": f"file://{MEDIA}/{spec['source_video']}.mp4",
                     "role": "reference", "start_time_seconds": 0.0}]
        return [{"type": "image", "uri": f"file://{MEDIA}/{spec['source']}_frame.png",
                 "role": "reference"}]
    raise ValueError(task)

The client is straightforward: submit, poll on a four second interval, download, then probe the result with ffprobe and write a JSON record holding the exact request, the job id, the wall clock and the measured media properties. Those records are where every timing and every media property quoted here comes from.

BASE = "http://127.0.0.1:30010"

def run_one(task, spec, steps):
    body = {
        "model": "MiniMaxAI/MiniMax-H3",
        "prompt": spec["prompt"],
        "seconds": spec["seconds"],
        "task": task,
        "conditions": build_conditions(task, spec),
        "target": {"short_edge": 768, "aspect_ratio": spec["ratio"],
                   "duration_seconds": float(spec["seconds"])},
        "num_outputs_per_prompt": 1,
        "num_inference_steps": steps,
        "flow_shift": 12.0,
        "audio_flow_shift": 3.0,
        "seed": spec["seed"],
    }
    t0 = time.time()
    job = post("/v1/videos", body)
    vid = job.get("id")

    while True:
        time.sleep(4)
        st = get(f"/v1/videos/{vid}")
        status = st.get("status")
        if status in ("completed", "succeeded"):
            break
        if status in ("failed", "cancelled", "error"):
            return {"id": spec["id"], "error": st}
    gen_s = time.time() - t0

    mp4 = os.path.join(OUT, f"{spec['id']}.mp4")
    with urllib.request.urlopen(f"{BASE}/v1/videos/{vid}/content", timeout=600) as r, \
         open(mp4, "wb") as f:
        f.write(r.read())

    # ffprobe every file, so the record holds measured properties
    return {"id": spec["id"], "task": task, "generation_seconds": round(gen_s, 1),
            "num_inference_steps": steps, "request": body, "job_id": vid,
            "media": probe(mp4), "mp4": mp4}

Prompting Is the Entire Quality Lever

Because H3-Context-IR is not part of the open release, nothing sits between what you type and what the transformer reads, so a vague prompt stays vague. MiniMax anticipated this and published the specification its own service writes to, as a set of prompt-writing skills in the MiniMax-H3 repository. Writing to that specification is not a style preference, it is the deployment.

The specification has two shapes. Text-to-video and frame conditioning take exactly three sections. Reference conditioning takes exactly six.

The four rules that carry the most weight

• Sound is a first-class section, not an afterthought. Splitting diegetic sound from non-diegetic music is what lets the model place a bell motif in the score and a fabric rustle in the room at the same time.
• Camera motion has a canonical phrasing: motion type, then amplitude, then speed. "The camera performs a tracking shot forward with medium amplitude at slow speed" is understood. "Slow dolly in" is a guess.
• First-frame conditioning has a mandatory opening sentence. "For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced." Without it the supplied frame is treated as loose inspiration rather than as frame zero.
• Dialogue is tagged, and speakers are stable. The speaker is described outside the tag and the words go inside it with a language label, so the same identifier keeps the same voice across the clip.

A composed male presenter in his thirties (S1) says, <d>[English] Every frame you are
watching, and every sound, came out of one model in a single pass.</d>

Writing a brand palette into the pixels

Every clip here is in the Hyperstack palette, and none of it was colour graded afterwards. The palette was written into every prompt in natural language, so H3 rendered those colours directly. Hex codes mean nothing to a video model. Colour names do.

The clause that does the most work is the negative one. Every prompt ends its visual section with the same sentence: a strict colour palette of electric violet, deep royal indigo and rich amethyst, with luminous orchid highlights and pale lilac-white bloom; no warm oranges, no greens and no reds anywhere in frame. Stating what must not appear is what holds the palette steady across eleven independent generations with eleven different seeds.

integrated_multimodal_description: [Shot 1] Abstract macro cinematography with a tight,
centred composition and shallow depth of field. A single sheet of weightless liquid silk
fills the frame against a seamless near-black background. Its surface carries a smooth
135-degree gradient running from electric violet at the upper left to deep royal indigo
at the lower right. The silk folds and unfurls continuously in slow motion; rich amethyst
shadows pool inside each fold while luminous orchid catch-lights trace the rolling edges,
and a pale lilac-white bloom haloes the brightest crests. The camera holds a static shot
throughout, letting the fabric move within the frame. Rendered with a strict colour
palette of electric violet, deep royal indigo and rich amethyst, with luminous orchid
highlights and pale lilac-white bloom; no warm oranges, no greens and no reds anywhere
in frame.
overall_soundscape: A soft, airy low-frequency swell of fabric moving through still air
in a large quiet room. A faint high shimmer rises each time the light catches a fold.
There are no voices and no mechanical noise.
non_diegetic_music: Minimal ambient electronic score: one slow sustained synthesiser pad
in a major key with a single soft bell motif every few seconds and gentle sub-bass, no
percussion.

Confirming the structure before you spend GPU time on it

The specification is easy to hold to once it is encoded. Two small helpers build the prompts in the official order, and a block of assertions checks every one of them at import, so the structure is confirmed in a second rather than inferred from the finished video four minutes later.

PALETTE = ("a strict colour palette of electric violet, deep royal indigo and rich amethyst, "
           "with luminous orchid highlights and pale lilac-white bloom; no warm oranges, "
           "no greens and no reds anywhere in frame")

# The official opener, required whenever a real first frame is supplied
FIRST_FRAME_ANCHOR = ("For the target video, at 0.00 seconds into the target video, "
                      "<Picture 1> (from [Shot 1]) is fully referenced.")


def P(desc, sound, music):
    """T2VA and FL2VA: exactly three sections, in the official order."""
    return (f"integrated_multimodal_description: {desc}\n"
            f"overall_soundscape: {sound}\n"
            f"non_diegetic_music: {music}")


def R(subjects, summary, retention, detail, sound, music):
    """Ref2VA: exactly six sections, in the official order."""
    return (f"subject_definitions:\n{subjects}\n\n"
            f"summary:\n{summary}\n\n"
            f"retention_analysis:\n{retention}\n\n"
            f"detailed_description:\n{detail}\n\n"
            f"overall_soundscape:\n{sound}\n\n"
            f"non_diegetic_music:\n{music}")

# Run at import, so the structure is confirmed before any GPU time is spent.
for p in T2VA + FL2VA:
    assert p["prompt"].startswith("integrated_multimodal_description:"), p["id"]
    assert "\noverall_soundscape:" in p["prompt"], p["id"]
    assert "\nnon_diegetic_music:" in p["prompt"], p["id"]

for p in REF2VA:
    for section in ["subject_definitions:", "summary:", "retention_analysis:",
                    "detailed_description:", "overall_soundscape:", "non_diegetic_music:"]:
        assert section in p["prompt"], (p["id"], section)

for p in FL2VA:
    assert "<Picture 1>" in p["prompt"] and "0.00 seconds" in p["prompt"], p["id"]

This is the cheapest quality control in the whole pipeline. On a node costing $8.00 per hour, confirming the structure at import rather than after generation is worth roughly sixty cents of GPU time each time it runs.

Eleven Clips Across Three Task Modes

Every reference image and every reference clip used below came out of this same run. Nothing here is stock footage. After the seven text-to-video clips finished, we pulled one frame out of four of them at roughly forty per cent in, past the opening ramp where the composition has settled, and used those frames to drive the conditioned modes.

#!/bin/bash
# Pull frames out of our own T2VA output, to use as reference material.
# Frames are taken ~40% into each clip, past the opening ramp, where the composition
# has settled. /ephemeral/media is mounted read-only in the server at /data/minimax-h3.
set -u
OUT=/ephemeral/out
MEDIA=/ephemeral/media
mkdir -p "$MEDIA"

extract () {
  local src="$OUT/$1.mp4"
  local dur at
  dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$src")
  at=$(python3 -c "print(round(float('$dur') * 0.4, 2))")
  ffmpeg -y -v error -ss "$at" -i "$src" -frames:v 1 "$MEDIA/$1_frame.png"
}

for id in "$@"; do extract "$id"; done

# Ref2VA video conditioning needs the clip itself visible inside the container too
for id in "$@"; do cp -f "$OUT/$id.mp4" "$MEDIA/$id.mp4"; done

# T2VA and FL2VA are both served by the fl2va checkpoint, so these run back to back
python3 mmh3_generate.py --task t2va
bash mmh3_extract_frames.sh 01_brand_hero 02_gpu_datacentre 03_data_viz 04_aurora
python3 mmh3_generate.py --task fl2va

# Ref2VA is a separate checkpoint partition, so the server has to be restarted
bash serve_mmh3.sh ref2va
python3 mmh3_generate.py --task ref2va

Text to video and audio

Seven clips, no conditioning of any kind, straight from the three-section prompt. Each one has sound generated in the same pass as the picture, so unmute them.

Liquid gradient silk

A sheet of weightless silk carrying the 135-degree brand gradient, folding in slow motion against near-black. The soundtrack is a fabric swell with a soft bell motif over a sustained pad.

GPU data-centre aisle

A forward tracking shot with medium amplitude at slow speed down a symmetrical aisle. The volumetric haze and the reflected floor came from the prompt, not from a render pass.

Data visualisation materialising

An arc shot around glass bar-chart columns that grow in sequence. This clip later became the style reference for clip 11.

Violet aurora over dark peaks

The widest geometry in the set at 21:9 and eight seconds, which makes it the most expensive clip we generated. Note the aurora reflected in the lake, which no part of the prompt describes twice.

Violet ink bloom, macro

Square, five seconds, and the cheapest clip in the set at 104.1 seconds. Fewer pixels and fewer frames mean a shorter sequence, and the denoising loop scales with it directly.

Refracting glass prisms

An arc shot with medium amplitude at slow speed around three prisms, with caustics thrown across the floor. The audio is glassy interface tones rather than music.

Speech, lip sync and a generated voice

The same three-section prompt, with one dialogue line tagged in the official form. H3 produced the speech, the lip movement and the voice timbre together with the picture, in the same pass, on the same 50 steps. Nothing here was dubbed.

A studio presenter delivering one tagged line. The voice, the lip sync and the rim lighting are all model output from a single prompt.

This clip cost 276.2 seconds against 272.2 for the three silent 16:9 clips of the same geometry. Adding speech, lip sync and a voice to a generation added four seconds, or about 1.5 per cent. The audio stream is cheap next to the video stream it is synchronised with.

First frame to video and audio

Both clips below start from a PNG pulled out of the text-to-video output above, passed as a keyframe condition at frame_index: 0, with the mandatory anchor sentence at the top of the prompt. The task is continuation: hold the composition, keep the motion going.

The hero clip continues from its own frame

Conditioned on a frame taken 40 per cent into clip 01. The gradient keeps sliding and the folds keep forming from exactly where the reference left them.

The aurora continues from its own frame

Conditioned on a frame from clip 04, at 21:9. The peaks hold as a static silhouette while the curtains keep moving, which is what the prompt asked for explicitly.

Reference to video and audio

Ref2VA is the most capable mode and the most expensive one. It is also a separate checkpoint partition, so the server has to be restarted with --model-variant ref2va before either of these will run. The prompt shape changes completely: six sections, labelled references, and a retention marker on every label saying how much of it should survive.

Video to video: clip 02, restyled

The data-centre aisle from clip 02 as a reference video, restyled into glowing monoliths. The forward camera motion and the aisle symmetry are preserved, the surfaces and the light are not.

The retention markers are doing the work here. <Video 1> is marked partially_preserved, which keeps the motion and the pacing. <Subject 1>, the racks, is marked attribute_transfer, which keeps their position and rhythm along the aisle while changing what they are made of. The markers are the dial: fully_preserved holds a reference exactly, weak_reference keeps only its atmosphere, and the two in between set how much of it carries across.

subject_definitions:
<Video 1> is the source video for the editing task: a forward tracking shot down the
centre of a modern GPU data-centre aisle lit by vertical strips of electric violet light.
<Subject 1> is the double row of tall black server racks receding toward a vanishing
point in <Video 1>.

summary:
[video editing + reference generation] The target video restyles <Video 1> into a
futuristic night-time control room. <Subject 1> is transformed from server racks into
tall glowing monoliths while the original forward camera motion, aisle symmetry and
timing are preserved.

retention_analysis:
<Video 1> (source video editing): partially_preserved - the forward tracking motion, the
symmetrical aisle layout and the overall pacing are retained, while the surfaces, haze
density and light intensity are restyled.
<Subject 1> (appears in [Shot 1]): attribute_transfer - the racks keep their position,
scale and rhythm along the aisle but take on the appearance of smooth glowing monoliths.

detailed_description:
The target video is in a sleek, high-contrast, photoreal science-fiction style with heavy
volumetric atmosphere.
[Shot 1] Cinematic wide symmetrical composition down the centre of a darkened control-room
corridor. The two facing rows from <Subject 1> now read as tall, smooth, edge-lit monoliths
whose vertical seams glow electric violet. Denser volumetric haze fills the corridor so each
seam casts a hard directional beam across the floor, and the polished surface below mirrors
the light into long deep royal indigo streaks. Scattered luminous orchid indicator points
pulse slowly along the monolith faces, and a pale lilac-white light source marks the far end
of the corridor. The camera performs a tracking shot forward with medium amplitude at slow
speed, matching the motion of <Video 1>. Rendered with a strict colour palette of electric
violet, deep royal indigo and rich amethyst, with luminous orchid highlights and pale
lilac-white bloom; no warm oranges, no greens and no reds anywhere in frame.

overall_soundscape:
A deep mechanical hum with a wide cavernous reverberation fills the corridor, layered with
a fine airy hiss of moving air and occasional low electronic pulses.

non_diegetic_music:
Low pulsing synth ostinato with sparse metallic percussion, darker and more spacious than
the source.

Image reference: a style carried onto a new subject

A frame from clip 03 supplied as a style reference only, marked weak_reference. The bar-chart geometry does not appear: the glass material, the colour system and the particle atmosphere carry across to a network graph instead.

weak_reference is the marker that makes this work. It tells the model to take the material treatment, the lighting and the colour system, and to leave the subject behind. The same image marked fully_preserved would have produced bar charts again.

Verifying all eleven

One loop over the output directory confirms what every file contains.

for f in /ephemeral/out/*.mp4; do
  echo "$(basename $f) -> $(ffprobe -v error \
    -show_entries stream=codec_name,width,height,r_frame_rate,nb_frames,sample_rate,channels \
    -of csv=p=0 "$f" | tr '\n' ' ')"
done

01_brand_hero.mp4      -> h264,1344,768,24/1,158  aac,32000,2
02_gpu_datacentre.mp4  -> h264,1344,768,24/1,158  aac,32000,2
03_data_viz.mp4        -> h264,1344,768,24/1,158  aac,32000,2
04_aurora.mp4          -> h264,1536,672,24/1,192  aac,32000,2
05_studio_dialogue.mp4 -> h264,1344,768,24/1,158  aac,32000,2
06_ink_bloom.mp4       -> h264,768,768,24/1,124   aac,32000,2
07_glass_prisms.mp4    -> h264,1344,768,24/1,124  aac,32000,2
08_hero_continue.mp4   -> h264,1344,768,24/1,124  aac,32000,2
09_aurora_continue.mp4 -> h264,1536,672,24/1,124  aac,32000,2
10_v2v_restyle.mp4     -> h264,1344,768,24/1,124  aac,32000,2
11_image_ref_style.mp4 -> h264,1344,768,24/1,124  aac,32000,2

Every file is H.264 and AAC, at 24 frames per second, 32000 Hz, two channels. Look at that column of frame counts: 158, 192 and 124 are all of the form 17n + 5, which is exactly the frame quantisation the vLLM recipe documents. A request for six seconds becomes 158 frames, which is 6.58 seconds, and the extra 0.58 seconds is the model rounding up to the nearest legal length.

Where the Generation Time Goes

Eleven clips, 3,090.1 seconds of generation, 65.3 seconds of finished video. That is about 47 seconds of four-GPU wall clock for every second you end up with. The server log shows exactly where that time goes.

[08-11 03:53:08] Running pipeline stages: ['InputValidationStage',
  'MiniMaxH3PartitionAdmissionStage', 'MiniMaxH3TextEncodingStage',
  'MiniMaxH3VisualEncodingStage', 'MiniMaxH3AudioEncodingStage',
  'MiniMaxH3LatentPreparationStage', 'MiniMaxH3TimestepPreparationStage',
  'MiniMaxH3DenoisingStage', 'MiniMaxH3DecodingStage']
[08-11 03:53:16] [MiniMaxH3VisualEncodingStage]     finished in 5.8935 seconds
[08-11 03:53:16] [MiniMaxH3AudioEncodingStage]      finished in 0.3196 seconds
[08-11 03:53:16] [MiniMaxH3LatentPreparationStage]  finished in 0.0214 seconds
[08-11 03:53:16] [MiniMaxH3TimestepPreparationStage] finished in 0.0003 seconds
minimax_h3 denoise: 100%|##########| 49/49 [09:35<00:00, 11.75s/it]
[08-11 04:02:52] [MiniMaxH3DenoisingStage]          finished in 575.9068 seconds
[08-11 04:03:06] [MiniMaxH3DecodingStage]           finished in 2.0632 seconds
[08-11 04:03:07] Peak memory usage: 54888.00 MB

Denoising is 98.6 per cent of server-side time in both jobs. Everything else together is under nine seconds. That leaves two settings worth tuning: the number of inference steps, and the length of the sequence being denoised. Nothing else moves the needle.

The two jobs also differ by exactly one thing, and it shows. The video-reference job ran at 11.75 seconds per step; the image-reference job, at the same resolution and the same step count, ran at 6.08. A reference video is decoded, encoded and then carried in the attention sequence for every one of the 50 steps, so it nearly doubles the per-step cost. It also lifts peak memory from 46,020 MB to 54,888 MB.

Sequence length is the price, not resolution

Two of the clips isolate frame count directly. A 1344x768 frame and a 1536x672 frame contain exactly the same 1,032,192 pixels, so the only difference between clip 01 and clip 04 is the frame count: 158 against 192. Twenty-two per cent more frames cost thirty-one per cent more time.

If cost were linear in sequence length that line would be flat. It climbs from 34.1 to 43.1 seconds per megapixel-second instead, because this release ships inference with full attention only: the initial open source drop is full attention, and the sparse attention implementation MiniMax trained with is published separately at a later date. Until it lands, doubling the length of a clip costs more than twice as much.

What each task mode costs

The four clips below are the cleanest comparison in the run: identical geometry at 1344x768, identical duration at 5.17 seconds, identical 50 steps. Only the conditioning path changes.

How four NVIDIA H100 GPUs compare

The vLLM recipe publishes measured end-to-end figures for the same model on other four-GPU configurations. Normalising all of them to megapixel-seconds of finished video makes them roughly comparable, with the caveat that the NVIDIA B300 figure is a first-frame job at 1248x768 while the other two are text to video at 1344x768.

NVIDIA H100 is roughly four times the wall clock of NVIDIA B300 on this workload, which is what you would expect from a diffusion transformer that is almost entirely attention and feed-forward compute in BF16. It is also a good deal cheaper per GPU hour, at $2.00 on Spot against $7.40 for NVIDIA B300 on demand, so the cost per finished clip is much closer than the latency suggests. If you want faster iteration on NVIDIA H100 rather than faster hardware, the lever is step count: there is a community Turbo LoRA that distils first-and-last-frame generation down to four steps instead of fifty, with a distillation recipe behind it. We ran the full 50-step path throughout, on the released weights as published.

What the Run Cost

Four NVIDIA H100 80GB PCIe cards on Spot are $2.00 per GPU per hour, so the node runs at $8.00 per hour, or 0.22 cents per second. That makes the cost of every clip a simple multiplication.

The session as a whole is a different number, because the 196 GB download and two server starts are on the clock as well. End to end, from an empty node to eleven verified clips, took about two hours.

The download is the single largest line, and it is the one that does not get cheaper on faster GPUs. Everything after it is minutes, which means the marginal cost of another clip is measured in cents once the node is warm.

Generate video with native audio

Run MiniMax H3 on four NVIDIA H100 GPUs

One node on Spot at $8.00 per hour. Our run pulled 196 GB, served both checkpoint partitions and produced eleven clips with synchronised stereo audio for under $16.

4x NVIDIA H100 80GBTP 4 x Ulysses 1768p with 32 kHz stereo$8.00 per hour

Launch an NVIDIA GPU node on Hyperstack today.

FAQs

What hardware do you need to run MiniMax H3?

One task partition is about 135 GiB in BF16, so four GPUs of 80 GB is the practical floor. We ran it on 4x NVIDIA H100 80GB PCIe on a single Hyperstack node, peaking at 46.0 to 54.9 GB per GPU with tensor parallel 4 and Ulysses degree 1.

Can you get 2K video out of the open MiniMax H3 weights?

No. 2K comes from H3-Regenerate-2K, which is not part of the open release, so a self-hosted deployment produces a 768 pixel short edge. Every published aspect ratio from 21:9 to 9:16 works, and sets the long edge from there.

How long does MiniMax H3 take to generate a video on NVIDIA H100?

On 4x NVIDIA H100 80GB at 50 inference steps we measured 104.1 seconds for a five second square clip, 272.2 seconds for six seconds at 1344x768, and 600.4 seconds with a reference video. Denoising is 98.6 per cent of that.

How much does it cost to run MiniMax H3 on Hyperstack?

The 4x NVIDIA H100 80GB PCIe Spot node is $8.00 per hour, or 0.22 cents per second. The eleven clips cost $6.87 of GPU time between them, and the whole session including the 196 GB download came to about $16.

How do you write a MiniMax H3 prompt without H3-Context-IR?

Follow the prompt-writing skill in the model repository. Text-to-video and frame conditioning take exactly three sections in order: integrated_multimodal_description, overall_soundscape and non_diegetic_music. Reference conditioning takes six, and every labelled reference needs a retention marker.