Can Qwen 4 Compete With Claude Opus 5.5? We Put It to the Test

New models launch with their makers’ own benchmark scores, but public benchmark questions can leak into training data, so those scores may not show how a model handles work it has not seen. That is why we test new models on the same sealed tasks, built on Kestrelmoor Freight, a synthetic company.

Our latest blog puts Alibaba’s Qwen 4 through those tests, covering the results first, then what they mean for your workloads, then the code to reproduce the test yourself.

How the Qwen 4 Benchmark Works

We built Kestrelmoor Freight for this test: a fictional logistics company whose invoices, dashboards, app screens, screen recordings and case files are all generated by code. The generator knows every answer, so each task is graded exactly, and none of it can appear in a model’s training data.

The map put Qwen 4, Claude Opus 5.5, Kimi K3.1 Pro and three earlier Qwens through all nine jobs. The duel then reran the four jobs where Qwen 4 looked strongest (dense screens, paperwork, hard paperwork and writing to 16 to 20 rules) on 400 new tasks, each sealed with a cryptographic hash before any model saw it. A scale run then had Qwen 4 read 1,250 invoices and receipts.

Qwen 4 vs Claude Opus 5.5 on Sealed Tests

On all four sealed jobs, Opus led Qwen 4 by 0.2 to 0.8 points, and only the paperwork gap holds up statistically. The largest generational step is writing to 16 to 20 rules: 99.6 against 87.3 for Qwen3.8 Max, which ran out of tokens on 18 of 100 tasks.

Claude Fable 5.1, which joined only the duel, led Qwen 4 on all four jobs, by 0.3 to 1.1 points. The open-weight Kimi K3.1 Pro scored 0.2 to 0.8 points above Qwen 4 on three jobs, within 0.1 points of Opus on each, and Qwen 4 led it only on hard paperwork, by 0.5 points. The earlier Qwens trail Qwen 4: Qwen3.8 Max by 0.5 to 12.3 points, and Qwen3.8 Flash-Next and Omni-Flash by up to 31.2 and 34.9 points, on 16 to 20 rules.

Where Qwen 4 Stands Out

Five tasks from the test show the range:

Where Claude Opus 5.5 Keeps an Edge

Opus leads on two jobs: dashboard threshold times, 100% against 71% in the first round, and long archives, 99.0 against 84.4. Qwen 4’s dashboard misses are chart-reading errors: given the same data as text, it answered every question. Its archive misses all came at 128K tokens or more.

Opus also wrote about a third of Qwen 4’s output tokens, at a lower reasoning-effort setting: high against xhigh.

Qwen 4 Speed and Token Efficiency

On the same 912 tasks, Qwen 4 wrote 20% fewer output tokens than Qwen3.8 Max and answered in 40% less time. It also used fewer input tokens than Claude on a screenshot, 1,262 against 1,728, and fewer on a 12-second clip sent as video than as frames, 7,823 against 15,453:

Sending Images and Video to Qwen 4

One converter builds every Qwen 4 request as an OpenAI-compatible chat completion, with images and video inline as base64 data URLs:

MEDIA_TYPES = {".png": "image/png", ".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".mp4": "video/mp4"}
# ...

def _b64(rel: str) -> tuple[str, str]:
    path = EVAL / rel
    return MEDIA_TYPES[path.suffix.lower()], base64.b64encode(path.read_bytes()).decode()


def to_openai(prompt: dict) -> list:
    parts = []
    for rel in prompt.get("images") or []:
        mt, data = _b64(rel)
        parts.append({"type": "image_url", "image_url": {"url": f"data:{mt};base64,{data}"}})
    if prompt.get("video"):
        mt, data = _b64(prompt["video"])
        parts.append({"type": "video_url", "video_url": {"url": f"data:{mt};base64,{data}"}})
    parts.append({"type": "text", "text": prompt["text"]})
    msgs = [{"role": "system", "content": prompt["system"]}] if prompt.get("system") else []
    msgs.append({"role": "user", "content": parts if len(parts) > 1 else prompt["text"]})
    return msgs

The body streams the reply, reports usage and caps output, reasoning included; the map and duel ran Qwen 4 at effort xhigh:

body = {"model": self.model, "messages": messages, "stream": True,
        "stream_options": {"include_usage": True}}
if effort is not None:
    body["reasoning_effort"] = effort
if max_completion_tokens is not None:
    body["max_completion_tokens"] = max_completion_tokens

The photographed Russian invoice from the showcase cost 2,147 image tokens, and Qwen 4 reasoned for 7,613 tokens before a 2,090-token answer:

{
  "prompt_tokens": 2401,
  "total_tokens": 12104,
  "completion_tokens": 9703,
  "prompt_tokens_details": {
    "image_tokens": 2147,
    "text_tokens": 254
  },
  "completion_tokens_details": {
    "reasoning_tokens": 7613,
    "text_tokens": 2090
  }
}

Run Qwen Models on Hyperstack

Qwen 4 is not available on Hyperstack, as we tested it through early API access. To get our next model test as soon as it is published, subscribe below. You can start building with the Qwen family now, as two models from this test run on Hyperstack:

Both use an OpenAI-compatible API, as our GLM-5.2 tutorial shows; Dedicated Inference serves a model on its own GPUs.

Hyperstack AI Studio

Build With Qwen Models on Hyperstack

Call Qwen3.8 Max and Qwen3.8 Flash-Next through AI Studio, or run open models on NVIDIA H100 GPUs.

OpenAI-compatible APINVIDIA H100Per-minute billing

Get started with Hyperstack in minutes.

FAQs

What is Qwen 4?

Qwen 4 is the next major release of Alibaba’s Qwen models, tested here through early API access.

How does Qwen 4 compare with Claude Opus 5.5?

Opus finished slightly ahead, within one point, on all four sealed jobs, and led more clearly on dashboard times and the longest archives.

Is Qwen 4 more efficient than Qwen3.8 Max?

On the same 912 tasks, both at effort xhigh, it wrote 20% fewer output tokens and answered in 40% less time.

Can I run Qwen models on Hyperstack?

Yes. Qwen3.8 Flash-Next and Qwen3.8 Max run on Hyperstack AI Studio.