qwen-bench

This is the index repository for an ongoing series of inference benchmarks measuring Qwen models on consumer- and workstation-class NVIDIA Blackwell hardware (RTX PRO 6000 Blackwell, RTX 5090, etc.) under realistic agent-style production load.

Each study is a separately-versioned satellite repo with raw data, configs, logs, and a written report. This hub repo provides:

• 📊 The current SOTA leaderboard across all studies → SOTA.md
• 📚 A chronological index of every study → STUDIES.md
• 📈 Merged result CSVs combining numbers from all studies → data/
• 🛠️ Pointers to the toolchain that produced these results

The hub is the front door. The studies are the substance. The tools are reusable primitives. None of these collapse into the others; each has its own permanent, citable URL.

Last updated: 2026年05月15日 · Hardware: ×ばつ NVIDIA RTX PRO 6000 Blackwell (TP=2, SM120, 96 GB each, PCIe Gen5 x16)

The configuration currently live on production (vllm-qwen36-27b-sota.service, promoted 2026年05月12日 ~10:03 MSK). 88.4 % HE / 89.1 % MBPP / 369 tok/s peak / 98 tok/s single-user / 0 length-truncated. Reasoning routed cleanly into the OpenAI reasoning field; content clean. For single-user OpenCode coding, the current deployed client profile now adds thinking_token_budget=2048, eliminating runaway thinking loops while preserving thinking mode.

Benchmark-only — NOT deployable: FP8+MTP=5 scored 93.3 % HE / 402 tok/s peak in the offline harness but leaks raw <think>...</think> blocks into the OpenAI content field on production traffic. The deployed MTP=3 config was validated leak-free across 420 trials (300 plain-chat @ T=0.7 + 120 tool/function-calling @ T=0.7, scanning content, tool_calls[*].function.name, and tool_calls[*].function.arguments) by the permanent dual-mode leak probe (harness/leak_probe.py). See v3 suite Production Incident and LEAK_DETECTION.md.

⚠️ HumanEval scores corrected 2026年05月11日. A bug in the harness's extract_code() deflated every prior HE result by 13–23 pp. Original (pre-correction) numbers are preserved with strikethrough in SOTA.md § 2.1. See the ADDENDUM for the full bug analysis and fix.

See SOTA.md for the complete cross-study record book.

Each study is a self-contained satellite repo or hub-native production addendum. Studies are listed newest-first.

studies/2026-05-15-thinking-budget-generalization · cross-domain validation · GPQA Diamond / GSM-Plus 2k / MMLU-Pro 1.4k · 3,598 trials · GPQA budget sweep

Repne challenge: prove thinking_token_budget=2048 generalizes beyond the morning's coding probes. Result: strict Pareto improvement on every benchmark. GPQA +33.8 pp accuracy at ×ばつ wall (z=6.81, p=1e-11; 57.1% of unbounded responses never emit an answer because they bump into finish_reason=length inside <think>). GSM-Plus accuracy parity at ×ばつ wall, ×ばつ tighter p95 token tail. MMLU-Pro Δ +9.5 pp accuracy at ×ばつ wall. GPQA budget sweep at tb ∈ {1024, 2048, 4096, 8192} shows peak at tb=4096 → sweet spot exists.

studies/2026-05-15-single-user-thinking-budget · client-side hard thinking budget · 5-problem coding probe · c=1/2/4/8 fanout sweep

Follow-up to the v3 production rollout: the engine was correct, but unbounded Qwen3 thinking caused coding prompts to spiral into 50k-character reasoning tails. thinking_token_budget=2048 kept thinking enabled, improved pass rate 70%→80%, eliminated stuck responses 10%→0%, and cut c=1 p95 latency 130.8s→19.6s.

qwen-bench-2026-05-dflash-v2-sweep · 13 configs · 195 cells · 421 quality problems · 7h total wall time

Two-stage sweep on Qwen3.6-27B BF16+DFlash, TP=2: ×ばつ3 grid of --max-num-batched-tokens ×ばつ --max-cudagraph-capture-size (Stage A), then num_speculative_tokens ∈ {4, 8, 15, 16} at the winner (Stage B), plus a two-config Quality comparison establishing the c=8 noise floor.

Headline: Buffer/graph axis is flat (Δ=3.6 %). Speculative-tokens axis is decisive: n=4 → −62 %, n=8 → winner, n=16 → −42.6 % cliff. Confirms Repne's published defaults (batched=32768 capture=256 num_spec=8) are optimal.

qwen36-27b-blackwell-stress-validation · 5 configs ×ばつ 4 phases ×ばつ 2,105 hard problems · Zero crashes

A controlled head-to-head of 5 speculative-decoding configurations (FP8+MTP={3,5}, BF16+DFlash={7,8,15}) on Qwen3.6-27B across functional gates, throughput matrix, HumanEval, and MBPP. Adds two addenda re-running DFlash variants without the deprecated gumbel flag and characterizing the previously-untested FP8+DFlash pairing across N ∈ {7, 8, 15}.

Headline: FP8+MTP=3 holds HumanEval pass-rate SOTA on this study (corrected: 92.1 %, was 79.3 % pre-harness-bug-fix); BF16+DFlash variants hold MBPP SOTA (89.5 %); FP8+DFlash N=8 wins long-context.

qwen36-27b-blackwell-inference-study · 8 experiments · 1,200+ benchmark runs · 102K KLD probe

A systematic 24-hour sprint comparing the repne/vllm fork against upstream vLLM v0.19.1 / v0.20.1, plus llama.cpp Q8_0 GGUF, across four quantization schemes (BF16, FP8 W8A8, NVFP4, GGUF Q8_0) and three speculative-decoding methods.

Headline: Peak 2,083 tok/s at c=32. Q8 GGUF KLD vs BF16 = 0.0018 (noise floor).

The following pairwise-comparison repos contain the same data as subsections of inference-study. They remain published as independent citable artifacts but the canonical home is the parent study:

• qwen36-27b-bf16-dflash-repne-vs-upstream → exp 05
• qwen36-27b-fp8-repne-vs-upstream → exp 04
• qwen36-27b-nvfp4-mtp-experiment → exp 03
• repne-dflash-newimage → image-version regression sub-experiment

closing-the-opus-gap · 3,500 API calls · 10 phases · Qwen3 235B and GLM-4.7 on Cerebras

The earlier study that kicked this whole line of work off. Tool-calling optimization on wafer-scale hardware. Different model and different hardware class, but methodologically the same family of work. Included here for context.

See STUDIES.md for the full chronological index with abstracts.

All studies above were instrumented with:

The diagnostic toolkit — generic, model-agnostic, OpenAI-compatible:

• harness/stress_harness.py — 322-line failure-taxonomic correctness probe
• orchestrator/four_phase_harness.sh — end-to-end 4-phase validation runner
• launchers/launch_*.sh — parametric vLLM launcher templates (FP8+MTP, FP8+DFlash, BF16+DFlash)
• utils/wait_vllm_ready.sh — readiness probe with post-ready settle window

The toolkit lives in its own repo so it can grow to other models and stay reusable. Studies pin a specific commit of the toolkit for reproducibility.

 ┌───────────────────────────┐
 │ you are here │
 │ jcartu/qwen-bench │
 │ (the hub / index) │
 └────────┬──────────────────┘
 │
 ┌──────────────────────────┼──────────────────────────┐
 │ │ │
 ▼ ▼ ▼
┌──────────────┐ ┌────────────────┐ ┌────────────────┐
│ STUDIES │ │ SOTA │ │ data/ │
│ list of all │ │ rolling │ │ merged CSVs │
│ studies │ │ leaderboard │ │ (machine │
│ │ │ │ │ readable) │
└──────────────┘ └────────────────┘ └────────────────┘
 │
 ├──────► jcartu/qwen36-27b-blackwell-stress-validation
 ├──────► jcartu/qwen36-27b-blackwell-inference-study
 ├──────► jcartu/closing-the-opus-gap
 └──────► (future studies land here)
 │
 └──── instrumented with ────► jcartu/llm-stress-harness

Three-question reader flow:

1. "What's the best config for X?" → start at SOTA.md
2. "How was that measured?" → click through to the linked study repo
3. "Can I reproduce this?" → see Reproduce a result below

All studies in this hub follow shared conventions so numbers compare across studies:

• Pass@k for k>1. All correctness scores are single-shot at t=0.
• Prefill-only / decode-only throughput. Use vllm bench for that. These studies are end-to-end.
• Reasoning-content quality. We log reasoning_chars for diagnostics, not for grading.
• Cross-model leaderboards. This hub is Qwen-only by design. Cross-model is out of scope.

# 1. Read the study's METHODOLOGY section
gh repo view jcartu/qwen36-27b-blackwell-stress-validation --web
# 2. Pull the toolkit at the version the study used
git clone https://github.com/jcartu/llm-stress-harness
cd llm-stress-harness && git checkout <commit-sha-from-study>
# 3. Launch the documented config
NUM_SPEC=3 ./launchers/launch_fp8_mtp.sh # or whichever the study used
# 4. Run the same orchestrator
./orchestrator/four_phase_harness.sh <config-label> <kv-budget> ./out/
# 5. Diff your numbers against the published CSV
python3 -c "import pandas as pd; print(pd.read_csv('your_results.csv').compare(pd.read_csv('published.csv')))"

Numbers should land within ±2 % at the same hardware tier. If not, file an issue against the study repo with both CSVs.

When you publish a new benchmark study:

1. Name the new study repo qwen-bench-{YYYY-MM}-{slug} (convention applies to studies created 2026-05 onward; earlier study repos keep their original names — their URLs are stable forever)
2. Add a ← qwen-bench hub badge to the new study's README (line 1)
3. Set GitHub topics: qwen, qwen3, vllm, blackwell, benchmark, plus study-specific tags
4. Open a PR against this hub that:
   • Adds an entry to STUDIES.md with abstract + headline
   • Updates SOTA.md if any record was broken
   • Drops the study's master.csv or hub-native addendum CSV into data/{YYYY-MM}-{slug}.csv
   • Bumps the studies-N_published shield count at the top of this README

Full per-study checklist, slug guidelines, and URL-stability rationale: CONTRIBUTING.md

A future automated script (planned) will compute SOTA-record diffs from the merged CSV and prevent regressions from being missed.

The work indexed here builds on:

• Qwen team — open-weight model lineage
• vLLM — primary inference engine
• Repne — vLLM fork with DFlash + advanced spec-decode support
• HumanEval, MBPP, EvalPlus — benchmark sources
• AesSedai — sliding-window perplexity branch used in KLD probe

Hub illustrations generated with Gemini 3.1 nano banana (gemini-3.1-flash-image-preview).

MIT. See LICENSE.

Each satellite study repo is independently licensed; check the study's own LICENSE file.