TrustedRouter Benchmarks

Cheap, hard, model-distinguishing capability evals, run on the leading Chinese open-weight models through the TrustedRouter gateway and published with an open, reproducible harness.

Public leaderboards under-cover the newest Chinese flagships and almost never run the Western-built factuality / instruction-following evals on them. The goal here is the data nobody else publishes:

1. the latest Chinese open-weight models (GLM-5, Kimi K2.7/K2-Thinking, DeepSeek V3.2/V4, Qwen3, MiniMax M3, Hunyuan, MiMo) on a fixed harness,
2. each eval run solo and through TrustedRouter Fusion — does fusing the open models beat the best single one, and beat frontier?
3. only evals that are cheap (small item counts, deterministic or short judges, no exotic infra) yet still unsaturated and discriminating.

Companion to PrometheusBench (refusal/permissiveness) — that one measures whether a model will answer; these measure whether it can.

Beyond the single-shot panel evals, this repo hosts two deeper studies of whether TrustedRouter Fusion beats the best single model — in agentic (tau2-bench retail) and code-generation (SciCode, vendored under scicode/) settings. Start at docs/HANDOFF.md. Headline: fusion beats solo only with a no-dominant-member, diverse multi-model panel — same-model panels (even diverse "stances") ≈ solo, and the lever is panel composition, not the synth prompt.

TrustedRouter model ids, in trbench/panel.py. Chinese open-weight models plus a small Western frontier reference line. Prune to what your account routes.

Picked from a deep-research sweep for signal-per-dollar ×ばつ Chinese-model separation ×ばつ ease of faithful public replication. Full rationale and the saturated benchmarks we deliberately skip are in EVALS.md.

GPQA Diamond is intentionally excluded — near-saturated at the frontier, so it barely separates top models.

uv venv && uv pip install -e .
export TRUSTEDROUTER_API_KEY="sk-..." # a throwaway key
# cheap smoke (first 20 prompts, a couple of models)
python -m trbench.evals.ifeval.run --models z-ai/glm-5.1,deepseek/deepseek-v4-pro \
 --prompt-limit 20 --out results/ifeval_smoke.json
# full panel (541 prompts)
python -m trbench.evals.ifeval.run --out results/ifeval.json
# score + render chart + splice into a README
python -m trbench.evals.ifeval.score results/ifeval.json \
 --svg assets/ifeval.svg --readme README.md

IFEval is the cheapest eval here: zero-shot, no system prompt, no judge model, no sandbox — just 541 prompts scored by Google's deterministic verifiers.

IFEval snapshot: 2026年06月17日T19:39:21.984871+00:00 via api.trustedrouter.com. 100 prompts, 11 models. Deterministic Python verifiers (no judge).

IFEval chart

z-ai/glm-5.2 is scored via the tinfoil confidential provider. Its default TrustedRouter route returns empty responses (a provider bug) — at face value it scored near-zero and was omitted. Routed through tinfoil it answers cleanly (0 empty) and lands at its real level (#3 here). Reproduce: TRBENCH_PROVIDER=tinfoil. The same applies to its MMLU-Pro row below.

export TRUSTEDROUTER_API_KEY="sk-..." # a throwaway key
# cheap smoke (first 10 problems, a couple of models)
python -m trbench.evals.gsm8k.run --models z-ai/glm-5.1,deepseek/deepseek-v4-flash \
 --prompt-limit 10 --out results/gsm8k_smoke.json
# full panel (1319 problems)
python -m trbench.evals.gsm8k.run --out results/gsm8k.json
# score + render chart + splice into a README
python -m trbench.evals.gsm8k.score results/gsm8k.json \
 --svg assets/gsm8k.svg --readme README.md

GSM8K is grade-school math word problems. Like IFEval it needs no judge and no sandbox: the model is asked to end with a #### <answer> marker and a deterministic numeric matcher checks exact match (falling back to the last number in the output). The canonical 1319-problem test split is fetched from the official OpenAI repo at runtime and cached under .data/.

GSM8K snapshot: 2026年06月17日T23:14:26.377661+00:00 via api.trustedrouter.com. 30-problem subset, 12 models. Deterministic numeric match (no judge). Three models omitted (z-ai/glm-5.2, moonshotai/kimi-k2.6, deepseek/deepseek-v3.2) for >10% TrustedRouter route errors during the run. Note GSM8K is near-saturated at this level — the whole panel clusters at 93–100%, the open models at 100% — so it separates these models far less than the harder evals here; it's kept as a cheap deterministic sanity check, not a discriminator.

GSM8K chart

GSM8K is near-saturated, so AIME 2025 and MATH-500 are the real math discriminators (and they have published numbers to calibrate against).

export TRUSTEDROUTER_API_KEY="sk-..." # a throwaway key
# AIME 2025 — 30 competition problems, integer answers
python -m trbench.evals.aime.run --out results/aime.json
python -m trbench.evals.aime.score results/aime.json --svg assets/aime.svg --readme README.md
# MATH-500 — 500 contest problems, LaTeX answers
python -m trbench.evals.math500.run --out results/math500.json
python -m trbench.evals.math500.score results/math500.json --svg assets/math500.svg --readme README.md

Both are deterministic (no judge): the model reasons and puts its final answer in \boxed{}. AIME checks exact integer match; MATH-500 uses the vendored Hendrycks answer-equivalence checker (LaTeX normalization), so grading matches published results. --max-tokens defaults to 16384 — these need long solutions, and a reasoning model that truncates will look artificially weak.

• Calibrated against published numbers. Before trusting any new result, we run reference models with authoritative published scores and confirm the harness reproduces them (e.g. SimpleQA Verified gemini-2.5-pro should land on the published F1 of 55.6). See CALIBRATION.md.
• Faithful, not reinvented. Each eval uses the canonical dataset and scorer. IFEval vendors Google's official verifiers verbatim (only the imports are made package-relative); see NOTICE.
• No judge where possible. IFEval and Aider score deterministically; the factuality evals use a short LLM judge run with no tools.
• Untrusted code is sandboxed. Aider runs model-generated Python, so by default each test executes in a throwaway Docker container (--network none, read-only FS, caps dropped, non-root, memory/CPU/PID limits). --sandbox host falls back to the host (throwaway VM only); --sandbox docker requires it.
• Reproducible: the raw run replays are published. Every run's per-item responses live in results/*.json and are committed, so any number here can be re-scored and audited end to end. The one exception is Chinese SimpleQA, whose replay embeds a dataset that ships no license; only its aggregate scores are published. Raw dataset caches (.data/) stay out of git.

Apache-2.0. Vendored IFEval verifiers are Apache-2.0 from google-research.

Aider polyglot (Python subset) snapshot: 2026年06月17日T13:21:24.049770+00:00. 34 Exercism exercises, pass@1, real unit tests (no judge).

Aider polyglot chart

SimpleQA Verified snapshot: 2026年06月17日T13:23:05.762381+00:00. 250 closed-book questions, no tools. Judge: openai/gpt-4.1. F-score = harmonic mean of accuracy and accuracy-given-attempted.

SimpleQA Verified chart

| Rank | Model | F-score | Correct% | Attempted% | Acc|attempted | Empty | Errors | |---:|---|---:|---:|---:|---:|---:|---:| | 1 | deepseek/deepseek-v4-pro | 55.1 | 52.7 | 91.3 | 57.7 | 0 | 9 | | 2 | anthropic/claude-opus-4.8 | 53.1 | 40.8 | 53.6 | 76.1 | 0 | 0 | | 3 | z-ai/glm-5.1 | 51.6 | 45.2 | 75.2 | 60.1 | 0 | 0 | | 4 | moonshotai/kimi-k2.6 | 49.7 | 43.2 | 74.0 | 58.4 | 0 | 0 | | 5 | z-ai/glm-5.2 | 48.3 | 40.0 | 65.6 | 61.0 | 8 | 0 | | 6 | z-ai/glm-5 | 46.5 | 44.8 | 92.8 | 48.3 | 0 | 0 | | 7 | moonshotai/kimi-k2.7-code | 41.1 | 28.6 | 39.4 | 72.6 | 0 | 9 | | 8 | deepseek/deepseek-v4-flash | 37.2 | 32.9 | 76.8 | 42.9 | 0 | 4 | | 9 | xiaomi/mimo-v2.5-pro | 33.6 | 29.2 | 73.6 | 39.7 | 0 | 0 | | 10 | tencent/hy3-preview | 28.0 | 27.6 | 96.8 | 28.5 | 0 | 0 | | 11 | deepseek/deepseek-v3.2 | 25.9 | 25.6 | 97.6 | 26.2 | 0 | 4 | | 12 | minimax/minimax-m3 | 24.1 | 17.6 | 46.0 | 38.3 | 0 | 0 | | 13 | xiaomi/mimo-v2.5 | 20.9 | 19.6 | 87.6 | 22.4 | 0 | 0 |

Grading & budget. Graded with Google's exact autorater — openai/gpt-4.1 plus the published modified SimpleQA Verified grader prompt (direct-answer + numeric acceptable-range rules). Generation budget is 32768 tokens: the verbose reasoning models (the GLM family, Kimi K2.6) spend >8192 tokens thinking on hard questions and would otherwise truncate before the answer — re-running them at the higher budget moved glm-5.1 29.8→49.7 and kimi-k2.6 31.0→49.2. The Empty column counts answers that came back blank (truncated/no committed answer): they grade NOT_ATTEMPTED, so a high count means a budget/generation problem, not knowledge. z-ai/glm-5.2 is pathological here — it returns empty on 231/250 even at 32768 (runaway reasoning that never commits), so its 6.1 reflects that, not its factual knowledge. See CALIBRATION.md.

Chinese SimpleQA snapshot: 2026年06月17日T13:26:37.168268+00:00. 250 closed-book Chinese questions, no tools. Judge: google/gemini-2.5-flash.

Chinese SimpleQA chart

| Rank | Model | F-score | Correct% | Attempted% | Acc|attempted | Errors | |---:|---|---:|---:|---:|---:|---:| | 1 | deepseek/deepseek-v4-pro | 75.9 | 73.9 | 94.8 | 78.0 | 1 | | 2 | deepseek/deepseek-v3.2 | 72.6 | 71.8 | 98.0 | 73.3 | 5 | | 3 | deepseek/deepseek-v4-flash | 72.4 | 68.8 | 90.0 | 76.4 | 0 | | 4 | moonshotai/kimi-k2.6 | 71.6 | 62.0 | 73.2 | 84.7 | 0 | | 5 | anthropic/claude-opus-4.8 | 71.3 | 61.8 | 73.5 | 84.2 | 1 | | 6 | moonshotai/kimi-k2.7-code | 71.3 | 57.7 | 61.9 | 93.2 | 11 | | 7 | z-ai/glm-5.2 | 70.4 | 59.1 | 67.8 | 87.1 | 101 | | 8 | xiaomi/mimo-v2.5-pro | 68.3 | 65.2 | 90.8 | 71.8 | 0 | | 9 | z-ai/glm-5 | 67.1 | 58.5 | 74.2 | 78.8 | 2 | | 10 | tencent/hy3-preview | 66.5 | 65.2 | 96.0 | 67.9 | 0 | | 11 | xiaomi/mimo-v2.5 | 62.3 | 59.2 | 90.0 | 65.8 | 0 | | 12 | minimax/minimax-m3 | 59.7 | 54.0 | 80.8 | 66.8 | 0 | | 13 | z-ai/glm-5.1 | 59.7 | 46.0 | 54.0 | 85.2 | 0 |

tau2-bench snapshot: 2026年06月19日T02:23:54.885426+00:00. Domain retail, 20 tasks x 1 trial(s), agent vs fixed user openai/gpt-4.1. Metric: pass^1 (task reward == 1). Small-subset numbers run high; the ranking is the signal.

tau2-bench chart

MMLU-Pro snapshot: 2026年06月19日T04:56:04.874288+00:00 via api.trustedrouter.com. 150 questions, 15 models. Zero-shot CoT, deterministic letter-match (no judge).

MMLU-Pro chart

export TRUSTEDROUTER_API_KEY="sk-..." # a throwaway key
uv add datasets # one-time, for the HF pull
# generate answers (resumable; per-item JSONL sidecar)
python -m trbench.evals.beam.run --models z-ai/glm-5.2,deepseek/deepseek-v4-pro \
 --limit 40 --resume --out results/beam_128k_panel.json
# judge (BEAM's exact grader: gpt-4.1-mini) + render chart + splice README
python -m trbench.evals.beam.score results/beam_128k_panel.json --readme README.md

BEAM ("Beyond a Million Tokens", ICLR 2026) tests long-term memory: each probing question ships with its full ~128K-token conversation, and the model must answer from memory across 10 abilities (info extraction, temporal/multi-hop reasoning, abstention, contradiction resolution, etc.). We vendor BEAM's exact grader verbatim (beam_grader_prompt.py + gpt-4.1-mini judge + int() 0.5→0 flooring) rather than reimplement it. --limit is type-balanced (equal items per ability); reasoners need --max-tokens 8192 (thinking truncates a 1–2K budget to empty). This is the open-weight panel + the paper's two baselines (gpt-4.1-nano, llama-4-maverick-fp8); cost scales with input price ×ばつ the 128K context.

BEAM 128K snapshot: 2026年06月20日T12:03:12.956958+00:00. 40 probing questions across 20 conversations (~128K tokens each). Rubric-based LLM judge (openai/gpt-4.1-mini). Overall = mean across 10 memory-ability categories.

BEAM 128K chart

Terminal-Bench results coming soon.