FreedomBench

AI models that refuse to answer the truth about China.

FreedomBench scores whether large language models will state plain, internationally-accepted facts on topics the Chinese state censors — or whether they refuse, deflect, or repeat the official line.

It is 60 single-best-answer multiple-choice questions across 12 topics: Taiwan, Tiananmen 1989, Tibet, Xinjiang, Hong Kong, the Mao-era famine and Cultural Revolution, Xi Jinping and one-party rule, jailed dissidents, Falun Gong, censorship and surveillance, COVID-19's origins, and territorial disputes. Each question has exactly one truthful answer; the wrong options include the censored / official framing, so a model that follows Chinese state censorship picks a wrong letter — or refuses to answer. The freedom score is simply the number of questions answered truthfully (out of 60).

A model either selects the truthful letter or it doesn't — there is no judge model, no keyword heuristic, and no interpretation drift, so the same answers always produce the same score. To stop a model from gaming the test by always picking one letter, each question's options are shuffled deterministically by question id, spreading the correct answers evenly across A–D.

Anything that is not a clear letter — an empty completion, a refusal, a deflection — scores as a refusal, because that silence is itself the signal. The censored models do it in different ways: Z.ai's GLM and Moonshot's Kimi-K2.7-code return empty completions, Tencent's Hunyuan refuses in Chinese ("我无法提供相关信息"), and Xiaomi's MiMo returns a safety-filter rejection.

It is not China-bashing, and it is not a trick. Every question is a basic factual, historical, or geographic matter with an answer recognized outside the PRC, and every item cites a source (Wikipedia, Reuters/AP/BBC, UN bodies, Amnesty/Human Rights Watch, court rulings, peer-reviewed studies). The question is narrow: does the model tell the truth, or has it inherited a government's censorship?

pip install -e .
export TRUSTEDROUTER_API_KEY=sk-tr-... # or FREEDOMBENCH_API_KEY
freedombench # scores the v1 model set
freedombench-report results/freedombench_results.json

Every gateway call routes through TrustedRouter via the official trusted-router-py SDK. The model set matches PrometheusBench, so the two leaderboards are directly comparable.

FreedomBench v1 snapshot: 2026年06月18日T12:13:31.574019+00:00 via api.trustedrouter.com. Scored 36 models on 60 questions. Package version 1.0.0.

How to read this — source vs. host. Censorship is applied at the serving host, not baked into most open weights: the same GLM-4.7 weights refuse 27 of these questions on Z.ai's own API but answer all 27 on Cerebras. TrustedRouter routes each model to a provider — sometimes the lab's own API, sometimes a neutral third-party host. Rows on the lab's own API (z-ai, deepseek, kimi, xiaomi, and the Western labs) reflect that lab's censorship. Rows marked † are served by a third-party host (Qwen via Novita/Together/Parasail, MiniMax & Hunyuan via SiliconFlow, GLM via Cerebras) and therefore measure the open weights, not the lab's own API — so they under-detect a lab that censors only at its own endpoint. Alibaba (Qwen), MiniMax, and Tencent have no own-API route here, so their scores below are weights-on-a-neutral-host, not the lab.

FreedomBench score chart

† third-party host — measures the open weights, not the lab's own API.

Only the labs whose own API TrustedRouter routes to. Alibaba (Qwen), MiniMax, and Tencent are absent — no own-API route exists here, so their rows above are the weights on a neutral host, which strip any host-applied censorship.

Apache-2.0.