Copied to Clipboard
What you need on hand
Before compiling, confirm you have Git, CMake 3.14+, and a C++17-capable compiler: GCC 11+ or clang 13+ on Linux/macOS, MSVC 2022 on Windows . These are current project minimums; newer versions work fine.
You also need a GGUF model file. A practical starting point is qwen3-8b-q4_k_m.gguf — fetch it with huggingface-cli download or let llama-server's --hf flag pull it at startup. The path goes into llama-bench's -m argument.
A GPU is optional but required for -fa auto to activate flash attention. Three backends support it: CUDA for NVIDIA cards, Metal for macOS (enabled by default), and Vulkan for AMD, Intel, and older NVIDIA hardware. On a CPU-only host, -fa auto stays off — no error, no change to the output format, just standard attention.
Hands-on: compile and run the corrected bench
These steps target Linux/macOS. On Windows, substitute -j$(nproc) with -j%NUMBER_OF_PROCESSORS% and run from a Developer Command Prompt for MSVC builds. Full platform-specific options are in docs/build.md.
-
Clone the repository and confirm your build is at b9437 or later.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git log --oneline -1
The top commit should reference the -fa bench PR or show a hash at or after b9437. Continuous builds don't carry semantic version tags; cross-check against the releases page if you're unsure.
- Compile for your backend.
CPU-only:
cmake -B build && cmake --build build --config Release -j$(nproc)
CUDA (NVIDIA):
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j$(nproc)
Metal is on by default on macOS — no extra flag needed. Vulkan (cross-platform AMD/Intel/NVIDIA):
cmake -B build -DGGML_VULKAN=ON && cmake --build build --config Release -j$(nproc)
-
Run the benchmark with the b9437 defaults made explicit.
./build/bin/llama-bench -m ./models/qwen3-8b-q4_k_m.gguf \
-ngl -1 -fa auto -p 512 -n 128 -r 3
-p 512 sets prompt tokens (prefill throughput), -n 128 sets generated tokens (generation throughput), -r 3 repeats the run three times and averages. Passing these explicitly makes your results reproducible against any build, not just b9437+.
-
Confirm FA actually activated.
./build/bin/llama-bench -m ./models/qwen3-8b-q4_k_m.gguf \
-ngl -1 -fa auto -p 512 -n 128 -r 3 --verbose
Look for flash_attn = 1 in the model load output. If you see flash_attn = 0 on a CUDA host, the backend was compiled without -DGGML_CUDA=ON — delete your build directory and recompile with the flag.
-
Reproduce pre-b9437 behavior for a direct comparison.
./build/bin/llama-bench -m ./models/qwen3-8b-q4_k_m.gguf \
-fa off -ngl 99
This resets both flags to their pre-b9437 defaults, giving you an apples-to-apples baseline if you have historical numbers to compare against.
Where old comparisons break
Any llama-bench run before b9437 used -fa off as the implicit default — even on hardware that fully supports flash attention. If you have recorded t/s numbers from those builds and your hardware supports FA, those figures captured the slower attention path without indicating it. To align old results with new defaults, either re-run old baselines with -fa off -ngl 99 (matching the original behavior) or re-run everything with -fa auto to get forward-comparable numbers. In either case, make the -fa state an explicit column in your benchmark output going forward.
The -ngl 99 legacy default also caused a quiet footgun on CPU-only hosts: with no -ngl flag set, the runtime attempted to load all 99 layers to GPU, triggering CUDA initialization errors even with no GPU present. With -ngl -1, the runtime skips GPU offload when no backend is detected, removing that noise from logs entirely.
Multi-Token Prediction gains for Qwen 3.6 27B dense — approximately 77 to 96 t/s on an RTX 4090 , a 24% throughput increase via PR #22673 — were measured in a separate context from b9437's defaults change. If you're trying to reproduce those figures, verify the -fa state from the original run; a mismatch gives you a result that is neither the clean MTP baseline nor a combined MTP+FA measurement.
What to explore beyond b9437
Three nearby builds are worth pulling alongside b9437:
-
b9436 (May 30, 2026, 14:25 UTC) : The OpenCL backend gains BF16-via-FP16 conversion. If you run BF16-format models on AMD or Intel hardware via the OpenCL or Vulkan path, this expands compatibility without requiring native BF16 support in the GPU.
-
b9439 (May 31, 2026, 06:57 UTC) : Multi-GPU hosts now default to using only one integrated GPU, preventing automatic selection of a low-performance iGPU alongside a discrete card. If you run a hybrid system — laptop with a discrete GPU and Intel UHD, for example — verify your device selection is still correct after updating.
-
Multi-Token Prediction for Qwen 3.6 (PR #22673): Approximately 24% throughput gain on dense 27B models . Enable with
--mtp-n-draft and confirm your GGUF quant is compatible. MoE variants (Qwen 3.6 35B-A3B) show mixed results — expert-union verifier overhead can negate the gains on consumer hardware.
Frequently Asked Questions
What does -fa auto mean in llama-bench after b9437?
-fa auto sets flash attention to LLAMA_FLASH_ATTN_TYPE_AUTO, telling the runtime to enable flash attention when the backend supports it. Before b9437, llama-bench always defaulted to -fa off — unlike llama-server and llama-cli, which already had the tri-state auto|off|on flag. After b9437, all three tools use the same flag semantics.
Are pre-b9437 llama-bench numbers still valid?
Yes, with caveats. If your original run explicitly passed -fa off, or the host hardware does not support flash attention, the numbers remain comparable. If you relied on the default and ran on FA-capable hardware — CUDA, Metal, or Vulkan — those measurements were taken without flash attention even though the GPU supported it. Re-run with matched flags to produce a clean, apples-to-apples comparison.
Why did -ngl default to 99 instead of -1?
99 was a legacy sentinel meaning "offload all layers to GPU." The project later standardized -1 as the runtime-decides value across the toolchain. llama-bench was simply never updated to match until b9437 brought it into alignment with llama-server and llama-cli.
Do I need to recompile from source to get b9437?
Yes, for a local source build: pull the latest commit from ggml-org/llama.cpp and recompile. Tagged binary releases lag the continuous builds. Check the GitHub releases page for a pre-built artifact if you want to skip compilation, but verify the build number includes the b9437 changes before treating it as current.
Does -fa auto now behave the same across llama-bench, llama-cli, and llama-server?
Yes — b9437 closes the gap. llama-cli and llama-server already supported the -fa auto|off|on tri-state. b9437 brings llama-bench into parity, so flag semantics are now consistent across all three tools. A flag value you validated in llama-server means exactly the same thing when passed to llama-bench.
Rebaseline before your next regression run
After pulling b9437 or later, the immediate action is straightforward: re-baseline any llama-bench results used for regression tracking, and make the -fa state an explicit column in your output going forward. The default change is a minor toolchain alignment, but its effect on benchmark validity is concrete — any pre-b9437 run on CUDA, Metal, or Vulkan was silently measuring the slower attention path.
If you're on a multi-GPU system, pull at least b9439 alongside for the iGPU default fix. And if Qwen 3.6 throughput is in your test matrix, keep Multi-Token Prediction's --mtp-n-draft flag in scope — the roughly 24% gain on dense 27B is worth measuring, but MoE variant results vary enough that you'll want numbers from your own hardware and quant configuration.
Last updated: 2026年06月01日. Based on llama.cpp continuous builds b9436–b9439 (May 30–31, 2026) .