Add llama.cpp parameter estimator for VRAM fit #6

Copilot commented 2026年03月14日 19:32:51 +01:00 (Migrated from github.com)

This change introduces a script to determine the llama.cpp flags needed to run a model on a given system, accounting for VRAM fit, model layers, MoE offsets, context size, batch size, and KV cache quantization.

• Estimator script: adds bin/llama-cpp-params.py to parse GGUF metadata or overrides, compute memory usage, and recommend --n-gpu-layers plus core runtime flags.
• Memory modeling: estimates per-layer weights, KV cache, and batch buffers; supports MoE CPU layer offsets and cache quantization sizes.
• Docs & tests: documents usage in bin/README.md and adds focused unit tests for parsing and estimation helpers.

Example:

./bin/llama-cpp-params.py \
 --model-path /models/mixtral.gguf \
 --vram 24GB \
 --ctx-size 8192 \
 --batch-size 2048 \
 --ubatch-size 512 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0 \
 --moe-cpu-offset 4

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

This change introduces a script to determine the llama.cpp flags needed to run a model on a given system, accounting for VRAM fit, model layers, MoE offsets, context size, batch size, and KV cache quantization. - **Estimator script**: adds `bin/llama-cpp-params.py` to parse GGUF metadata or overrides, compute memory usage, and recommend `--n-gpu-layers` plus core runtime flags. - **Memory modeling**: estimates per-layer weights, KV cache, and batch buffers; supports MoE CPU layer offsets and cache quantization sizes. - **Docs & tests**: documents usage in `bin/README.md` and adds focused unit tests for parsing and estimation helpers. Example: ```bash ./bin/llama-cpp-params.py \ --model-path /models/mixtral.gguf \ --vram 24GB \ --ctx-size 8192 \ --batch-size 2048 \ --ubatch-size 512 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --moe-cpu-offset 4 ``` <!-- START COPILOT CODING AGENT TIPS --> --- 🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security)