imp

License CUDA 13.3 C++23

imp is an LLM inference engine that targets exactly one chip: the NVIDIA RTX 5090.

What it is

• A from-scratch C++23/CUDA engine for consumer Blackwell (sm_120a), with its own GGUF and SafeTensors loaders, tokenizer, paged KV cache and kernels.
• A server that speaks both the OpenAI and the Anthropic APIs natively, so an agent stack written against either runs without a shim.
• Also a C library and a CLI, not only a server.

What it is not

• Portable. There is no CPU path, no other GPU, no fallback.
• A multi-GPU or datacenter-batching engine.
• A supported product. One author, no SLO, no support rotation.

If more than one cell on the right applies to you, use llama.cpp for breadth or vLLM for scale-out across GPUs and machines. Both are better at those jobs than imp is, and imp does not try to be. On one 5090 the picture depends on the model: on the Qwen3.8-27B hybrid imp leads vLLM by 25 % at 32 concurrent streams and by 16 % at 8 on the same checkpoint and client; on a dense NVFP4 checkpoint (Qwen3-14B) imp leads by 8 % at 8 streams and by 3 % at 32, and vLLM by 1.3x at 32 with ~1000-token prompts (table with provenance in docs/BENCHMARKS.md, "re-measured on one client").

Everything runs in Docker; the host needs no CUDA toolkit. The worked example is Qwen3.8-27B, a 27B multimodal model quantized to NVFP4 so it fits one 5090 with room for a real context. The 60 seconds start once the weights are on disk; the two steps before that are a build and a download.

1. Build the image (~3.5 min). scripts/stage-model.sh runs the downloader inside the imp:test image this produces and exits 1 without it (#1682). docker compose build imp-server makes imp:latest, a different tag.

make build

2. Get the weights (19.2 GiB, download only, nothing to convert). The checkpoint is kekzle/Qwen3.8-27B-NVFP4-vllm, an imp-quantize --format vllm export; the same directory also loads in vLLM (verified on 0.27.1 and 0.28.0).

scripts/stage-model.sh kekzle/Qwen3.8-27B-NVFP4-vllm ~/models/Qwen3.8-27B-NVFP4-vllm

3. Serve. The cache volume is optional and pays for itself on the second start: it holds the transformed weights (Qwen3-14B-NVFP4 init 7.9 s → 2.1 s).

docker run --gpus all -v ~/models:/models -v imp-cache:/home/imp/.cache/imp \
 -p 127.0.0.1:8080:8080 ghcr.io/kekzl/imp:latest --model /models/Qwen3.8-27B-NVFP4-vllm

4. Ask. Open http://localhost:8080. The built-in UI streams the answer, draws one bar per token as it arrives, and shows the server's own counts for the run: prompt tokens, prefix-cache hits, reasoning tokens, context used. If the models directory holds more than one model, the header becomes a picker and choosing an unloaded entry swaps it in.

the built-in web UI after one answer: transcript on the left, one bar per token under it, run and usage readouts on the right

_{Captured against the UI's dev mock (tools/imp-server/webui/dev/): the numbers in the picture are the mock's, not a measurement.}

Or from the shell:

curl -s http://localhost:8080/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{"model":"Qwen3.8-27B-NVFP4-vllm","messages":[{"role":"user","content":"Why is the sky blue?"}],"max_tokens":64}'

Expected: a JSON completion object whose choices[0].message.content explains Rayleigh scattering. The model id is the file or directory basename, GET /v1/models lists it, and the field is required. A GGUF file works the same way: swap the path.

The weights take 18.3 GiB on the card and answer at ~102 tok/s, leaving ~7 GiB for the KV cache on a 32 GB 5090.

[PROV: commit=f243179c date=2026年08月31日 hw=RTX5090 model=Qwen3.8-27B-NVFP4-vllm quant=NVFP4 cuda=13.3 path=nvfp4-safetensors n=2 image=ghcr.io/kekzl/imp:latest cmd=imp-cli --model ... --prompt ... --max-tokens 128 --temperature 0 (102.8/101.9 tok/s)]

That checkpoint was made with the in-tree quantizer, and the same command builds one from any BF16 or FP8 release. Qwen3.8-27B needs it: no published NVFP4 export of it runs here, because they keep attention in FP8 and this card has no kernel for that.

scripts/stage-model.sh Qwen/Qwen3.8-27B-FP8 ~/models/my-Qwen3.8-NVFP4

28.8 GiB down, ~25 minutes of conversion, 18.8 GiB out. The FP8 release is a third of the BF16 download and costs 0.24 % perplexity against converting the BF16 one. The script forecasts the output size and whether it fits the card before writing anything, so a model that would not fit costs seconds rather than half an hour. Details and the quality numbers: docs/quantization.md.

Same card, same GGUF, same flags, decode tok/s. imp v0.33.0 against llama.cpp (image pinned by digest, ghcr.io/ggml-org/llama.cpp@sha256:c49f4d48...), measured 2026年08月30日 on one RTX 5090:

[PROV: commit=83cb5178 date=2026年08月30日 hw=RTX5090 model=six-model-sweep quant=per-row cuda=13.3 path=gguf cmd=make bench-competitive n=6x2 note2=sixth row Gemma-4-26B-A4B UD-Q4_K_M 245.0 vs 214.4, +14 %, omitted here for length note=imp defaults vs llama.cpp defaults, full offload, flash attention on]

The last row is the honest bottom of the range, not an outlier we forgot to delete: on a MoE that is already bandwidth-bound at batch 1 there is little left to win. imp's default enables n-gram speculation, which is most of the Qwen3-8B figure; with it off that row reads 284.6, still +78 %.

The 32 GB is the other half of the story. NVFP4 KV means a 27B model holds 126 432 tokens of context on this card, where 8-bit KV stops at 86 848. At a 77k prompt it decodes 74.3 tok/s, or 100.2 with sparse decode switched on.

[PROV: commit=3921547d date=2026年08月30日 hw=RTX5090 model=Qwen3.8-27B-NVFP4 quant=NVFP4 cuda=13.3 path=nvfp4-kv cmd=imp-server --max-batch 8 + a client that caps max_tokens so both arms emit the same count n=3x3 note=sparse arm adds --set attention.sparse_topk_tokens=8192; it trades retrieval accuracy for the speed, see docs/MODELS.md]

Serving on one card. The same checkpoint served by imp and by vLLM (0.27.1, Marlin NVFP4) to one client, fresh server per arm, three alternating trials, aggregate tok/s as the median of three waves. Qwen3.8-27B (a Gated-DeltaNet hybrid) unless noted; the dense row is where vLLM leads:

[PROV: commit=a44298cb date=2026年09月02日 hw=RTX5090 model=Qwen3.8-27B-NVFP4-vllm quant=NVFP4 cuda=13.3 path=server-api cmd=tools/analysis/vllm_conc_ab.sh n=3 trials x 3 waves per arm, alternating; vLLM v0.27.1 with --gpu-memory-utilization 0.90 --max-model-len 16384 --max-num-seqs N; imp --set runtime.max_batch_size=32 --set runtime.max_seq_len=4096 --set kv_cache.max_blocks=2387; vLLM 0.28.0 reads 1410.7 at 32 streams; dense rows 2026年09月03日 imp 6281802f (attention.paged_fp8_multitok=4), model=Qwen3-14B-NVFP4 quant=NVFP4-Modelopt, --set kv_cache.max_blocks=8192]

Per-model history with dates, commits and exact commands: docs/BENCHMARKS.md. Methodology and what counts as a number at all: docs/PERF.md.

The table above is a sweep. The number the regression gate pins on every push is a different one, on one model:

[PROV: commit=unknown date=2026年07月26日 hw=RTX5090 model=Qwen3-8B-Q8_0 quant=Q8_0 cuda=unknown path=gguf-dp4a cmd=make verify-fast n=5x5]

Decode on this host moves several percent between sessions with nothing changed, and prefill moves more (the MoE path, not cuBLAS); the measurements behind that are in docs/PERF.md. That is why the thresholds are 8 % and not 3 %.

Full matrix with per-item status: docs/FEATURES.md. ✅ = code path plus a gated test, 🟡 = code path, no test.

The five that should decide against imp, in plain terms. The rest are in docs/LIMITATIONS.md.

1. One GPU, one chip. No multi-GPU, no tensor parallelism, and no plan for either: consumer Blackwell has no NVLink, so TP is net-negative on the workload imp targets (why).
2. No GPU in continuous integration. Every kernel correctness and performance check runs on one person's desktop before a push. If that fails your risk model, it should.
3. One model resident at a time. 32 GB fits one. Serving a second is a swap, and the requesting call pays the load.
4. Measurements move. Decode varies several percent between sessions on this host; prefill varies more. Any single number, including ours, is one sample.
5. Single-author project. No support rotation, no SLO, no security response process.

A request arrives at the HTTP layer and becomes a Request on the scheduler, which runs continuous batching: it admits as many requests as the KV budget allows and steps them together rather than serving them one at a time.

Before a new request generates anything, its prompt has to be read. That is prefill, and it is compute-bound: thousands of tokens go through the model at once, so the GEMMs are large and the GPU is busy. If an earlier request shares a prefix with this one, the prefix cache hands over the already-computed key/value pairs and prefill only runs on the tail, which is why a multi-turn agent loop gets cheaper as it grows.

Those key/value pairs live in the paged KV cache: fixed-size blocks with an index, so a sequence does not need one contiguous allocation and blocks can be shared between requests that share a prefix. Same idea as virtual memory.

Then decode starts, one token at a time, and the picture inverts. There is only one token to process, so the arithmetic is trivial and the cost is entirely reading the weights out of memory. Decode is bandwidth-bound, which is the single most important fact about this engine: making decode faster means moving fewer bytes, not doing less maths. That is why the weights are stored in 4 bits (NVFP4) and read by kernels that multiply them without unpacking them to 16 bits first. It is also why CUDA graphs matter, replaying a recorded sequence of kernel launches instead of re-issuing hundreds of them per token.

The generated token is streamed back as it appears, not after the reply is done.

That is the whole loop: request → scheduler → prefill (compute-bound) → paged KV → decode (bandwidth-bound) → stream.

Everything above this line is the five-minute version. The real thing is in docs/internals/.

Tracks main rather than the latest release.

git clone https://github.com/kekzl/imp.git && cd imp
docker compose build imp-server
docker run --gpus all -v ./models:/models -v imp-cache:/home/imp/.cache/imp \
 -p 127.0.0.1:8080:8080 \
 imp:latest --model /models/your-model.gguf

Contributor workflow, build targets and the test lanes: CONTRIBUTING.md.

Every line of imp was written by an AI coding agent (Claude Code), across ~138k lines of engine C++/CUDA plus tooling and tests. The repository keeps its own audit trail of that: what was measured, what was refuted, and what was shelved, in docs/audit/ and docs/MISSION_JOURNAL.md.

MIT. See LICENSE. One kernel file is adapted from Apache-2.0 code (SageAttention); its licence text and notice are in THIRD_PARTY_LICENSES.md, shipped in the image at /usr/share/doc/imp/.