System Architecture: Deterministic Claim-Level Halting for LLM Hallucinations using Rust and Dual-Entropy Scoring

DEV Community

Top comments (2)

Embedded multi-modal database for edge AI

Joined

Feb 27, 2026

The micro-batch buffering before scoring is the critical design choice here. Every token you buffer before checking adds latency — at 21+ TPS on CPU (like Mixtral), a 3-token buffer costs ~140ms before the user sees anything. Have you benchmarked the latency impact of different buffer sizes?

The dual-entropy approach (NLI + RAG) is solid, but I'm curious about the NLI model choice. FactCG-DeBERTa at 0.4B is lightweight for real-time scoring, but NLI-based contradiction detection has known blind spots: it struggles with implicit contradictions (things the model "knows" but didn't state explicitly). A smaller entainment classifier might miss subtle errors.

On the Rust side — 12 hot paths with 9.4x speedup over Python is impressive. Are you using SIMD for the entropy calculations, or is the gain mainly from avoiding Python's GC overhead in the streaming loop?

anulum profile image

Miroslav Šotek

https://independent.academia.edu/fortisstudio

Location

Marbach SG, Switzerland
Joined

Mar 29, 2026

• Jun 22 • Edited on Jun 22 • Edited

Great questions — let me take them in order, with a bit of nuance where the post oversimplified.

1 - Buffering / granularity. One correction worth making: the production halt is claim-level, not strictly per-token. Tokens stream through; the proxy scores the buffered text on a fixed cadence (default every 8 tokens, configurable), and the halt fires on a completed claim that contradicts the governed premise — not on each token. So you're not paying an NLI forward pass per token, and the tunable isn't "buffer N before first byte"; it's how early a contradicting claim is caught vs. scoring overhead. There's a separate token-level span detector for when you want sub-claim resolution. I haven't published a buffer-size sweep — fair ask, I'll add one.

2 - NLI choice / blind spots. You're right that NLI misses implicit, world-knowledge contradictions — I won't pretend otherwise. Two design points that address it:

The halt model isn't the 0.4B FactCG. FactCG is the grounding model and it's 2-class (supported / not-supported), with no contradiction class. The halt uses a 3-class NLI (DeBERTa-v3-large MNLI/FEVER/ANLI) for raw P(contradiction), and it halts only on contradiction — never on "neutral / unsupported." A correct-but-unstated claim is neutral, so it doesn't false-halt; that's the exact failure mode a 2-class grounding model has on streaming text.

For implicit contradictions NLI can't see at inference time, the answer is RAG grounding (put the fact in the premise) plus optional escalation to a stronger judge on low-confidence scores. NLI is one signal, not the whole gate.

3 - Rust gains — SIMD or GC? Neither, honestly. No SIMD in the current kernel — the gains come from moving the hot loops (BM25, the divergence/entropy reductions, softmax, aggregation) into native Rust over PyO3 and out of the Python per-call/interpreter overhead in the streaming loop, plus lock-based concurrency (parking_lot) instead of fighting the GIL. Small correction to the framing: CPython is refcount + cyclic GC, so what we avoid is interpreter/dispatch + GIL contention, not Java-style GC pauses. Per-path numbers (Python-vs-Rust medians + checksum parity) are in BENCHMARKS.md; the geomean has moved since the v3.14 figure in the post — I owe that an update.

KR, Miroslav