zg index crashes: ggml-cuda.cu:98 CUDA error on RTX 4090 / Windows, while identical direct node-llama-cpp calls work

Status: root cause identified, workaround verified. (Details in the comment below; this body is the full structured report.)

Summary

On Windows 11 + RTX 4090, zg index with local/qwen3-embedding-0.6B always dies during the model "preparing" stage with a fatal, uncatchable CUDA error. The same bundled node-llama-cpp, driven directly with identical options, works end-to-end. Root cause: the VRAM-based auto-parallelism formula assumes ~150 MB per embedding context, but for a 152K-vocab model at contextSize 8192 each context's compute/logits buffer is ~5 GB (allocated lazily at first inference). With 8 parallel contexts the first inference demands ~40 GB from 24 GB VRAM and the CUDA failure kills the process instead of reaching any of the existing CPU-fallback paths.

Environment

• Windows 11 (10.0.26200) x64, RTX 4090 (compute capability 8.9), NVIDIA driver 616.56
• Node v24.16.0, zvec-grep 0.2.1 (npm global), bundled node-llama-cpp 3.18.1
• Model: local/qwen3-embedding-0.6B (Q8_0 GGUF from the local model cache)

How it was found

1. zg index --rebuild --embedding local/qwen3-embedding-0.6b failed 6/6 times with:

   D:\a\node-llama-cpp\node-llama-cpp\llama\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:98: CUDA error
   

   Note there is no error string after "CUDA error" — cudaGetErrorString() appears to return empty, and the process exits abnormally (exit code 127 under MSYS bash).
2. Every obvious suspect was eliminated first (each in isolation): reboot after NVIDIA driver update, --device cuda|cpu|vulkan (CLI flag and zg config model set), --mode direct|server|auto, --embedding-concurrency 1, fresh-binary downloads (none), native-module conflict (loading the @zvec/zvec binding before llama.cpp in one process is fine).
3. A standalone script importing the same bundled node-llama-cpp copy with zg's exact getLlama options succeeds on GPU end-to-end (~2 s, 1024-dim embedding of Chinese text), including createEmbeddingContext({contextSize: 8192 | 32768 | undefined}).
4. stderr instrumentation inside llama-cpp.js (probes around getLlama / loadModel / createEmbeddingContexts) localized the crash: all 8 contexts are created OK; the process dies on the first embedding inference.

Root cause

LlamaCppEmbeddingModel.resolveParallelism():

Math.max(1, Math.min(DEFAULT_PARALLELISM_CAP /* 8 */, Math.floor((freeMb * 0.25) / 150)))

• Assumes 150 MB per context. With ~21 GB free VRAM this yields parallelism = 8.
• For a 152K-vocab embedding model at contextSize: 8192, each context's logits/compute buffer is roughly 8192 ×ばつ 152K ×ばつ 4B ≈ 5 GB (allocated lazily at first inference — which is why context creation succeeds and everything looks fine until indexing starts).
• 8 contexts ≈ 40 GB demanded from 24 GB VRAM → CUDA allocation failure at first inference → fatal ggml-cuda.cu:98 error (empty message) that bypasses every CPU-fallback try/catch in loadLlamaWithFallback / loadModelWithFallback / createEmbeddingContextsWithFallback.

The 150 MB assumption may be reasonable for small static models (model2vec), but not for large-vocab transformer embedding models at 8K context.

Repro

zg index --rebuild --embedding local/qwen3-embedding-0.6b
# → dies at "Preparing local/qwen3-embedding-0.6b" with ggml-cuda.cu:98 CUDA error, every time

Any large-vocab local embedding model (qwen3-embedding family) on a GPU with >~6 GB free should reproduce it — the formula keeps adding contexts until the cap.

Workaround (verified)

ZVEC_GREP_LLAMA_CONTEXT_PARALLELISM=1 (the advanced override documented in zg help):

ZVEC_GREP_LLAMA_CONTEXT_PARALLELISM=1 zg index --rebuild --embedding local/qwen3-embedding-0.6b
# → index completed: 1652 files / 3648 entities / 1m34s on RTX 4090, zero failures

--embedding-concurrency does not help — it only throttles the task queue, not the number of llama.cpp contexts.

Suggested fixes

1. Factor vocab size (logits buffer ≈ contextSize ×ばつ vocab ×ばつ 4B) into the per-context VRAM estimate before deriving parallelism.
2. Make the CUDA failure catchable so the existing CPU-fallback paths can engage instead of the process dying.
3. Consider surfacing ZVEC_GREP_LLAMA_CONTEXT_PARALLELISM more prominently (docs/help) as the remedy for large-vocab embedding models.