Artifact for "Marconi: Prefix Caching for the Era of Hybrid LLMs" [MLSys '25 Outstanding Paper Award, Honorable Mention]
-
Updated
Mar 5, 2025 - Python
#
prefix-caching
Here are 10 public repositories matching this topic...
Unified execution runtime for LLM and ML programs.
machine-learning deep-learning transformers pytorch agents execution-engine kv-cache dataflow-graph llm generative-ai ai-runtime workflow-optimization program-optimization prefix-caching agent-runtime llm-runtime compiler-runtime cross-call-caching
-
Updated
May 1, 2026 - C++
Optimized vLLM setup for Gemma 4 31B NVFP4 with MTP on dual RTX PRO 6000 Blackwell using vllm and docker: native FP4 Tensor Cores, Multi-Token Prediction (96.5% acceptance rate), and prefix caching. Includes benchmark results and replication scripts.
docker amd cuda gemma blackwell vllm llm-inference am5 speculative-decoding fp4 prefix-caching multi-token-prediction nvfp4 rtx-6000 gemma4 tensor-parallel
-
Updated
May 10, 2026 - Shell
C++ inference runtime for llama.cpp that shares a single document KV-cache prefill across multiple analytical branches via snapshot fan-out. Eliminates redundant GPU compute and dramatically reduces TTFT in DAG-based multi-agent pipelines.
-
Updated
Jun 10, 2026 - C++
Cache-aware router for OpenAI-compatible LLM servers, in Go. Per-worker radix trees route each request to the worker holding its KV prefix. Validated on 4x A100 + vLLM and Apple Silicon + llama.cpp.
-
Updated
May 24, 2026 - Go
A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.
metal gpu-acceleration systems-programming apple-silicon openai-api tensor-parallelism llm-inference speculative-decoding paged-attention continuous-batching prefix-caching disaggregated-serving
-
Updated
Jun 13, 2026 - Rust
Context engineering toolkit for LLMs — pack, cache, debug, red-team, and orchestrate context windows. Council of Experts, adversarial testing, immune system, context compiler, drift detection, multi-agent entanglement. TypeScript + Python.
python typescript ai multi-agent rag llm prompt-engineering llm-tools context-window prefix-caching context-engineering adversarial-testing token-budget council-of-experts context-packing
-
Updated
Jun 12, 2026 - Python
Benchmarking LLM prefix-cache eviction policies against Tree-Constrained Belady on ShareGPT traces.
radix-tree lru-cache page-replacement-algorithm kv-cache llm-serving vllm llm-inference sglang prefix-caching
-
Updated
May 2, 2026 - HTML
Reproducible Qwen3-1.7B prefix cache benchmark on RTX 3070 Laptop (8GB). Hand-written reference inference loop + vLLM v1 comparison.
-
Updated
May 17, 2026
Edge-optimized OpenCUA-7B computer-use agent evaluated on OSWorld, exploring systematic vLLM inference optimizations across CPU and GPU, including precision tuning, image history management, speculative decoding, and prefix caching.
quantization agents multimodal inference-optimization edge-ai vllm speculative-decoding gui-agents prefix-caching osworld opencua
-
Updated
Dec 18, 2025 - Python
Improve this page
Add a description, image, and links to the prefix-caching topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the prefix-caching topic, visit your repo's landing page and select "manage topics."