Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.
Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.
RWKV linear attention batch inference server β auto-queue, dynamic capacity, OpenAI compatible.
A Go-based LLM serving control plane that models token-aware scheduling, request lifecycle, streaming metrics, prefix cache, and KV block pressure around mock inference backends.
A high-performance deep learning model inference server based on TensorRT, supporting fast inference for Embedding, Reranker, and NLI models.
PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.
"A practical guide to NLP model optimization and serving with NVIDIA Triton." / "μ€μ΅μΌλ‘ μ΅νλ κ³ μ±λ₯ NLP λͺ¨λΈ μ΅μ ν λ° NVIDIA Triton μλ² λ°°ν¬."
Production-grade LLM inference API built from scratch. NestJS gateway + Python GPU workers. Scheduling, batching, KV cache, tensor parallelism, multi-modal β all against real GPUs.
π Process JSON data in batches with `llm-batch`, leveraging sequential or parallel modes for efficient interaction with LLMs.
A lightweight, framework-agnostic middleware that dynamically batches inference requests in real time to maximize GPU/TPU utilization.
LM Multi-Bin Dynamic Scheduler Simulator - Implementation combining Multi-Bin batching with SLA-constrained dynamic batching
Zephyros is a lock-free, zero-allocation MPSC (Multi-Producer, Single-Consumer) ring buffer for Go, engineered for applications that demand extreme throughput, predictable latency, and uncompromising efficiency under high concurrency.
Add a description, image, and links to the dynamic-batching topic page so that developers can more easily learn about it.
To associate your repository with the dynamic-batching topic, visit your repo's landing page and select "manage topics."