kvcache.ai

SGLang is a fast serving framework for large language models and vision language models.

AgentENV (AENV) is a distributed platform for running agent environments at scale.

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A high-throughput and memory-efficient inference and serving engine for LLMs

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

AFD-Ledger is an analytical toolkit for provisioning and evaluating attention–FFN disaggregated LLM inference deployments.