Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

vllm-project/semantic-router

Innovations ✨

Intelligent Routing 🧠

Auto-Reasoning and Auto-Selection of Models

An Mixture-of-Models (MoM) router that intelligently directs OpenAI API requests to the most suitable models from a defined pool based on Semantic Understanding of the request's intent (Complexity, Task, Tools).

This is achieved using BERT classification. Conceptually similar to Mixture-of-Experts (MoE) which lives within a model, this system selects the best entire model for the nature of the task.

As such, the overall inference accuracy is improved by using a pool of models that are better suited for different types of tasks:

Model Accuracy

The screenshot below shows the LLM Router dashboard in Grafana.

LLM Router Dashboard

The router is implemented in two ways: Golang (with Rust FFI based on Candle) and Python. Benchmarking will be conducted to determine the best implementation.

Auto-Selection of Tools

Select the tools to use based on the prompt, avoiding the use of tools that are not relevant to the prompt so as to reduce the number of prompt tokens and improve tool selection accuracy by the LLM.

Enterprise Security πŸ”’

PII detection

Detect PII in the prompt, avoiding sending PII to the LLM so as to protect the privacy of the user.

Prompt guard

Detect if the prompt is a jailbreak prompt, avoiding sending jailbreak prompts to the LLM so as to prevent the LLM from misbehaving.

Similarity Caching ⚑️

Cache the semantic representation of the prompt so as to reduce the number of prompt tokens and improve the overall inference latency.

Documentation πŸ“–

For comprehensive documentation including detailed setup instructions, architecture guides, and API references, visit:

πŸ‘‰ Complete Documentation at Read the Docs

The documentation includes:

Community πŸ‘‹

For questions, feedback, or to contribute, please join #semantic-router channel in vLLM Slack.

Citation

If you find Semantic Router helpful in your research or projects, please consider citing it:

@misc{semanticrouter2025,
 title={vLLM Semantic Router},
 author={vLLM Semantic Router Team},
 year={2025},
 howpublished={\url{https://github.com/vllm-project/semantic-router}},
}

AltStyle γ«γ‚ˆγ£γ¦ε€‰ζ›γ•γ‚ŒγŸγƒšγƒΌγ‚Έ (->γ‚ͺγƒͺγ‚ΈγƒŠγƒ«) /