High-speed Large Language Model Serving for Local Deployment
-
Updated
Aug 2, 2025 - C++
#
local-inference
Here are 12 public repositories matching this topic...
Sort: Most stars
Sort options
[ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration
-
Updated
Nov 18, 2024 - Python
Tool for test diferents large language models without code.
api-rest chat-application flask-api inference-api large-language-models llm local-inference llamacpp llm-inference llama2 llama-cpp-python llama2-7b mixtral-8x7b
-
Updated
Jul 16, 2025 - Python
Local LLM Inference Library
-
Updated
Jan 30, 2025 - Pascal
LLM chatbot example using OpenVINO with RAG (Retrieval Augmented Generation).
natural-language-processing offline chatbot intel edge-computing rag openvino huggingface edge-inference cloud-free llm local-inference langchain dolly2 retrieval-augmented-generation llama2 neural-chat
-
Updated
Jan 25, 2024 - Python
Study Buddy is a desktop application that provides AI tutoring without requiring internet access or accounts
electron javascript css typescript desktop-application privacy-focused local-inference ai-tutor offline-application
-
Updated
Jun 25, 2025 - TypeScript
Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant
desktop-app rust productivity machine-learning opensource privacy cross-platform offline chatbot svelte artificial-intelligence candle conversational-ai tauri ai-chat llm local-inference tauri-rust local-llm qwen
-
Updated
Oct 14, 2025 - Rust
Lightweight 6GB VRAM Gradio web app with auto-installer for running AuraFlow locally — no cloud, no clutter.
python open-source image-generation webui gradio text-to-image stable-diffusion diffusers local-inference generative-ai ai-image-generator auraflow low-vram
-
Updated
Jun 7, 2025 - Python
Verify claims using AI agents that debate using scraped evidence and local language models.
python scraping beautifulsoup autonomous-agents local-inference qwen lm-studio langgraph private-llm agentic-ai claim-verification phi-4-mini evidence-based-ai
-
Updated
Jun 1, 2025 - Python
script which performs RAG and use a local LLM for Q&A
-
Updated
Sep 16, 2024 - Python
Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.
-
Updated
Nov 19, 2024 - Python
Script which takes a .wav audio file, performs speech-to-text using OpenAI/Whisper, and then, using Llama3, summarization and action point from the transcript generated
summarization speech-to-text whisper meeting-minutes local-inference langchain-python llm-inference ollama
-
Updated
Sep 16, 2024 - Python
Improve this page
Add a description, image, and links to the local-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the local-inference topic, visit your repo's landing page and select "manage topics."