Where On-Device and Cloud LLMs Meet

Table of Contents

Setup

git clone https://github.com/HazyResearch/minions.git
cd minions
pip install -e . # installs the minions package in editable mode

pip install -e ".[mlx]"

pip install -e ".[secure]"

pip install tokasaurus

# OpenAI
export OPENAI_API_KEY=<your-openai-api-key>
export OPENAI_BASE_URL=<your-openai-base-url> # Optional: Use a different OpenAI API endpoint
# Together AI
export TOGETHER_API_KEY=<your-together-api-key>
# OpenRouter
export OPENROUTER_API_KEY=<your-openrouter-api-key>
export OPENROUTER_BASE_URL=<your-openrouter-base-url> # Optional: Use a different OpenRouter API endpoint
# Perplexity
export PERPLEXITY_API_KEY=<your-perplexity-api-key>
export PERPLEXITY_BASE_URL=<your-perplexity-base-url> # Optional: Use a different Perplexity API endpoint
# Tokasaurus
export TOKASAURUS_BASE_URL=<your-tokasaurus-base-url> # Optional: Use a different Tokasaurus API endpoint
# DeepSeek
export DEEPSEEK_API_KEY=<your-deepseek-api-key>
# Anthropic
export ANTHROPIC_API_KEY=<your-anthropic-api-key>
# Mistral AI
export MISTRAL_API_KEY=<your-mistral-api-key>
# MiniMax
export MINIMAX_API_KEY=<your-minimax-api-key>
# Gemini
export GEMINI_API_KEY=<your-gemini-api-key>
# Groq
export GROQ_API_KEY=<your-groq-api-key>
# Grok (xAI)
export XAI_API_KEY=<your-xai-api-key>
# Cohere
export COHERE_API_KEY=<your-cohere-api-key>

Minions Demo Application

pip install torch transformers
streamlit run app.py

An error occurred: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download

OLLAMA_FLASH_ATTENTION=1 ollama serve

Minions WebGPU App

Features

Quick Start

Example code: Minion (singular)

from minions.clients.ollama import OllamaClient
from minions.clients.openai import OpenAIClient
from minions.minion import Minion
local_client = OllamaClient(
 model_name="llama3.2",
 )
remote_client = OpenAIClient(
 model_name="gpt-4o",
 )
# Instantiate the Minion object with both clients
minion = Minion(local_client, remote_client)
context = """
Patient John Doe is a 60-year-old male with a history of hypertension. In his latest checkup, his blood pressure was recorded at 160/100 mmHg, and he reported occasional chest discomfort during physical activity.
Recent laboratory results show that his LDL cholesterol level is elevated at 170 mg/dL, while his HDL remains within the normal range at 45 mg/dL. Other metabolic indicators, including fasting glucose and renal function, are unremarkable.
"""
task = "Based on the patient's blood pressure and LDL cholesterol readings in the context, evaluate whether these factors together suggest an increased risk for cardiovascular complications."
# Execute the minion protocol for up to two communication rounds
output = minion(
 task=task,
 context=[context],
 max_rounds=2
)

Example Code: Minions (plural)

from minions.clients.ollama import OllamaClient
from minions.clients.openai import OpenAIClient
from minions.minions import Minions
from pydantic import BaseModel
class StructuredLocalOutput(BaseModel):
 explanation: str
 citation: str | None
 answer: str | None
local_client = OllamaClient(
 model_name="llama3.2",
 temperature=0.0,
 structured_output_schema=StructuredLocalOutput
)
remote_client = OpenAIClient(
 model_name="gpt-4o",
)
# Instantiate the Minion object with both clients
minion = Minions(local_client, remote_client)
context = """
Patient John Doe is a 60-year-old male with a history of hypertension. In his latest checkup, his blood pressure was recorded at 160/100 mmHg, and he reported occasional chest discomfort during physical activity.
Recent laboratory results show that his LDL cholesterol level is elevated at 170 mg/dL, while his HDL remains within the normal range at 45 mg/dL. Other metabolic indicators, including fasting glucose and renal function, are unremarkable.
"""
task = "Based on the patient's blood pressure and LDL cholesterol readings in the context, evaluate whether these factors together suggest an increased risk for cardiovascular complications."
# Execute the minion protocol for up to two communication rounds
output = minion(
 task=task,
 doc_metadata="Medical Report",
 context=[context],
 max_rounds=2
)

Python Notebook

Docker support

Build the Docker Image

docker build -t minions-docker .

Run the Container

# Basic usage (includes Ollama service)
docker run -i minions-docker
# With Docker socket mounted (for Docker Model Runner)
docker run -i -v /var/run/docker.sock:/var/run/docker.sock minions-docker
# With API keys for remote models
docker run -i -e OPENAI_API_KEY=your_key -e ANTHROPIC_API_KEY=your_key minions-docker
# With custom Ollama host
docker run -i -e OLLAMA_HOST=0.0.0.0:11434 minions-docker
# For Streamlit app (legacy usage)
docker run -p 8501:8501 --env OPENAI_API_KEY=<your-openai-api-key> --env DEEPSEEK_API_KEY=<your-deepseek-api-key> minions-docker

Docker Minion Protocol Usage

{
 "local_client": {
 "type": "ollama",
 "model_name": "llama3.2:3b",
 "port": 11434,
 "kwargs": {}
 },
 "remote_client": {
 "type": "openai",
 "model_name": "gpt-4o",
 "kwargs": {
 "api_key": "your_openai_key"
 }
 },
 "protocol": {
 "type": "minion",
 "max_rounds": 3,
 "log_dir": "minion_logs",
 "kwargs": {}
 },
 "call_params": {
 "task": "Your task here",
 "context": ["Context string 1", "Context string 2"],
 "max_rounds": 2
 }
}

Usage Examples

echo '{
 "local_client": {
 "type": "ollama",
 "model_name": "llama3.2:3b"
 },
 "remote_client": {
 "type": "openai",
 "model_name": "gpt-4o"
 },
 "protocol": {
 "type": "minion",
 "max_rounds": 3
 },
 "call_params": {
 "task": "Analyze the patient data and provide a diagnosis",
 "context": ["Patient John Doe is a 60-year-old male with hypertension. Blood pressure: 160/100 mmHg. LDL cholesterol: 170 mg/dL."]
 }
}' | docker run -i -e OPENAI_API_KEY=$OPENAI_API_KEY minions-docker

echo '{
 "local_client": {
 "type": "ollama",
 "model_name": "llama3.2:3b"
 },
 "remote_client": {
 "type": "openai",
 "model_name": "gpt-4o"
 },
 "protocol": {
 "type": "minions"
 },
 "call_params": {
 "task": "Analyze the financial data and extract key insights",
 "doc_metadata": "Financial report",
 "context": ["Revenue increased by 15% year-over-year. Operating expenses rose by 8%. Net profit margin improved to 12%."]
 }
}' | docker run -i -e OPENAI_API_KEY=$OPENAI_API_KEY minions-docker

Supported Client Types

Supported Protocol Types

Output Format

{
 "success": true,
 "result": {
 "final_answer": "The analysis result...",
 "supervisor_messages": [...],
 "worker_messages": [...],
 "remote_usage": {...},
 "local_usage": {...}
 },
 "error": null
}

{
 "success": false,
 "result": null,
 "error": "Error message"
}

Environment Variables

Tips and Advanced Usage

docker run -i \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v $(pwd)/logs:/app/minion_logs \
 -v $(pwd)/workspace:/app/workspace \
 -e OPENAI_API_KEY=$OPENAI_API_KEY \
 minions-docker

docker run -it \
 -e OPENAI_API_KEY=$OPENAI_API_KEY \
 minions-docker bash

python minion_stdin_interface.py 

Persistent Container Usage

docker run -d --name minions-container -e OPENAI_API_KEY="$OPENAI_API_KEY" minions-docker

echo '{
 "local_client": {"type": "ollama", "model_name": "llama3.2:3b"},
 "remote_client": {"type": "openai", "model_name": "gpt-4o"},
 "protocol": {"type": "minion", "max_rounds": 1},
 "call_params": {"task": "How many times did Roger Federer end the year as No.1?"}
}' | docker exec -i minions-container /app/start_minion.sh

echo '{
 "local_client": {"type": "ollama", "model_name": "llama3.2:3b"},
 "remote_client": {"type": "openai", "model_name": "gpt-4o"},
 "protocol": {"type": "minion", "max_rounds": 1},
 "call_params": {"task": "What is the capital of France?"}
}' | docker exec -i minions-container /app/start_minion.sh

docker stop minions-container
docker rm minions-container

CLI

export MINIONS_LOCAL=ollama/llama3.2
export MINIONS_REMOTE=openai/gpt-4o

minions --help

minions --context <path_to_context> --protocol <minion|minions>

Secure Minions Local-Remote Protocol

Prerequisites

pip install -e ".[secure]"

Basic Usage

Python API

from minions.clients import OllamaClient
from secure.minions_secure import SecureMinionProtocol
# Initialize local client
local_client = OllamaClient(model_name="llama3.2")
# Create secure protocol instance
protocol = SecureMinionProtocol(
 supervisor_url="https://your-supervisor-server.com",
 local_client=local_client,
 max_rounds=3,
 system_prompt="You are a helpful AI assistant."
)
# Run a secure task
result = protocol(
 task="Analyze this document for key insights",
 context=["Your document content here"],
 max_rounds=2
)
print(f"Final Answer: {result['final_answer']}")
print(f"Session ID: {result['session_id']}")
print(f"Log saved to: {result['log_file']}")
# Clean up the session
protocol.end_session()

Command Line Interface

python secure/minions_secure.py \
 --supervisor_url https://your-supervisor-server.com \
 --local_client_type ollama \
 --local_model llama3.2 \
 --max_rounds 3

Secure Minions Chat

pip install -e ".[secure]"

Apps

Inference Estimator

Command Line Usage

python -m minions.utils.inference_estimator --model llama3.2 --tokens 1000 --describe

Python API Usage

from minions.utils.inference_estimator import InferenceEstimator
# Initialize the estimator for a specific model
estimator = InferenceEstimator(
 model_name="llama3.2", # Model name
 is_quant=True, # Is model quantized?
 quant_bits=4 # Quantization level (4, 8, 16)
)
# Estimate performance for 1000 tokens
tokens_per_second, estimated_time = estimator.estimate(1000)
print(f"Estimated speed: {tokens_per_second:.1f} tokens/sec")
print(f"Estimated time: {estimated_time:.2f} seconds")
# Get detailed stats
detailed_info = estimator.describe(1000)
print(detailed_info)
# Calibrate with your actual model client for better accuracy
# (requires a model client that implements a chat() method)
estimator.calibrate(my_model_client, sample_tokens=32, prompt="Hello")

Miscellaneous Setup

Using Azure OpenAI with Minions

Set Environment Variables

export AZURE_OPENAI_API_KEY=your-api-key
export AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com/
export AZURE_OPENAI_API_VERSION=2024年02月15日-preview

Example Code

from minions.clients.ollama import OllamaClient
from minions.clients.azure_openai import AzureOpenAIClient
from minions.minion import Minion
local_client = OllamaClient(
 model_name="llama3.2",
)
remote_client = AzureOpenAIClient(
 model_name="gpt-4o", # This should match your deployment name
 api_key="your-api-key",
 azure_endpoint="https://your-resource-name.openai.azure.com/",
 api_version="2024-02-15-preview",
)
# Instantiate the Minion object with both clients
minion = Minion(local_client, remote_client)

Maintainers