LLMProxyAS

Table of contents

Features

How it works

┌──────────────┐ OpenAI HTTP API ┌──────────────┐ HTTPS ┌─────────────────┐
│ Your app │ ───────────────────► │ LLMProxyAS │ ───────────► │ Upstream LLM │
│ SDK / curl │ :20182/v1/... │ (this repo) │ │ OpenAI / Groq... │
└──────────────┘ └──────────────┘ └─────────────────┘
 │
 ┌───────────────────────┼───────────────────────┐
 │ │ │
 model allow-list API key injection timeouts / SSE
 (local filter) (config or client) logging / recovery

Requirements

Quick start

1. Clone and configure

git clone https://github.com/dev2k6/LLMProxyAS.git
cd LLMProxyAS
cp configs/config.example.json configs/config.json

{
 "base_url": "https://api.openai.com/v1",
 "api_key": "sk-your-real-upstream-key",
 "port": 20182,
 "log_level": "info"
}

2. Run

go mod tidy
go run ./cmd/server
# or: make run

docker compose up -d --build
# or: make docker-up

3. Verify

curl -s http://localhost:20182/health
curl -s http://localhost:20182/v1/models

{
 "status": "ok",
 "service": "LLMProxyAS"
}

4. First chat call

curl -s http://localhost:20182/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [{"role": "user", "content": "Hello from LLMProxyAS"}]
 }'

Authentication

Configuration

CLI flags

configs/config.json

{
 "base_url": "https://api.openai.com/v1",
 "api_key": "sk-xxxxxxxxxxxxxxxx",
 "timeout_seconds": 120,
 "port": 20182,
 "log_level": "info",
 "log_body": false,
 "max_body_bytes": 33554432,
 "response_header_timeout_seconds": 60,
 "stream_idle_timeout_seconds": 120,
 "stream_max_seconds": 600,
 "read_header_timeout_seconds": 10,
 "idle_timeout_seconds": 120
}

configs/allowed_models.json

{
 "models": [
 "gpt-4o",
 "gpt-4o-mini",
 "gpt-4-turbo",
 "claude-3-5-sonnet-20241022",
 "deepseek-chat",
 "qwen2.5-72b-instruct"
 ]
}

Provider examples

{
 "base_url": "https://api.openai.com/v1",
 "api_key": "sk-..."
}

{
 "base_url": "https://api.groq.com/openai/v1",
 "api_key": "gsk_..."
}

{
 "base_url": "https://api.deepseek.com/v1",
 "api_key": "sk-..."
}

{
 "base_url": "http://127.0.0.1:8000/v1",
 "api_key": "ollama"
}

Docker Compose environment

HTTP API

Client examples

Python (OpenAI SDK)

from openai import OpenAI
client = OpenAI(
 base_url="http://localhost:20182/v1",
 api_key="not-needed",
)
resp = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[{"role": "user", "content": "Count to five"}],
 stream=True,
)
for chunk in stream:
 if chunk.choices and chunk.choices[0].delta.content:
 print(chunk.choices[0].delta.content, end="", flush=True)

Tools / function calling

tools = [{
 "type": "function",
 "function": {
 "name": "get_weather",
 "description": "Get weather by city",
 "parameters": {
 "type": "object",
 "properties": {"city": {"type": "string"}},
 "required": ["city"],
 },
 },
}]
resp = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[{"role": "user", "content": "Weather in Hanoi?"}],
 tools=tools,
)
print(resp.choices[0].message)

Vision (image URL)

resp = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": [
 {"type": "text", "text": "What is in this image?"},
 {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
 ],
 }],
)

Node.js

import OpenAI from "openai";
const client = new OpenAI({
 baseURL: "http://localhost:20182/v1",
 apiKey: "not-needed",
});
const resp = await client.chat.completions.create({
 model: "gpt-4o-mini",
 messages: [{ role: "user", content: "Hello" }],
});
console.log(resp.choices[0].message.content);

cURL

# Models
curl -s http://localhost:20182/v1/models | jq .
# Chat (key optional if configured server-side)
curl -s http://localhost:20182/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [{"role": "user", "content": "Hi"}]
 }' | jq .
# Stream
curl -N http://localhost:20182/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "stream": true,
 "messages": [{"role": "user", "content": "Say hello slowly"}]
 }'

Rejected model (example)

curl -s http://localhost:20182/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{"model":"not-allowed","messages":[{"role":"user","content":"x"}]}'

{
 "error": {
 "message": "model 'not-allowed' is not allowed by this proxy; see GET /v1/models for permitted models",
 "type": "invalid_request_error",
 "param": "model",
 "code": "model_not_allowed"
 }
}

Docker

Compose (recommended)

# Configure configs/config.json and allowed_models.json first
docker compose up -d --build
docker compose ps
docker compose logs -f
curl -s http://localhost:20182/health
docker compose down

# bash
HOST_PORT=8080 docker compose up -d --build
# PowerShell
$env:HOST_PORT = "8080"; docker compose up -d --build

Build & run image

docker build -t llmproxyas:latest .
# Linux / macOS
docker run --rm -p 20182:20182 \
 -v "$(pwd)/configs:/app/configs:ro" \
 llmproxyas:latest
# Windows PowerShell
docker run --rm -p 20182:20182 `
 -v "${PWD}/configs:/app/configs:ro" `
 llmproxyas:latest

Docker files

Prebuilt binaries

cd bin
.\LLMProxyAS.exe

powershell -ExecutionPolicy Bypass -File .\scripts\build-all.ps1

https://github.com/dev2k6/LLMProxyAS/raw/main/bin/LLMProxyAS-windows-amd64.exe
https://github.com/dev2k6/LLMProxyAS/raw/main/bin/LLMProxyAS-linux-amd64
https://github.com/dev2k6/LLMProxyAS/raw/main/bin/LLMProxyAS-darwin-arm64

Build from source

go mod tidy
# Linux / macOS
go build -o bin/LLMProxyAS ./cmd/server
./bin/LLMProxyAS -config configs/config.json -models configs/allowed_models.json
# Windows
go build -o bin/LLMProxyAS.exe ./cmd/server
.\bin\LLMProxyAS.exe -config configs\config.json -models configs\allowed_models.json

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \
 go build -trimpath -ldflags="-s -w" -o bin/LLMProxyAS ./cmd/server

powershell -ExecutionPolicy Bypass -File .\scripts\build-all.ps1

Makefile targets

Project layout

.
├── cmd/server/ # Process entrypoint (flags, exit codes)
├── internal/
│ ├── app/ # DI, lifecycle, config path resolution
│ ├── config/ # config.json + allowed_models loaders
│ ├── handler/ # Proxy, models, health handlers
│ ├── httpserver/ # Gin router + http.Server wrapper
│ ├── logging/ # slog JSON logger
│ ├── middleware/ # Recovery, request log, model filter
│ └── oaierr/ # OpenAI error JSON helpers
├── configs/
│ ├── config.example.json # Safe template for distribution
│ ├── config.json # Local runtime config (keep secrets out of git)
│ └── allowed_models.json # Model allow-list
├── Dockerfile
├── docker-compose.yml
├── Makefile
├── LICENSE
├── CHANGELOG.md
├── CONTRIBUTING.md
├── SECURITY.md
└── README.md

Resilience

Logging

{
 "time": "2026年07月19日T12:00:00Z",
 "level": "INFO",
 "msg": "request",
 "method": "POST",
 "path": "/v1/chat/completions",
 "status": 200,
 "latency_ms": 342,
 "model": "gpt-4o-mini",
 "client_ip": "127.0.0.1"
}

Error format

{
 "error": {
 "message": "human-readable message",
 "type": "invalid_request_error",
 "param": "model",
 "code": "model_not_allowed"
 }
}

Security

Troubleshooting

"log_level": "debug"

Roadmap

Contributing

Changelog

License