Machine Learning/LiftWing/Large Language Models

LiftWing has been expanded to bring large language model capabilities to Wikimedia projects and beyond. The platform hosts small and medium open-source LLMs (up to roughly 30B parameters), served with vLLM on AMD GPUs.

The platform serves general-purpose open-weight LLMs for public use, alongside project-specific models for internal use. It currently focuses on single-GPU inference with horizontal scaling; models are quantized (FP8 where supported) to fit within a single GPU partition, running on the LiftWing Kubernetes / KServe infrastructure.

There are two families of models:

• General-purpose llm-* models (Qwen) are the public offering: they are reachable through the public API and appear in LiftWing Studio.
• Project-specific models (safeguard / policy-violation, embeddings, etc.) are internal only — they are not exposed publicly and do not appear in Studio.

Program pages capture what the platform offered at a given event, aimed at that event's participants. Past events are archived here as a record of how the service has evolved.

The fastest way to try a model:

1. No code: open LiftWing Studio and chat with a general-purpose model in the browser.
2. From the terminal: the public API is OpenAI-compatible, so a single curl gets you a completion.

curlhttps://api.wikimedia.org/service/lw/inference/v1/models/llm-qwen3-14b/openai/v1/chat/completions\
-H"Content-Type: application/json"\
-d'{"model": "llm-qwen3-14b", "messages": [{"role": "user", "content": "In two sentences, what is the Wikimedia Foundation?"}]}'

LiftWing Studio gives the LiftWing LLMs a user-friendly chat interface, so you can use the models in the browser instead of only through the API. Pick a model, type a prompt, and iterate interactively — no code required. It is the recommended first stop for exploring what the models can do.

Register an account on the Studio sign-in page. New accounts must be approved by an admin before you can sign in and start chatting.

• Chat history across sessions
• Document uploads
• Model selection from the available general-purpose models

Studio runs as three Docker services on a Cloud VPS instance:

The public models run on a single MI300X GPU or GPU partition; context windows are listed per model below.

Public models are served through the Wikimedia REST Gateway at api.wikimedia.org. Each model exposes an OpenAI-compatible interface, so existing OpenAI client libraries work by pointing base_url at the model's endpoint.

The supported interface is chat completions, which is OpenAI-compatible. Paths are relative to https://api.wikimedia.org , and only services named llm-* (the public Qwen models) are routed here.

The Qwen models are instruction-tuned chat models, so chat completions is the interface to use: send a messages array and the server applies the model's chat template automatically. Because it is OpenAI-compatible, any OpenAI client library works by pointing base_url at the model's endpoint.

fromopenaiimport OpenAI
client = OpenAI(
 base_url="https://api.wikimedia.org/service/lw/inference/v1/models/llm-qwen3-14b/openai/v1",
 api_key="none", # public endpoint; no key required
)
resp = client.chat.completions.create(
 model="llm-qwen3-14b",
 messages=[{"role": "user", "content": "Explain vLLM in one sentence."}],
 stream=True, # server-sent events supported
)
for chunk in resp:
 print(chunk.choices[0].delta.content or "", end="")

LLM inference services are deployed as KServe InferenceServices via the kserve-inference chart, defined in operations/deployment-charts → helmfile.d/ml-services/llm/.

Per-model deployment details, derived from helmfile.d/ml-services/{llm,experimental}/values-ml-serve-eqiad.yaml. GPU count equals the tensor-parallel size. VRAM figures are weights-only estimates — parameter count ×ばつ precision bytes (2 for BF16/FP16, 1 for FP8, 0.5 for 4-bit int4/MXFP4, using total parameters for MoE models); actual runtime use is higher (KV cache, activations, and vLLM overhead).

Public llm-* endpoints are governed by the LiftWingLLM rate-limit policy in the REST Gateway (T426749). It is a coarse, interim limit until cost/token-based limiting lands.

Running from Toolforge is the recommended way to get the higher rate limit. Toolforge is part of WMCS, so its egress traffic is classified as a known network and the shared 100 req/hour cap is lifted to effectively unlimited — no API key or extra configuration required. For a quick call you don't need to build anything; step 1 is one-time.

1. Create a Wikimedia developer account and join Toolforge. If you don't have one, create a developer account and request Toolforge membership — the Toolforge Quickstart covers both in a few minutes.

2. Log in to the Toolforge bastion:

ssh<your-username>@login.toolforge.org

3. Call the model. Run the same request you would anywhere — because it now egresses through Toolforge, it automatically gets the higher rate limit:

curlhttps://api.wikimedia.org/service/lw/inference/v1/models/llm-qwen3-14b/openai/v1/chat/completions\
-H"Content-Type: application/json"\
-d'{"model": "llm-qwen3-14b", "messages": [{"role": "user", "content": "Hello from Toolforge!"}]}'

Becoming a known client or a community-approved bot (JWT auth) also lifts the limit — contact the ML team (see below) if your use case needs it.

How your data is handled differs between the API and LiftWing Studio.

The API does not log or retain your data — the prompts you send and the responses returned are not persisted, and are not used to train or fine-tune the models. Each request is processed only to generate its response.

Unlike the API, LiftWing Studio stores your conversations by default:

• Saved chats: by default, chats are saved in Studio's database, so your history is available across sessions.
• Temporary chats: Studio offers temporary chats, where nothing is persisted in the database — use these if you don't want your prompts stored.
• Sharing: chats can be shared with other users via the share button.

GPU and vLLM serving metrics are collected in Grafana. The work-in-progress ML infrastructure dashboard covers two layers — infrastructure (GPU compute & capacity, VRAM, system metrics) and service (Kubernetes deployments, vLLM). All LLM inference services now emit vLLM Prometheus metrics (T431136).

An initial performance baseline has been recorded for llm-qwen36-27b (FP8) on the internal endpoint, measured with vllm-bench.

Key takeaways:

• Single-stream (concurrency 1): ~197 ms to first token, ~35 output tokens/s, ~3.6 s for a 128-token response.
• Throughput saturates around concurrency 64 (~410 output tok/s, ~4.2 req/s); scaling is sub-linear beyond ~8–16, and latency rises sharply under load (end-to-end p50 3.6 s → 18 s).
• No failed requests across the sweep.
• Capacity is added by scaling out replicas (horizontal scaling), not by pushing a single replica harder.

This is a single-replica, internal-endpoint baseline on a synthetic fixed-length workload; public-endpoint and additional-model (e.g. llm-qwen3-14b) results will follow, along with comparisons across GPU-partitioning changes. Full procedure, wrapper script, and raw results live in benchmarks/ in the inference-services repository; metric definitions follow T431554; further benchmarking and load-testing work is tracked in T431851.

New here? Start with the Quick Start, then try LiftWing Studio and the API.

• Machine Learning