train-dspark-draft-models

Train a DSpark draft model for speculative decoding, with Qwen/Qwen3-0.6B as the verifier. Two notebooks do the training job in two ways — offline and online — and differ only in where the verifier's hidden states come from. A third notebook evaluates the drafter that comes out.

A drafter guesses several tokens ahead, the verifier checks them in one forward pass and keeps what it agrees with — same output, faster. DSpark drafts a whole block at a time and conditions on the verifier's hidden states, which is why a vLLM server runs during training. Built on vllm-project/speculators.

• What's in here
• Choosing a training mode
• Run it
  • 1. Train a drafter — pick one
  • 2. Evaluate it
• Trained models
  • Evaluation
• License

.
├── notebooks/
│ ├── [offline] train-dspark-drafter-qwen-3-0.6b.ipynb
│ ├── [online] train-dspark-drafter-qwen-3-0.6b.ipynb
│ └── evaluate-dspark-qwen-3-0.6b.ipynb
├── LICENSE
└── README.md

Everything runs inside the notebooks — there is no package to install from this repo, and speculators is cloned by each notebook at run time.

speculators can source hidden states three ways — online (fetched from a live vLLM server on demand and discarded), offline (all pre-generated to disk before training), and hybrid (generated during epoch 0, cached, reused). These notebooks cover the first two.

Both were tested on Kaggle with ×ばつ NVIDIA T4, and the table describes that setup. Data generation uses both T4s either way; the modes diverge once training starts.

Offline is strongly recommended if you have a single GPU — it materializes the hidden states, frees the GPU, and then gives the whole thing to training, whereas online has the server and the trainer competing for the same memory the entire run. With two or more GPUs both work: offline still trains twice as wide, so pick online mainly when disk, not GPU count, is the tighter constraint.

The wall-clock row is whole-notebook time — clone to Hub upload — at each notebook's own sample count, so it is not a clean head-to-head. The three-hour gap runs the same direction as the recommendation anyway: offline finishes sooner despite paying for an extra extraction pass, because training gets both GPUs. Runtime is roughly linear in the sample count, so scale these figures with the knob below.

Prerequisites: vllm>=0.22.0, a GPU (two if you want the online notebook), and a Hugging Face write token for the upload step. Everything else installs from inside the notebooks.

[offline] train-dspark-drafter-qwen-3-0.6b.ipynb

clone speculators → regenerate + tokenize data → serve the verifier → extract hidden states → stop the server → train on both GPUs → push to yosefw/Qwen3-0.6B-DSpark-v2.

[online] train-dspark-drafter-qwen-3-0.6b.ipynb

clone speculators → regenerate + tokenize data → serve the verifier → train against it live → push to yosefw/Qwen3-0.6B-DSpark.

Both write output/checkpoints/checkpoint_best and upload it to the Hub. Budget for a long session: end to end on ×ばつ T4 the offline notebook took 5 h 53 min and the online one 8 h 54 min — both fit a 12 h Kaggle session, neither by a comfortable margin. The sample count below is the lever if you need them shorter.

The one knob that matters is the sample count, and it appears in more than one place: --limit on response regeneration and --max-samples on prepare_data.py must always move together (offline has a third one to match, on data_generation_offline.py). The defaults are 1200 offline and 1600 online; drop them to ~200 to smoke-test the pipeline, and raise them well beyond the defaults for a drafter you actually plan to deploy.

evaluate-dspark-qwen-3-0.6b.ipynb

clone speculators → serve the drafter in vLLM, which pulls in its verifier automatically → evaluate.py throughput → acceptance metrics in acceptance.csv. Its sweep subcommand runs the full performance benchmark instead, across all 9 RedHatAI/speculator_benchmarks subsets.

Point it at either the Hub id or the local checkpoint_best path from step 1. Takes about 40 min on ×ばつ T4, most of it model download and server startup.

The drafters these notebooks produced, on the Hugging Face Hub:

Both are trained on small sample counts, so treat them as pipeline demonstrations rather than deployment-ready drafters.

Per-subset results for yosefw/Qwen3-0.6B-DSpark (the online-mode drafter, trained on 1600 samples), produced by evaluate-dspark-qwen-3-0.6b.ipynb — evaluate.py throughput against a vLLM server running the drafter on ×ばつ T4:

acceptance_length is the mean number of tokens kept per verification round (1 + accepted / drafts), capped at 5 here because the drafter is trained with --block-size 4. acceptance_at_pos_N is how often the Nth drafted token in a block survives verification; it decays across the block, which is exactly what DSpark's semi-autoregressive head exists to slow down.

Acceptance is highest on math_reasoning (1.970) and HumanEval (1.871) and lowest on translation (1.436) and summarization (1.485) — the drafter does best where the verifier's next token is most predictable. Across all subsets, weighted by num_drafts, acceptance length is 1.806 (435,398 accepted tokens over 539,885 drafts).

Based on the official speculators training and evaluating performance tutorials.

Apache-2.0 — see LICENSE. Builds on speculators, also Apache-2.0.