-
Notifications
You must be signed in to change notification settings - Fork 0
Sonnet 4.6 SlimPajama Agent Run Mar 25 2026
Fourth cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing SlimPajama on the same hardware previously used by Claude Sonnet 4.0. SlimPajama is the hardest dataset to compress (highest baseline val_bpb) — a 7-source deduplicated web mix that proved resistant to optimization by both models.
Key finding: SlimPajama is optimization-resistant. Sonnet 4.6 found only 2 keeps in 101 experiments (2.0%) — the lowest keep rate of any Sonnet 4.6 run. Combined with Sonnet 4.0's 3 keeps, this dataset has proven the hardest to optimize across both model generations. The defaults that carry over from Cosmopedia-v2 (AR=21, β1=0.65, β2=0.955) are already near-optimal for this multi-source data.
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 101 (exp0–exp100) |
| Kept | 2 (2.0%) |
| Discarded | 97 (96.0%) |
| Crashed | 2 (2.0%) |
| Best val_bpb | 1.5267 (exp91) |
| Baseline val_bpb | 1.5312 (defaults) |
| Total Improvement | −0.0045 (0.30%) |
| LLM Backend | Claude Sonnet 4.6 |
| Results | results/sonnet-4-6/slimpajama/ |
| Metric | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Baseline val_bpb | 1.5410 (pre-optimized) | 1.5312 (defaults) |
| Best val_bpb | 1.5259 | 1.5267 |
| Improvement | −1.0% | −0.30% |
| Keeps | 3 (3.0%) | 2 (2.0%) |
| Crashes | 0 (0.0%) | 2 (2.0%) |
Both models find SlimPajama near-impossible to improve. Sonnet 4.0 edges out on absolute val_bpb (1.5259 vs 1.5267), but started from a worse baseline (1.5410 vs 1.5312). The optimization landscape for this multi-source dataset is remarkably flat.
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 1.5312 | — | Baseline (defaults) | Starting point |
| exp63 | 1.5277 | −0.23% | Combined β1=0.55, FINAL_LR_FRAC=0.01, WARMDOWN_RATIO=0.70 | Triple synergy |
| exp91 | 1.5267 | −0.30% | ADAM β2 0.955→0.945 | Lower β2 smoothing |
Phase 1 — Fruitless exploration (exp1–exp62): 62 experiments, zero keeps. The agent tried everything: AR changes (14, 16, 28), batch size reductions, LR adjustments, momentum changes, warmdown schedule variations. Nothing beat the baseline. This is the longest dry spell in any run across both models.
Phase 2 — Combination breakthrough (exp63): After 62 consecutive discards, the agent finally found improvement by combining three modest changes: β1 0.65→0.55, FINAL_LR_FRAC 0.02→0.01, WARMDOWN_RATIO 0.66→0.70. None worked alone — synergy was required.
Phase 3 — β2 fine-tuning (exp64–exp100): With exp63's config established, the agent exhaustively tested every remaining parameter. Only one more keep: β2 0.955→0.945 (exp91). 38 more experiments confirmed the configuration was near-optimal.
# Sonnet 4.6 SlimPajama Best (exp91) ASPECT_RATIO = 21 # unchanged from defaults MATRIX_LR = 0.068 # unchanged EMBEDDING_LR = 0.30 # was 0.50 (inherited default was different) SCALAR_LR = 0.13 # unchanged WEIGHT_DECAY = 0.1 # unchanged WARMDOWN_RATIO = 0.70 # was 0.66 FINAL_LR_FRAC = 0.01 # was 0.02 ADAM_BETAS = (0.55, 0.945) # was (0.65, 0.955) # Result: 1.5267 | mem: 14.5 GB | tok/sec: 50.6K | MFU: 15.6% | 464 steps
See also: Cross-LLM Comparison | Cross-Dataset Comparison | Sonnet 4.0 SlimPajama (Mar 20)