Sonnet 4.6 SlimPajama Agent Run Mar 24 2026

Fourth cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing SlimPajama on the same hardware previously used by Claude Sonnet 4.0. Unlike Cosmopedia-v2 where Sonnet 4.6 found a dramatically better architecture, SlimPajama proved to be the hardest dataset for both models — and the first where Sonnet 4.0 clearly wins.

Key finding: SlimPajama resists optimization. Both models achieved minimal improvement from baseline — Sonnet 4.0 found 1.0% improvement with 3 keeps, Sonnet 4.6 found only 0.29% with 2 keeps. The optimization landscape is extremely flat, likely due to the dataset's multi-source diversity (CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange).

Sonnet 4.6's defaults start better than Sonnet 4.0's pre-optimized baseline (1.5312 vs 1.5410), but Sonnet 4.0 found deeper improvements and achieved a better absolute result. This is the first dataset where Sonnet 4.0 definitively outperforms Sonnet 4.6.

Phase 1 — Exploration (exp1–exp35): Exhaustive but fruitless search. Tested architecture changes (AR=14, 16, 24, 28), batch sizes (half and quarter total batch), all learning rates, Adam betas, weight decay, and warmdown ratios. Nothing beat the baseline. 35 consecutive discards.

Phase 2 — Regression Investigation (exp36–exp49): The agent noticed that even exact baseline replication attempts were performing worse than exp0. This suggests variance in training outcomes — the baseline may have been a lucky run. The agent spent 14 experiments diagnosing this without resolution.

Phase 3 — First Breakthrough (exp58–exp63): The agent discovered that lower β1 (0.55 vs 0.65) combined with lower FINAL_LR_FRAC (0.01 vs 0.02) and higher WARMDOWN_RATIO (0.70 vs 0.66) could just barely beat the baseline. Exp63 combined all three for the first keep.

Phase 4 — Refinement (exp64–exp91): Extensive sweep around exp63's configuration. Most changes hurt. The only additional improvement came from tuning β2 from 0.955 to 0.945 (exp91, second and final keep).

Phase 5 — Confirmation (exp92–exp100): Verified that exp91's configuration was the local optimum — β2=0.935, 0.940, 0.950 all worse. Other parameter tweaks also failed. The landscape is definitively flat.

# Sonnet 4.6 SlimPajama Best (exp91)
ASPECT_RATIO = 21 # default (unchanged)
MATRIX_LR = 0.068 # default (unchanged)
EMBEDDING_LR = 0.50 # was 0.30 in some experiments, kept at default
SCALAR_LR = 0.13 # default (unchanged)
WEIGHT_DECAY = 0.1 # default (unchanged)
WARMDOWN_RATIO = 0.70 # was 0.66
FINAL_LR_FRAC = 0.01 # was 0.02
ADAM_BETAS = (0.55, 0.945) # was (0.65, 0.955) — both reduced
# Result: 1.5267 | mem: 14.5 GB | tok/sec: 50.6K | MFU: 15.6% | 464 steps

SlimPajama is the hardest dataset to compress (val_bpb 1.527 vs 0.955 for Cosmopedia-v2) due to its extreme source diversity. The optimization landscape appears to be:

1. Extremely flat — 2% keep rate for both models, smallest improvements of any dataset
2. High variance — the agent couldn't reliably replicate even the baseline result, suggesting training noise dominates small hyperparameter changes
3. Default-near-optimal — the inherited defaults from the Cosmopedia-v2 Sonnet 4.6 run (AR=21, β1=0.65, β2=0.955) were already well-tuned for diverse text

This contrasts with Cosmopedia-v2 (synthetic textbooks with consistent style) where the same model found 16 keeps and 2.63% improvement.

See also: Cross-LLM Comparison | Cross-Dataset Comparison | Sonnet 4.0 SlimPajama (Mar 20)