Sonnet 4.6 Cosmopedia v2 Agent Run Mar 24 2026

Third cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing Cosmopedia-v2 on the same hardware previously used by Claude Sonnet 4.0. This is the most surprising comparison yet because Sonnet 4.6 discovered ASPECT_RATIO=21 — breaking the universal AR=32 consensus that held across all five Sonnet 4.0 datasets.

Key finding: Sonnet 4.6 found a fundamentally different architecture (AR=21 vs AR=32) and beat Sonnet 4.0's best val_bpb by 0.6% (0.9549 vs 0.9606). With 16 keeps (vs 4), zero crashes (vs 2), and a systematic AR walk from 32→24→22→21, this run demonstrates that the "AR=32 is hardware-optimal" conclusion from Sonnet 4.0 was actually an artifact of the LLM's exploration strategy, not a true hardware constraint.

Sonnet 4.0 started from a pre-optimized baseline (AR=32, inherited from prior runs), while Sonnet 4.6 started from reset defaults. Despite starting from a worse baseline, Sonnet 4.6 found a significantly better absolute result — the first time across any dataset comparison that the later model has clearly and unambiguously won.

Phase 1 — Optimizer Tuning (exp1–exp8): Reduced β2 from 0.975 to 0.95 for faster adaptation, halved weight decay from 0.2 to 0.1. Two keeps established a better optimizer baseline.

Phase 2 — Learning Rate Rebalancing (exp9–exp18): Increased EMBEDDING_LR and MATRIX_LR, shortened WARMDOWN_RATIO from 0.77 to 0.65. Three more keeps, moving val_bpb from 0.9754 to 0.9681.

Phase 3 — Architecture Discovery (exp28–exp42): The pivotal phase. Reduced ASPECT_RATIO from 32→24 (keep), re-tuned MATRIX_LR for the new architecture, then reduced FINAL_LR_FRAC to 0.02. Three keeps, establishing the narrower architecture as superior.

Phase 4 — Architecture Walk (exp59–exp70): Systematically walked ASPECT_RATIO down: 24→22→21 (both keeps). Then re-tuned learning rates for the deeper model — EMBEDDING_LR down to 0.50, SCALAR_LR down through 0.18 to 0.13 (two consecutive keeps). Six keeps in this phase alone.

Phase 5 — Fine-Tuning (exp71–exp96): Exhaustive sweep of all parameters. WARMDOWN_RATIO 0.65→0.66 (tiny keep) and ADAM β2 0.95→0.955 (final keep). Most experiments in this phase confirmed the configuration was near-optimal.

This is the signature finding of this run. The systematic walk:

AR=20 was tried before 22 and 21 (at exp58, discarded). The sweet spot is AR=21–22, where the model gets 17% more gradient steps than AR=32 while remaining wider than the over-narrow AR=20. Sonnet 4.0 never explored below AR=32 on any dataset.

# Sonnet 4.6 Cosmopedia-v2 Best (exp96)
ASPECT_RATIO = 21 # was 32 — narrower, deeper model
MATRIX_LR = 0.068 # was 0.053
EMBEDDING_LR = 0.50 # was 0.45
SCALAR_LR = 0.13 # was 0.23 — dramatically lower
UNEMBEDDING_LR = 0.006 # unchanged
WEIGHT_DECAY = 0.1 # was 0.2
WARMDOWN_RATIO = 0.66 # was 0.77
FINAL_LR_FRAC = 0.02 # was 0.1
ADAM_BETAS = (0.65, 0.955) # was (0.67, 0.975)
# Result: 0.9549 | mem: 14.5 GB | tok/sec: 57.3K | MFU: 17.6% | 524 steps

Every single parameter differs. The ASPECT_RATIO divergence (32 vs 21) is the most dramatic — it represents a fundamentally different model architecture. Sonnet 4.6 traded model width for depth, getting a narrower but deeper network that runs more gradient steps in the time budget. The SCALAR_LR divergence (0.4 vs 0.13) is the second-largest — a 68% reduction.

See also: Cross-LLM Comparison | Cross-Dataset Comparison | Sonnet 4.0 Cosmopedia-v2 (Mar 20)