Sonnet 4.0 Stock ClimbMix Agent Run Mar 26 2026

Sonnet 4.0 stock-baseline run: Claude Sonnet 4.0 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). Part of the fair-baseline methodology — every model starts from identical, unoptimized hyperparameters.

Key finding: Sonnet 4.0 is architecturally blind. In 100 experiments, it never changed ASPECT_RATIO, DEPTH, HEAD_DIM, or WINDOW_PATTERN — the four parameters that define model architecture. All 15 keeps came from optimizer tuning (LRs, betas, warmdown). This architectural conservatism left it trapped at AR=64's low throughput, producing the worst absolute result of any model despite the highest keep rate.

Status: Complete (100/100 experiments)

Phase 1 — LR Discovery (exp1–5, 5 experiments, 2 keeps): Quick wins with MATRIX_LR=0.06 and EMBEDDING_LR=0.8. Established the higher-LR strategy that defined the rest of the run. Zero crashes — S4.0 never attempted risky parameter changes.

Phase 2 — β1 Walk (exp6–35, 30 experiments, 3 keeps): Systematic reduction of Adam β1: 0.8→0.7→0.65→0.64. Interspersed with failed attempts at weight decay, other LRs, warmup, and β2 changes. The β1 walk was productive but slow — 30 experiments for 3 keeps.

Phase 3 — MATRIX_LR Micro-Tuning (exp36–42, 7 experiments, 3 keeps): Fine-grained MATRIX_LR optimization: 0.065→0.066→0.067. Each increment of 0.001 produced a keep. This is S4.0's signature pattern — exhaustive micro-tuning of a single parameter.

Phase 4 — Combined β1 + LR (exp43–64, 22 experiments, 2 keeps): Continued β1 walk (0.63, then self-corrected to 0.635) while attempting other parameters. 20 experiments between exp43 and exp64 produced zero keeps — a long plateau.

Phase 5 — WARMDOWN Discovery (exp65–78, 14 experiments, 2 keeps): After extensive LR/beta optimization, finally explored warmdown schedule. Found WARMDOWN_RATIO=0.3 via two successive keeps (0.35, 0.3). One depth attempt (exp75: +1 layer, val_bpb=1.6795) was catastrophic — the only architecture experiment in the entire run.

Phase 6 — Terminal Micro-Tuning (exp79–99, 21 experiments, 2 keeps): Final MATRIX_LR refinement at sub-0.001 precision (0.0673→0.06725→0.0673). The last 10 experiments achieved a total improvement of 0.0003 — effectively zero. S4.0 exhausted its search space without ever discovering architecture.

S4.0 operates in a single dimension: optimizer parameters. It never considers that the model architecture itself could be the bottleneck. This is visible in the throughput data — S4.0's experiments all run at 207–251 gradient steps, while Haiku's architecture changes enabled 697 steps. More steps = more learning = better results.

S4.0 ran exactly one architecture experiment in 100 tries — and it was a depth increase on AR=64, which was catastrophic (1.6795 val_bpb). It never tried reducing AR, reducing depth, or changing window patterns. This single data point may have reinforced the belief that architecture changes are dangerous.

The stock AR=64 baseline runs at ~230 steps in 5 minutes. If S4.0 had discovered AR=32 (like Haiku), it would have run ~400+ steps. If it had also reduced depth (like Haiku), it would have run ~700 steps. S4.0's optimizer-only improvements on 230 steps produced 1.3588. Haiku's architecture + optimizer improvements on 697 steps produced 1.2953. The gap is 0.0635 — entirely attributable to architecture.

Memory: unchanged at 26.1 GB throughout — no architecture changes means no memory reduction. Throughput improved only 21% (22.6K→27.3K tok/sec) from optimizer tuning. Compare: Haiku tripled throughput and halved memory via architecture.

# Sonnet 4.0 ClimbMix Best (exp91) — Stock Baseline
ASPECT_RATIO = 64 # UNCHANGED from stock — never explored!
HEAD_DIM = 128 # unchanged
WINDOW_PATTERN = "SSSL" # UNCHANGED from stock — never explored!
DEPTH = _hp_defaults['depth'] # UNCHANGED — one failed attempt
MATRIX_LR = 0.0673 # was 0.04 — micro-tuned extensively
EMBEDDING_LR = 0.8 # was 0.6
UNEMBEDDING_LR = 0.004 # unchanged
SCALAR_LR = 0.5 # unchanged
WEIGHT_DECAY = 0.2 # unchanged
ADAM_BETAS = (0.635, 0.95) # β1 was 0.8 — walked down
WARMDOWN_RATIO = 0.3 # was 0.5
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.3588 | mem: 26.1 GB | tok/sec: 27.3K | MFU: 24.2% | 251 steps

S4.0's 100 experiments demonstrate a clear hierarchy: architecture changes dominate optimizer tuning in short-budget training. S4.0 achieved the highest MFU (24.2%) of any model by perfectly optimizing the compute it had — but it had the fewest gradient steps (251) because it never reduced the model's compute footprint.

More steps consistently wins, even at lower MFU. The most efficient use of 5 minutes is to make the model smaller and train it longer — not to perfectly tune a model that's too big to train adequately.

Run complete (100/100 experiments). Sonnet 4.0 finishes 3rd with val_bpb = 1.3588.

See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.6 Stock ClimbMix (Mar 26)