Sonnet 4.0 ClimbMix Stock Baseline Mar 26 2026

Sonnet 4.0 re-run from stock upstream defaults (AR=64, stock LRs) — the same fair-baseline methodology used for Haiku 4.5. This replaces the original pre-optimized Sonnet 4.0 ClimbMix run (which started from AR=32 and achieved 1.2959).

Key finding: Sonnet 4.0 is blind to architecture. It never attempted to change ASPECT_RATIO, DEPTH, or WINDOW_PATTERN — the three parameters that produced Haiku's largest improvements. Sonnet 4.0 optimized exclusively within the optimizer parameter space (learning rates, Adam betas, warmdown), achieving a respectable 4.03% improvement but finishing 0.0635 behind Haiku in absolute val_bpb.

Status: Complete (100/100 experiments)

Same stock upstream defaults as Haiku 4.5, ensuring a fair comparison:

Note: Sonnet 4.0's stock baseline (1.4159) is slightly worse than Haiku's (1.4085) due to run-to-run variance. Both start from identical code.

Phase 1 — LR Discovery (exp1–20, 20 experiments, 2 keeps): Explored batch sizes (discards), LR values, weight decay, warmup. Found MATRIX_LR=0.06 (exp3) and EMBEDDING_LR=0.8 (exp5). Also tested SCALAR_LR, UNEMBEDDING_LR, WEIGHT_DECAY, WARMUP_RATIO, WARMDOWN_RATIO, and ADAM_BETAS changes — all discards. Zero crashes.

Phase 2 — β1 Walk (exp21–35, 15 experiments, 3 keeps): Discovered that reducing Adam β1 improves performance. Walked β1 from 0.8 → 0.7 → 0.65 → 0.64 in three keeps. Also tested β2=0.9 and β2=0.85 — both discards. This is Sonnet 4.0's signature optimization dimension.

Phase 3 — MATRIX_LR Refinement (exp36–42, 7 experiments, 3 keeps): Incrementally improved MATRIX_LR from 0.065 → 0.066 → 0.067. Each step was a keep. 0.068 and above were discards. Very systematic and efficient.

Phase 4 — β1 + LR Synergy (exp43–63, 21 experiments, 1 keep): Combined β1 reduction to 0.63 with the optimized LR (exp43, keep). Then spent 20 experiments micro-tuning β1 (0.61–0.638), MATRIX_LR (0.0665–0.0672), EMBEDDING_LR (0.75–0.85), and other params — all discards. One depth increase attempt (exp75, 1.6795) was catastrophic but notably not a crash.

Phase 5 — β1 Reversal + Warmdown (exp64–78, 15 experiments, 3 keeps): Made the surprising discovery that β1=0.635 beats β1=0.63 (exp64). Then found WARMDOWN_RATIO improvements: 0.5 → 0.35 → 0.3 in two consecutive keeps (exp77–78).

Phase 6 — Final MATRIX_LR Micro-Tuning (exp82–99, 18 experiments, 3 keeps): Pushed MATRIX_LR from 0.067 → 0.0673 → 0.06725 → 0.0673 in three keeps. The final 8 experiments explored fourth- and fifth-decimal variations with diminishing returns.

The most striking aspect of this run is what Sonnet 4.0 did not explore:

Sonnet 4.0 stayed at AR=64 for all 100 experiments. At AR=64, the model is too wide and slow — only 207–251 gradient steps per 5-minute budget. Haiku's AR=32 tripled throughput to 697 steps, which is the primary reason Haiku won.

Sonnet 4.0's optimization happened entirely within the "optimizer knob" space. It discovered strong improvements:

• β1 walk (0.8→0.635): systematic and well-executed
• MATRIX_LR refinement (0.04→0.0673): precise and effective
• WARMDOWN_RATIO walk (0.5→0.3): late but valuable

But it never questioned the architecture. The single depth change (exp75) produced such a terrible result (1.6795, +18.7%) that it may have discouraged any further architecture exploration. This is a classic negative transfer from a single bad experiment — one catastrophic architecture result caused permanent avoidance of the entire architecture dimension.

Zero crashes in 100 experiments — the only model to achieve a perfect crash-free record.

Sonnet 4.0 never attempted aggressive batch size changes (unlike Haiku's exp1–2, which crashed). It explored DEVICE_BATCH_SIZE conservatively (4 and 2 only) with graceful discards rather than crashes.

Memory stayed constant at 26.1 GB throughout — because the architecture never changed. Throughput improved only 21% (22.6K→27.3K tok/sec) and steps increased 21% (207→251) through optimizer tuning alone.

Compare to Haiku: ×ばつ throughput increase (22K→76K), 56% memory reduction (26→12 GB), ×ばつ more steps (212→697).

# Sonnet 4.0 ClimbMix Stock Baseline Best (exp91)
ASPECT_RATIO = 64 # UNCHANGED from stock — never explored
DEPTH = 8 # UNCHANGED — one catastrophic attempt
HEAD_DIM = 128 # UNCHANGED
WINDOW_PATTERN = "SSSL" # UNCHANGED — never explored
MATRIX_LR = 0.0673 # was 0.04 (stock)
EMBEDDING_LR = 0.8 # was 0.6 (stock)
UNEMBEDDING_LR = 0.004 # unchanged
SCALAR_LR = 0.5 # unchanged
WEIGHT_DECAY = 0.2 # unchanged
ADAM_BETAS = (0.635, 0.95) # β1 was 0.8
WARMDOWN_RATIO = 0.3 # was 0.5
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.3588 | mem: 26.1 GB | tok/sec: 27.3K | MFU: 24.2% | 251 steps

The pre-optimized run started much closer to the optimum (AR=32 already set), leaving almost no room to improve. The stock baseline run shows Sonnet 4.0's true optimization capability — but also its blind spot: it can't discover architecture changes on its own.

Run complete (100/100 experiments).

See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.0 ClimbMix Pre-Optimized (Mar 19)