Opus 4.6 Stock ClimbMix Agent Run Mar 26 2026

Opus 4.6 stock-baseline run: Claude Opus 4.6 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). The fourth and final model in the fair-baseline ClimbMix comparison.

Key finding: The most expensive model finishes 3rd. Opus never discovered architecture changes despite trying both directions (AR=56 too timid, AR=68/80 catastrophic). With 13 crashes (13% — highest of any model) and no architecture innovation, Opus relied purely on optimizer tuning to reach 1.3569 — barely edging past Sonnet 4.0 (1.3588) but far behind architecture-discovering models Haiku (1.2953) and Sonnet 4.6 (1.3093).

Status: Complete (100/100 experiments)

Opus is the only model that crashed on stock defaults:

Four consecutive crashes before Opus discovered that not overriding defaults was the fix. The irony: exp1-3 crashed because Opus's manual "fixes" broke the dynamic config system. Compare: S4.0, S4.6, and Haiku all ran the baseline perfectly on first try.

Phase 1 — Crash Recovery (exp0–4, 5 experiments, 1 keep): Four crashes before establishing a working baseline. Opus over-engineered "fixes" that broke the dynamic config. 5% of budget wasted before training even started.

Phase 2 — Broad Exploration (exp5–28, 24 experiments, 1 keep): Found DEVICE_BATCH_SIZE=8 (exp8). Then catastrophic detours — tried making the model bigger: depth=10 (val_bpb=2.0016), AR=80 (val_bpb=1.6011), depth=8+batch=16 (crash). Three more crashes from batch/total_batch size changes. Opus explored in the wrong direction on nearly every parameter.

Phase 3 — MATRIX_LR Walk (exp29–34, 6 experiments, 4 keeps): The productive phase begins. Systematic MATRIX_LR walk: 0.045→0.05→0.048. Combined with WARMDOWN_RATIO=0.4 and WEIGHT_DECAY=0.15. Four keeps in 6 experiments — Opus's best efficiency.

Phase 4 — Continued LR Optimization (exp35–55, 21 experiments, 3 keeps): Extended MATRIX_LR walk: 0.05→0.052→0.054 (3 more keeps). Tried window pattern change but accidentally set it to the same value ("SSSL"→"SSSL"). HEAD_DIM=64 was catastrophic (1.5327). β1=0.85 (wrong direction), β2=0.92 tested.

Phase 5 — LR Reduction Discovery (exp56–63, 8 experiments, 3 keeps): Pivot to reducing learning rates. Found SCALAR_LR=0.45 (exp58) and EMBEDDING_LR=0.4 (exp63) — the final best. Notable: Opus initially tried increasing these LRs (both failed), then reversed direction.

Phase 6 — Final 37 Experiments (exp64–99, 37 experiments, 0 keeps): The longest drought of any model. Opus explored every remaining dimension: EMBEDDING_LR micro-tuning (0.3, 0.35, 0.38, 0.42, 0.45 — all worse), β1 (0.75, 0.78 — worse), β2 (0.93, 0.94, 0.97 — worse), DEVICE_BATCH_SIZE (9, 10, 12 — all crashed), AR=56 (discard), AR=68 (catastrophic), window patterns (SSSS, SSLL — discards), MLP_RATIO (3.0, 3.5 — discards), FINAL_LR_FRAC (0.05 — discard). Nothing improved in the final 37% of experiments.

At experiment 82, Opus finally tried reducing ASPECT_RATIO (64→56). But 56 was too timid — val_bpb=1.4184, worse than baseline. Then at exp89, Opus went the other direction (64→68) — catastrophic at 1.5291. Opus concluded AR changes don't work and abandoned the idea.

The tragedy: if Opus had reduced AR to 32 (like Haiku) or 19 (like S4.6), it would have roughly doubled throughput to ~500+ steps and dramatically improved results. But testing only AR=56 (too close to 64 to matter) and AR=68 (wrong direction) gave Opus exactly the wrong signal about architecture's potential.

Opus tested 3 AR values but never went below 56 — the critical gap. Architecture reduction requires bold steps, not timid ones.

Opus takes small, methodical steps. The MATRIX_LR walk (0.04→0.045→0.05→0.052→0.054) is its cleanest pattern. But this incrementalism is fatal for architecture: AR=56 is too close to 64 to unlock the throughput gains that AR=32 or AR=19 provide.

Both Opus and S4.0 are "pure optimizer tuners" — neither effectively changed architecture. But they took opposite LR strategies:

S4.0 goes higher on LRs and lower on momentum. Opus goes lower on most LRs. Different paths to nearly identical results (1.3569 vs 1.3588) — confirming the multi-optima landscape.

Opus showed adaptive behavior: initially tried increasing EMBEDDING_LR (0.6→1.0, discard at exp15) and SCALAR_LR (0.5→0.8, discard at exp21), then later discovered that decreasing them was the key (EMBEDDING_LR→0.4, SCALAR_LR→0.45). This reversal echoes S4.6's MATRIX_LR self-correction.

13 crashes (13.0%) — highest of any model by far:

Opus crashes ×ばつ more than any other model. The batch size sensitivity is unique — exp79 (BS=12), exp80 (BS=10), exp87 (BS=9) all crashed, while other models handled similar changes fine.

Memory essentially unchanged (24.8–25.0 GB) — no architecture changes. Throughput improved only 23% (23K→28K). Steps increased 23% (211→260). Compare:

The two architecture-blind models (S4.0, Opus) cluster together. The two architecture-exploring models (S4.6, Haiku) cluster together. The gap between clusters is enormous.

# Opus 4.6 ClimbMix Best (exp63) — 3rd place
ASPECT_RATIO = 64 # UNCHANGED — tried 56 (too timid), 68/80 (catastrophic)
HEAD_DIM = 128 # unchanged (tried 64, catastrophic)
WINDOW_PATTERN = "SSSL" # UNCHANGED — tried SSSS, SSLL (discards)
DEPTH = default # UNCHANGED — increases all failed
DEVICE_BATCH_SIZE = 8 # changed from default
MATRIX_LR = 0.054 # was 0.04 — walked up in 5 keeps
EMBEDDING_LR = 0.4 # was 0.6 — reduced (initial increase failed)
SCALAR_LR = 0.45 # was 0.5 — reduced
WEIGHT_DECAY = 0.15 # was 0.2 — reduced
WARMDOWN_RATIO = 0.4 # was 0.5 — reduced
ADAM_BETAS = (0.8, 0.95) # unchanged (both directions tested, none kept)
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.3569 | mem: 25.0 GB | tok/sec: 28.3K | MFU: 25.1% | 260 steps

Opus's 100 experiments prove the same lesson as S4.0: you cannot optimizer-tune your way past an architecture bottleneck. Both models achieved the highest MFU (24–25%) by perfectly utilizing the compute they had — but they had the fewest gradient steps (251–260) because they never reduced the model's compute footprint.

The ranking perfectly correlates with gradient steps, not model capability or cost. Bigger LLM ≠ better optimizer. Smaller model ≠ worse optimizer. What matters is whether the LLM discovers that architecture changes dominate optimizer tuning in time-constrained training.

Run complete (100/100 experiments). Opus 4.6 finishes 3rd with val_bpb = 1.3569.

See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.6 Stock ClimbMix (Mar 26) | Sonnet 4.0 Stock ClimbMix (Mar 26)