Opus 4.6 Stock ClimbMix Agent Run Mar 26 2026

Opus 4.6 stock-baseline run: Claude Opus 4.6 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). The fourth model in the fair-baseline comparison series.

Key finding: Opus is a methodical optimizer-tuner, not an architecture explorer. After a rocky start (4 consecutive baseline crashes, 26% early crash rate), Opus settled into a steady optimization pattern — walking MATRIX_LR systematically, discovering EMBEDDING_LR=0.4 and SCALAR_LR=0.45, and achieving 12 keeps in 70 experiments (17.1% — highest of any model). But it never changed architecture (AR=64, default depth, SSSL window), making it structurally similar to Sonnet 4.0's approach.

Status: In progress (70/100 experiments) — paused due to API 529 (overload)

Opus is the only model that crashed on stock defaults:

Four consecutive crashes before Opus discovered that not overriding defaults was the fix. The irony: exp1-3 crashed because Opus's manual "fixes" broke the dynamic config system. Compare: S4.0, S4.6, and Haiku all ran the baseline perfectly on first try.

Phase 1 — Crash Recovery (exp0–4, 5 experiments, 1 keep): Four crashes before establishing a working baseline. Opus over-engineered "fixes" that broke the dynamic config. 20% of early budget wasted.

Phase 2 — Broad Exploration (exp5–28, 24 experiments, 1 keep): Found DEVICE_BATCH_SIZE=8 (exp8). Then catastrophic detour — tried making the model bigger: depth=10 (val_bpb=2.0016), AR=80 (val_bpb=1.6011), depth=8 with batch=16 (crash). Three more crashes from batch size changes (exp23–25). Also tried increasing LRs (all worse), adding warmup (worse). Opus explored in the wrong direction on nearly every parameter.

Phase 3 — MATRIX_LR Walk (exp29–34, 6 experiments, 4 keeps): The productive phase begins. Systematic MATRIX_LR walk: 0.045→0.05→0.048. Combined with WARMDOWN_RATIO=0.4 and WEIGHT_DECAY=0.15. Four consecutive keeps — Opus found its groove.

Phase 4 — Continued LR Optimization (exp35–47, 13 experiments, 3 keeps): Extended MATRIX_LR walk: 0.05→0.052→0.054 (3 more keeps). Tried various other parameters (EMBEDDING_LR up, SCALAR_LR up, β changes, HEAD_DIM=64, window pattern) — all discards. exp43 attempted to change WINDOW_PATTERN but set it to "SSSL" (the existing default) — effectively a no-op.

Phase 5 — LR Reduction Discovery (exp48–63, 16 experiments, 2 keeps): Pivot to reducing learning rates. Found SCALAR_LR=0.45 (exp58) and EMBEDDING_LR=0.4 (exp63) — the current best. This is notable: Opus initially tried increasing these LRs (both failed), then discovered the opposite direction works.

Phase 6 — Plateau (exp64–69, 6 experiments, 0 keeps): Fine-tuning attempts around exp63's best config. All discards. Run paused at exp69 due to API 529 (overload).

Opus uses small, methodical steps — the MATRIX_LR walk (0.04→0.045→0.05→0.052→0.054 in 5 keeps) is its cleanest pattern. Unlike Haiku's binary search or S4.6's systematic architecture sweeps, Opus takes incremental steps and rarely jumps.

Opus is the only model to try increasing model size — and every attempt failed catastrophically. It then abandoned architecture exploration entirely, settling into pure optimizer tuning like S4.0.

Opus showed one interesting adaptive behavior: it initially tried increasing EMBEDDING_LR (0.6→1.0, discard) and SCALAR_LR (0.5→0.8, discard), then later discovered that decreasing them works (EMBEDDING_LR→0.4, SCALAR_LR→0.45). This reversal echoes S4.6's MATRIX_LR self-correction, though less dramatic.

Opus has the highest keep rate (17.1%) but is limited by architecture — 260 steps vs Haiku's 697. Its best (1.3569) is already within striking distance of S4.0's final (1.3588), suggesting Opus will likely surpass S4.0 but remain well behind architecture-exploring models.

9 crashes (12.9%) — highest of any model:

Pattern: 4 crashes from over-engineering, 5 from making the model/batch bigger. Zero crashes from the kind of reductions that produced keeps in other models.

Memory essentially unchanged (24.8–25.0 GB). Throughput improved only 23% (23K→28K tok/sec). Steps increased 23% (211→260). Compare: Haiku tripled throughput and more than tripled steps via architecture.

# Opus 4.6 ClimbMix Best (exp63) — 70/100 experiments
ASPECT_RATIO = 64 # UNCHANGED from stock — never explored reduction!
HEAD_DIM = 128 # unchanged (tried 64, catastrophic)
WINDOW_PATTERN = "SSSL" # UNCHANGED — tried to change but set it to same value
DEPTH = default # UNCHANGED — increases all failed
DEVICE_BATCH_SIZE = 8 # changed from default
MATRIX_LR = 0.054 # was 0.04 — walked up in 5 keeps
EMBEDDING_LR = 0.4 # was 0.6 — reduced (initial increase failed)
SCALAR_LR = 0.45 # was 0.5 — reduced
WEIGHT_DECAY = 0.15 # was 0.2 — reduced
WARMDOWN_RATIO = 0.4 # was 0.5 — reduced
ADAM_BETAS = (0.8, 0.95) # unchanged
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.3569 | mem: 25.0 GB | tok/sec: 28.3K | MFU: 25.1% | 260 steps

Both Opus and S4.0 are "pure optimizer tuners" — neither changed architecture. But they took opposite LR strategies:

S4.0 went higher on LRs and lower on momentum. Opus went lower on most LRs. Different paths to similar results — confirming the multi-optima landscape pattern seen across all models.

Status: In progress (70/100 experiments) — paused due to API 529. Will resume.

See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.6 Stock ClimbMix (Mar 26) | Sonnet 4.0 Stock ClimbMix (Mar 26)