Opus 4.6 Stock ClimbMix Agent Run Mar 26 2026

Opus 4.6 stock-baseline run: Claude Opus 4.6 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). The fourth model in the fair-baseline comparison series.

Key finding (early): Opus is the worst optimizer so far. It crashed on the stock baseline (the only model to do so), has a 26% crash rate, tried to make the model bigger (wider, deeper) instead of smaller, and has found only marginal improvements. At 23 experiments, it trails every other model at the same stage.

Status: In progress (23/100 experiments)

Spec	Value
Chip	Apple M5 Max
Unified Memory	64 GB
GPU Cores	40
Backend	MLX
Time Budget	5 minutes per experiment

Metric	Value
Total Experiments	23 (exp0–exp22)
Kept	2 (8.7%)
Discarded	15 (65.2%)
Crashed	6 (26.1%)
Best val_bpb	1.3898 (exp8)
Effective Baseline	1.4098 (exp4 — exp0 crashed)
Total Improvement	−0.0200 (1.42%)
LLM Backend	Claude Opus 4.6
Results	results/opus-4-6/climbmix/

Opus is the only model that crashed on stock defaults:

Exp	Description	Status
exp0	Baseline (no modifications)	CRASH
exp1	"Fix crash by setting explicit conservative hyperparameters"	CRASH
exp2	"Set minimal conservative hyperparameters with small batch size"	CRASH
exp3	"Debug crash by using very minimal hardcoded hyperparameters with depth=4"	CRASH
exp4	"Use dynamic defaults directly without any overrides"	✅ 1.4098

Four consecutive crashes before Opus discovered that not overriding defaults was the fix. The irony: exp1-3 crashed because Opus's "fixes" (hardcoded values, tiny batches, shallow depth) were worse than just using the dynamic defaults. This suggests Opus over-engineers solutions to problems that don't exist.

Compare: Sonnet 4.0 ran the baseline perfectly. Sonnet 4.6 ran it perfectly. Haiku ran it perfectly.

Exp	val_bpb	Δ from Baseline	Description	Key Change
exp4	1.4098	—	Effective baseline (dynamic defaults)	Starting point
exp8	1.3898	−1.42%	DEVICE_BATCH_SIZE=8	Throughput optimization

Only one improvement in 23 experiments — and it's a batch size change, not a hyperparameter insight.

Phase 1 — Crash Recovery (exp0–4, 5 experiments, 1 keep): Four crashes before establishing a working baseline. Opus spent 20% of its experiments just getting the code to run. No other model needed more than 0 experiments for this.

Phase 2 — Batch Size Exploration (exp5–6, 8–9, 12–13, 18, 7 experiments, 1 keep): Found DEVICE_BATCH_SIZE=8 improved throughput (exp8, keep). But then tried reducing TOTAL_BATCH_SIZE (exp9: more steps but worse val_bpb) and increasing it (exp12: crash). Also tried DEVICE_BATCH_SIZE=16 (exp18: worse).

Phase 3 — Going Bigger (Wrong Direction) (exp7, 10, 19–20, 4 experiments, 0 keeps): Opus's critical strategic error. Instead of reducing model size to increase throughput (like Haiku and S4.6), Opus tried to make the model bigger:

• exp7: Increased depth → CRASH
• exp10: Depth=10 → val_bpb=2.0016 (catastrophic, +42% worse)
• exp19: Depth=8 → 1.4260 (worse)
• exp20: AR=80 → val_bpb=1.6011 (catastrophic, +13.6% worse)

Every attempt to increase model capacity failed. Opus has not yet tried reducing AR or depth.

Phase 4 — LR/Optimizer Sweep (exp11, 14–17, 21–22, 7 experiments, 0 keeps): Standard LR exploration: MATRIX_LR (0.06, 0.08), EMBEDDING_LR (1.0), SCALAR_LR (0.8), WEIGHT_DECAY (0.1), WARMUP_RATIO (0.05), WARMDOWN_RATIO (0.3). All discards. Opus explored in the wrong direction on every parameter tested (increasing LRs that should decrease, adding warmup that hurts).

Opus exhibits a distinctive pattern: it treats every problem as a debugging exercise. The baseline crash led to 3 experiments of increasingly complex "fixes" when the answer was simply to use defaults. The exp10 depth increase and exp20 AR increase show Opus reasoning about model capacity rather than training throughput — the wrong mental model for 5-minute training.

Model	Direction of Architecture Exploration	Result
Haiku	Smaller (AR=32, depth−3)	✅ Won ClimbMix
S4.6	Smaller (AR=19)	✅ 2nd place
S4.0	None (stayed at AR=64)	3rd place
Opus	Bigger (AR=80, depth=10)	❌ Catastrophic

Opus is the only model to try increasing model size. This reflects a fundamental misunderstanding of the 5-minute training constraint: with limited time, fewer parameters = more gradient steps = better results. Opus seems to reason from a "bigger model = better performance" prior that doesn't apply to time-constrained training.

Metric	S4.0 (at exp23)	S4.6 (at exp23)	Haiku (at exp23)	Opus (at exp23)
Best val_bpb	~1.37	~1.33	~1.35	1.39 (worst)
Keeps	3	~3	3	2 (fewest)
Crashes	0	~2	3	6 (most)
AR explored	No	Yes (walking down)	Yes (48, 32)	Yes (80 — wrong way!)
Crash rate	0%	~9%	13%	26% (highest)

6 crashes (26.1%) — the highest of any model:

Exp	Description	Issue
exp0	Stock baseline	Unknown — unique to Opus
exp1	"Conservative hyperparameters"	Opus's manual overrides broke config
exp2	"Minimal conservative hyperparameters"	Same — over-engineering
exp3	"Very minimal hardcoded depth=4"	Same — over-engineering
exp7	Increased depth	OOM or numerical instability
exp12	Increased TOTAL_BATCH_SIZE	OOM or numerical instability

Pattern: 4 crashes from over-engineering fixes, 2 from making the model bigger. Zero crashes from the kind of exploratory architecture reductions that produced keeps in other models.

Exp	MFU	tok/sec	Peak Mem (GB)	Steps
exp4 (baseline)	20.2%	22,984	24.8	211
exp8 (best)	22.0%	25,000	25.0	229

Memory and throughput are virtually unchanged — no architecture changes means no efficiency gains. Compare: at 23 experiments, Haiku had already discovered AR=32 (doubling throughput).

If Opus discovers AR reduction in the remaining 77 experiments, it could still be competitive. The path is clear:

1. AR=64 → AR=32 would roughly double throughput (211→400+ steps)
2. Depth reduction would add another 50%+ (400→600+ steps)
3. Window pattern changes could add 10-20% more throughput

But Opus has shown no inclination to explore in this direction. Its "bigger is better" bias may prevent it from discovering the architecture changes that made Haiku and S4.6 competitive.

---

Status: In progress (23/100 experiments) — early results only. Will be updated when run completes.

See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.6 Stock ClimbMix (Mar 26) | Sonnet 4.0 Stock ClimbMix (Mar 26)