-
Notifications
You must be signed in to change notification settings - Fork 0
Haiku 4.5 ClimbMix Agent Run Mar 25 2026
First cross-generation Haiku comparison run: Claude Haiku 4.5 optimizing the climbmix-400b dataset on the same hardware used by Claude Sonnet 4.0 and Sonnet 4.6. This adds a third model tier to the cross-LLM comparison matrix.
Status: In progress (22/101 experiments completed)
Early finding: Haiku's inherited baseline already beats both Sonnets' best. Starting from Sonnet 4.6's accumulated config (AR=21, β2=0.964), Haiku's unmodified baseline (1.2966) is already lower than Sonnet 4.0's best (1.2959, after 101 experiments) and Sonnet 4.6's best (1.2997, after 119 experiments). After just one keep (exp15), Haiku reaches 1.2924 — the new absolute best on ClimbMix.
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 22 (exp0–exp21) |
| Kept | 1 (4.5%) |
| Discarded | 19 (86.4%) |
| Crashed | 2 (9.1%) |
| Best val_bpb | 1.2924 (exp15) |
| Baseline val_bpb | 1.2966 (inherited S4.6 config) |
| Total Improvement | −0.0042 (0.32%) |
| LLM Backend | Claude Haiku 4.5 |
| Results | results/haiku-4-5-20251001/climbmix/ |
Unlike the Sonnet runs, Haiku started from Sonnet 4.6's accumulated best configuration (from the FineWeb-Edu-High run). This gives it a significant head start:
| Metric | Sonnet 4.0 | Sonnet 4.6 | Haiku 4.5 |
|---|---|---|---|
| Baseline val_bpb | 1.2969 (pre-optimized) | 1.3213 (defaults) | 1.2966 (S4.6 config) |
| Starting AR | 32 | 32 | 21 (from S4.6) |
| Starting β2 | 0.95 | 0.95 | 0.964 (from S4.6) |
| Starting config | Prior characterization | Reset defaults | S4.6 accumulated best |
Haiku's baseline (1.2966) is already better than Sonnet 4.0's best-ever result (1.2959 after 101 experiments). Fair comparison requires accounting for this inherited advantage.
# From Sonnet 4.6's FineWeb-Edu-High accumulated best ASPECT_RATIO = 21 # S4.6 discovery (was 32 for both Sonnets) MATRIX_LR = 0.068 # S4.6 optimized EMBEDDING_LR = 0.60 # S4.6 optimized SCALAR_LR = 0.18 # S4.6 optimized WEIGHT_DECAY = 0.08 # S4.6 optimized WARMDOWN_RATIO = 0.75 # S4.6 optimized FINAL_LR_FRAC = 0.05 # S4.6 optimized ADAM_BETAS = (0.45, 0.964) # S4.6 β2 walk result WINDOW_PATTERN = "SSSS" # S4.6 all-sliding
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 1.2966 | — | Baseline (inherited S4.6 config) | Starting point |
| exp15 | 1.2924 | −0.32% | SCALAR_LR 0.18 → 0.12 | Lower scalar LR |
Phase 1 — Batch Size Exploration (exp1–3, 3 experiments): Haiku immediately tried to maximize gradient steps through batch size changes. TOTAL_BATCH_SIZE=256 crashed (exp1), DEVICE_BATCH_SIZE=8 degraded throughput (exp2), DEVICE_BATCH_SIZE=2 was slightly better but still worse than baseline (exp3). All batch experiments failed or degraded — the inherited config's batch settings are already near-optimal.
Phase 2 — LR and Schedule Sweeps (exp4–14, 11 experiments): Systematic exploration of MATRIX_LR, WARMDOWN_RATIO, WARMUP_RATIO, HEAD_DIM, DEVICE_BATCH_SIZE, and ASPECT_RATIO. All discards. Haiku tested AR=16 and AR=18 (both worse than inherited AR=21). HEAD_DIM=256 (exp11) came close (1.2994) but didn't beat baseline.
Phase 3 — Per-Parameter LR Tuning (exp15–21, 7 experiments): Found the only keep: SCALAR_LR 0.18→0.12 (exp15, val_bpb 1.2924). Then tried to extend the strategy to EMBEDDING_LR, UNEMBEDDING_LR, MATRIX_LR — all failed. SCALAR_LR 0.10 (exp21) overshot. Haiku correctly identified LR tuning as the right direction but struggled to find the next improvement.
2 crashes (9.1%) in 22 experiments:
| Exp | Parameter | Issue |
|---|---|---|
| exp1 | TOTAL_BATCH_SIZE → 256 | Numerical instability from extreme batch reduction |
| exp6 | DEVICE_BATCH_SIZE → 3 | Odd batch size crash |
Both crashes were in the first 6 experiments, from aggressive batch size changes. Haiku learned quickly and stopped attempting batch modifications after exp12.
| Behavior | Sonnet 4.0 | Sonnet 4.6 | Haiku 4.5 (early) |
|---|---|---|---|
| First keep | exp84 (84th) | exp25 (25th) | exp15 (15th) |
| Crash rate | 11.0% | 4.2% | 9.1% (early) |
| Strategy | Exhaustive sweeps | Compositional | LR micro-tuning |
| Batch experiments | Many (11 crashes) | Some (5 crashes) | Few, stopped early |
| Unique strength | Thoroughness | Multi-param synergy | Fast convergence |
Haiku found its first keep faster than both Sonnets. However, it's struggling to find a second improvement — every experiment after exp15 has been a discard. The inherited config may be close enough to a local optimum that Haiku's LR-tuning strategy can't escape it.
Status: In progress — will be updated when run completes.
See also: Cross-LLM Comparison | Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 ClimbMix (Mar 19)