-
Notifications
You must be signed in to change notification settings - Fork 0
Haiku 4.5 ClimbMix Agent Run Mar 25 2026
First Haiku 4.5 comparison run: Claude Haiku 4.5 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). This is the first run using the new fair-baseline methodology — every model starts from identical, unoptimized hyperparameters.
Key finding: Haiku is a fast but shallow optimizer. It achieves the highest keep rate of any model (11.9%) and discovers novel improvements both Sonnets missed (depth reduction, all-local attention, UNEMBEDDING_LR tuning). But it wastes 28% of its budget binary-searching HEAD_DIM (zero keeps) and never discovers multi-parameter synergies.
Status: In progress (67/100 experiments)
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 67 (exp0–exp66) |
| Kept | 8 (11.9%) |
| Discarded | 55 (82.1%) |
| Crashed | 3 (4.5%) |
| Best val_bpb | 1.3018 (exp55) |
| Baseline val_bpb | 1.4085 (stock defaults, AR=64) |
| Total Improvement | −0.1067 (7.58%) |
| LLM Backend | Claude Haiku 4.5 |
| Results | results/haiku-4-5-20251001/climbmix/ |
This run uses the new fair-baseline methodology. All models start from identical stock upstream defaults:
| Parameter | Stock Default |
|---|---|
| ASPECT_RATIO | 64 |
| EMBEDDING_LR | 0.6 |
| MATRIX_LR | 0.04 |
| SCALAR_LR | 0.5 |
| WEIGHT_DECAY | 0.2 |
| ADAM_BETAS | (0.8, 0.95) |
| WARMDOWN_RATIO | 0.5 |
| FINAL_LR_FRAC | 0.0 |
| WINDOW_PATTERN | SSSL |
The stock AR=64 baseline (1.4085) is significantly worse than the Sonnet baselines (S4.0: 1.2969, S4.6: 1.3213), which started from partially-optimized configs. Direct absolute val_bpb comparison requires accounting for this difference.
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 1.4085 | — | Baseline (stock defaults) | Starting point |
| exp6 | 1.4081 | −0.03% | WARMDOWN_RATIO 0.5 → 0.2 | Less aggressive LR decay |
| exp19 | 1.3729 | −2.53% | ASPECT_RATIO 64 → 48 | Architecture walk begins |
| exp20 | 1.3473 | −4.35% | ASPECT_RATIO 48 → 32 | Rediscovers AR=32 |
| exp30 | 1.3260 | −5.86% | DEPTH reduced | Novel — neither Sonnet tried |
| exp32 | 1.3166 | −6.52% | WINDOW_PATTERN SSSL → LLLL | Novel — all-local attention |
| exp44 | 1.3152 | −6.62% | DEPTH reduced further | More gradient steps |
| exp45 | 1.3047 | −7.37% | MATRIX_LR 0.04 → 0.045 | LR tuning on optimized arch |
| exp55 | 1.3018 | −7.58% | UNEMBEDDING_LR 0.004 → 0.0035 | Novel — neither Sonnet touched |
Phase 1 — Early Exploration (exp1–18, 18 experiments): Broad sweeps of batch sizes (3 crashes), LR values, weight decay, warmup ratio. Only one keep (exp6, WARMDOWN_RATIO). Haiku learned quickly from batch size crashes — zero crashes after exp5.
Phase 2 — Architecture Discovery (exp19–20, 2 experiments): The breakthrough phase. Haiku walked ASPECT_RATIO from 64→48→32 in two consecutive keeps, independently rediscovering Sonnet 4.0's AR=32 conclusion. This produced the largest single improvement (−4.35%).
Phase 3 — HEAD_DIM Obsession (exp21–29, 31–39, 19 experiments): Haiku's biggest weakness. Systematic binary search through HEAD_DIM values: 64, 96, 80, 76, 120, 116, 124, 72, 88, 84, 122, 126, 112, 100, 104. Every single experiment was a discard. HEAD_DIM=128 remained optimal throughout. This consumed 28% of total experiments on a dead parameter.
Phase 4 — Depth + Window Breakthrough (exp30, 32, 40–44, 15 experiments): Discovered two novel improvements neither Sonnet found: depth reduction (exp30) and all-local attention WINDOW_PATTERN=LLLL (exp32). Further depth reduction kept at exp44. Throughput jumped from 404 to 593 gradient steps.
Phase 5 — LR Micro-Tuning (exp45–55, 11 experiments): Found MATRIX_LR=0.045 (exp45) and UNEMBEDDING_LR=0.0035 (exp55). But also wasted 6 experiments on MATRIX_LR micro-variations (0.044, 0.046, 0.047, 0.0465, 0.0455) — all discards.
Phase 6 — Diminishing Returns (exp56–66, 11 experiments): All discards. Explored SCALAR_LR, EMBEDDING_LR, WARMDOWN_RATIO, WEIGHT_DECAY — nothing improved. Haiku appears stuck near a local optimum at 1.3018.
Haiku found three improvements that neither Sonnet 4.0 nor Sonnet 4.6 discovered:
| Discovery | Haiku | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|---|
| Depth reduction | 2 keeps (exp30, 44) | Never tried | Never tried |
| WINDOW_PATTERN=LLLL | Keep (exp32) | Kept SSSL | Found SSSS |
| UNEMBEDDING_LR tuning | Keep (exp55) | Never tried | Never tried |
Haiku uses a distinctive binary-search approach: find a promising parameter, then exhaustively bisect the range. This is visible in the HEAD_DIM exploration (19 values tested between 64–128) and MATRIX_LR micro-tuning (7 values tested between 0.044–0.05).
| Behavior | Sonnet 4.0 | Sonnet 4.6 | Haiku 4.5 |
|---|---|---|---|
| Strategy archetype | Exhaustive sweeps | Compositional synergy | Binary-search grid |
| AR discovery | AR=32 (pre-set) | AR=21 (novel) | AR=32 (rediscovered) |
| Window pattern | SSSL (default) | SSSS (all-sliding) | LLLL (all-local) |
| Depth exploration | Never tried | Never tried | Yes — 2 keeps |
| HEAD_DIM | Minimal | None | 19 experiments, 0 keeps |
| Multi-param synergy | No | Yes (signature strength) | No |
| Micro-tuning | Some | Moderate | Excessive |
| First keep | exp84 | exp25 | exp6 |
| Crash rate | 11.0% | 4.2% | 4.5% |
| Keep rate (at 67 exp) | ~1.5% | ~5% | 11.9% |
- Fastest first keep (exp6) and highest keep rate (11.9%) of any model
- Novel discoveries: Depth reduction, all-local attention, UNEMBEDDING_LR — all missed by both Sonnets
- Quick crash learning: Zero crashes after exp5
- Systematic architecture walks: AR=64→48→32 in two clean steps
- Binary-search trap: 28% of budget exhaustively searching HEAD_DIM, a dead parameter
- No compositional optimization: Every keep changed exactly one parameter. Never combined multiple changes (unlike S4.6's synergistic multi-param improvements)
- Micro-tuning addiction: After finding a sweet spot, spends 6+ experiments on ±0.001 variations instead of exploring new dimensions
- Misses Adam beta tuning entirely: Never attempted β1 or β2 changes (S4.6's signature discovery)
3 crashes (4.5%) — all in the first 5 experiments:
| Exp | Parameter | Issue |
|---|---|---|
| exp1 | TOTAL_BATCH_SIZE → 512 | Numerical instability |
| exp2 | TOTAL_BATCH_SIZE → 1024 | Numerical instability |
| exp5 | DEVICE_BATCH_SIZE reduced | OOM or instability |
Haiku learned from batch crashes quickly and never attempted risky batch changes again.
| Exp | MFU | tok/sec | Peak Mem (GB) | Steps |
|---|---|---|---|---|
| exp0 (baseline) | 20.3% | 22,977 | 26.1 | 212 |
| exp20 (AR=32) | 13.5% | 43,985 | 15.5 | 404 |
| exp32 (LLLL) | 18.3% | 61,666 | 13.3 | 564 |
| exp45 (best LR) | 18.6% | 72,594 | 11.6 | 664 |
| exp55 (current best) | 19.1% | 74,666 | 11.6 | 684 |
Throughput tripled (22K→74K tok/sec) and memory dropped 56% (26.1→11.6 GB) through architecture optimization. Steps increased ×ばつ (212→684).
# Haiku 4.5 ClimbMix Best (exp55) ASPECT_RATIO = 32 # was 64 (stock) DEPTH = max(2, _hp_defaults['depth'] - 3) # reduced from default HEAD_DIM = 128 # unchanged WINDOW_PATTERN = "LLLL" # was "SSSL" — all local attention MATRIX_LR = 0.045 # was 0.04 EMBEDDING_LR = 0.6 # unchanged UNEMBEDDING_LR = 0.0035 # was 0.004 SCALAR_LR = 0.5 # unchanged WEIGHT_DECAY = 0.2 # unchanged ADAM_BETAS = (0.8, 0.95) # unchanged WARMDOWN_RATIO = 0.2 # was 0.5 FINAL_LR_FRAC = 0.0 # unchanged # Result: 1.3018 | mem: 11.6 GB | tok/sec: 74.7K | MFU: 19.1% | 684 steps
Status: In progress (67/100 experiments) — will be updated when run completes.
See also: Cross-LLM Comparison | Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 ClimbMix (Mar 19)