Sonnet 4.6 FineWeb Edu Agent Run Mar 22 2026

Second cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing FineWeb-Edu on the same hardware previously used by Claude Sonnet 4.0. This is the most informative comparison yet because FineWeb-Edu was Sonnet 4.0's strongest dataset (19.3% keep rate, 17 keeps).

Key finding: Both models converge to essentially the same final val_bpb (1.3416 vs 1.3424), but via completely different optimization paths. Sonnet 4.6 started from a much better baseline (1.371 vs 1.409) — meaning the defaults improved between model generations — and still found 18 keeps. The final configs are substantially different, suggesting multiple near-equivalent optima in the educational text optimization landscape.

This run started from reset defaults, not the same defaults as the Sonnet 4.0 FineWeb-Edu run:

• Sonnet 4.6 baseline: 1.3710 (from defaults — better defaults than S4.0)
• Sonnet 4.0 baseline: 1.4088 (from defaults — original defaults)
• Sonnet 4.0's absolute best: 1.3424 — very close to Sonnet 4.6's 1.3416
• Sonnet 4.0 achieved 4.71% improvement vs Sonnet 4.6's 2.14% — but S4.0 had more room to work with

The different baselines reflect improved default hyperparameters between model generations. Both models converge to nearly identical absolute performance.

Phase 1 — Initial Exploration (exp1–10, 10 experiments): Broad sweeps of batch size, LR values, weight decay, warmdown. Only exp3 (smaller device batch) kept. Established that more gradient steps help.

Phase 2 — LR Schedule Discovery (exp11–18, 8 experiments): Three consecutive keeps: WARMDOWN_RATIO=0.75 (exp11), FINAL_LR_FRAC=0.1 (exp15), MATRIX_LR=0.06 (exp18). Sonnet 4.6 recognized the LR schedule as the most productive optimization dimension.

Phase 3 — Architecture Search (exp19–33, 15 experiments): Tested depth changes, Adam betas, various LRs. Then discovered ASPECT_RATIO as a lever — tried 24, then 32 (two successive keeps). AR=32 gave the biggest single-experiment jump (−0.005 bpb).

Phase 4 — Re-optimization for Wide Model (exp34–46, 13 experiments): Tried re-tuning warmdown, betas, weight decay for AR=32. All discarded — the model's old hyperparams transferred well to the wider architecture.

Phase 5 — Systematic Optimizer Tuning (exp47–57, 11 experiments): Four consecutive keeps on Adam β2 (0.95→0.96→0.97→0.98). Also refined β1 to 0.68, scalar LR to 0.25, matrix LR to 0.055. Most productive phase by count.

Phase 6 — Deep Fine-tuning (exp58–78, 21 experiments): Exhaustive sweeps around the current best. Found UNEMBEDDING_LR=0.006 (exp73) and refined β2 to 0.975 (exp78). Increasingly small deltas.

Phase 7 — Final Polish (exp79–100, 22 experiments): Warmdown fine-tuned to 0.77, embedding LR to 0.45, scalar LR to 0.23, β1 to 0.67. Four final keeps squeezing the last ~0.002 bpb.

MFU rose 21.8%, throughput rose 21.6%, steps rose 21.7%. Memory dropped from 15.5 to 14.5 GB. Every optimization that improved val_bpb also improved hardware utilization.

1 crash (1.0%) — the lowest crash rate of any run:

Sonnet 4.6 showed excellent crash avoidance on FineWeb-Edu, learning from the ClimbMix run's batch size failures.

# Architecture
ASPECT_RATIO = 32
DEPTH = 8
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
MLP_RATIO = 4.0
# Optimization (9 parameters changed from defaults)
DEVICE_BATCH_SIZE = default / 2 # More gradient steps
MATRIX_LR = 0.055 # Default: 0.05
SCALAR_LR = 0.23 # Default: 0.3
EMBEDDING_LR = 0.45 # Default: 0.4
UNEMBEDDING_LR = 0.006 # Default: 0.004
ADAM_BETAS = (0.67, 0.975) # Default: (0.71, 0.95)
WARMDOWN_RATIO = 0.77 # Default: 0.63
FINAL_LR_FRAC = 0.1 # Default: 0.0
WEIGHT_DECAY = 0.2 # Default (unchanged)
# Performance
# val_bpb: 1.3416 | mem: 14.5 GB | steps: 533 | MFU: 17.9%

The most striking finding is that Sonnet 4.0 and Sonnet 4.6 arrive at nearly identical final val_bpb (1.3424 vs 1.3416) via completely different configurations:

Only architecture (AR=32) and matrix LR are shared. Every other parameter is substantially different. This proves that FineWeb-Edu's optimization landscape has multiple near-equivalent local optima — different combinations of warmdown schedule, momentum, and learning rate ratios can achieve the same performance.

This contrasts with ClimbMix, where the optimization landscape is relatively flat (general text datasets converge to similar configs regardless of starting point).

See also: Cross-LLM Comparison | FineWeb-Edu Sonnet 4.0 (Mar 17) | Cross-Dataset Comparison