Sonnet 4.6 FineWeb Edu High Agent Run Mar 25 2026

Dave Graham edited this page Mar 25, 2026 · 1 revision

Sonnet 4.6 FineWeb-Edu-High Agent Run — Mar 25, 2026 (M5 Max)

Overview

Fifth and final cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing FineWeb-Edu-High on the same hardware previously used by Claude Sonnet 4.0. This completes the full 5-dataset comparison suite.

Key finding: Sonnet 4.6 beats Sonnet 4.0's best (1.3345 vs 1.3463) while maintaining high optimization quality — 13 keeps with a spectacular β2 walk at the end (5 consecutive keeps from 0.98→0.964). FineWeb-Edu-High confirms that educational text provides a rich optimization landscape for both models.

Hardware

Spec	Value
Chip	Apple M5 Max
Unified Memory	64 GB
GPU Cores	40
Backend	MLX
Time Budget	5 minutes per experiment

Summary Statistics

Metric	Value
Total Experiments	101 (exp0–exp100)
Kept	13 (12.9%)
Discarded	86 (85.1%)
Crashed	2 (2.0%)
Best val_bpb	1.3345 (exp100)
Baseline val_bpb	1.3478 (defaults)
Total Improvement	−0.0133 (0.99%)
LLM Backend	Claude Sonnet 4.6
Results	`results/sonnet-4-6/fineweb-edu-high/`

Baseline Context

Metric	Sonnet 4.0	Sonnet 4.6
Baseline val_bpb	1.3730 (pre-optimized)	1.3478 (defaults)
Best val_bpb	1.3463	1.3345
Improvement	−1.97%	−0.99%
Keeps	20 (20.0%)	13 (12.9%)
Crashes	5 (5.0%)	2 (2.0%)

Sonnet 4.6 achieves a significantly better absolute val_bpb (1.3345 vs 1.3463 — a 0.9% gap). It started from a much better baseline (1.3478 vs 1.3730), so its percentage improvement is smaller. The keep rates are both high, confirming educational text has a rich optimization landscape.

Cumulative Improvements (Kept Experiments)

Exp	val_bpb	Δ from Baseline	Description	Key Change
exp0	1.3478	—	Baseline (defaults)	Starting point
exp24	1.3459	−0.14%	WINDOW_PATTERN → all sliding-window	More local attention
exp26	1.3454	−0.18%	EMBEDDING_LR 0.30→0.60	Faster embeddings
exp28	1.3428	−0.37%	ADAM β2 0.96→0.98	More stable gradients
exp33	1.3419	−0.44%	SCALAR_LR 0.13→0.18	Higher scalar LR
exp37	1.3408	−0.52%	WARMDOWN_RATIO 0.70→0.75	Extended high-LR phase
exp58	1.3385	−0.69%	ADAM β1 0.55→0.50	Less momentum
exp60	1.3376	−0.76%	ADAM β1 0.50→0.45	Even less momentum
exp75	1.3367	−0.82%	ADAM β2 0.98→0.975	β2 walk begins
exp76	1.3366	−0.83%	ADAM β2 0.975→0.970	β2 walk continues
exp88	1.3357	−0.90%	WEIGHT_DECAY 0.1→0.05	Lower regularization
exp90	1.3353	−0.93%	WEIGHT_DECAY 0.05→0.08	Fine-tune WD
exp97	1.3353	−0.93%	ADAM β2 0.970→0.968	β2 walk resumes
exp99	1.3349	−0.96%	ADAM β2 0.968→0.966	β2 walk continues
exp100	1.3345	−0.99%	ADAM β2 0.966→0.964	β2 walk final

The β2 Walk — Signature Finding

The most striking optimization pattern: 5 consecutive keeps from systematic β2 reduction.

β2	val_bpb	Exp	Outcome
0.96 (default)	1.3478	exp0	Baseline
0.98	1.3428	exp28	Keep (initial increase)
0.985	1.3475	exp41	Discard
0.975	1.3367	exp75	Keep
0.970	1.3366	exp76	Keep
0.968	1.3353	exp97	Keep
0.966	1.3349	exp99	Keep
0.964	1.3345	exp100	Keep
0.960	1.3367	exp77	Discard
0.965	1.3372	exp79	Discard

The optimal β2 is precisely 0.964 — lower than the initial increase to 0.98 but higher than 0.96. This non-monotonic path (up then down) is a sophisticated optimization trajectory that Sonnet 4.0 never explored.

Best Configuration

# Sonnet 4.6 FineWeb-Edu-High Best (exp100)
ASPECT_RATIO = 21 # unchanged from defaults
MATRIX_LR = 0.068 # unchanged
EMBEDDING_LR = 0.60 # was 0.30
SCALAR_LR = 0.18 # was 0.13
UNEMBEDDING_LR = 0.006 # unchanged
WEIGHT_DECAY = 0.08 # was 0.1
WARMDOWN_RATIO = 0.75 # was 0.70
FINAL_LR_FRAC = 0.05 # was 0.01
ADAM_BETAS = (0.45, 0.964) # was (0.55, 0.96)
WINDOW_PATTERN = "SSSS" # was "SSSL" — all sliding-window
# Result: 1.3345 | mem: 14.5 GB | tok/sec: 60.3K | MFU: 17.9% | 551 steps

Navigation Menu

Sonnet 4.6 FineWeb Edu High Agent Run Mar 25 2026