FineWeb Edu Agent Run Mar 17 2026

First multi-dataset experiment: 101 autonomous experiments on FineWeb-Edu (10BT sample) using the TUI Dashboard agent mode on the Apple M5 Max. The agent achieved an 8.1% improvement over baseline, discovering that FineWeb-Edu requires a fundamentally different architecture than the default climbmix dataset — a smaller, faster model that trades width for throughput.

A dramatic late-stage breakthrough at experiment 89 shattered a 20-experiment plateau, dropping val_bpb from 1.342 to 1.295 via a tiny unembedding learning rate change.

FineWeb-Edu Agent Run Results — Mar 17, 2026

The agent found 19 improvements, with two major breakthroughs and many incremental gains:

# Architecture (DIFFERENT from climbmix optimal)
DEPTH = 8
ASPECT_RATIO = 32 # ↓ from 64 (exp32→33→34) — HALVED
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
MLP_RATIO = 4.0
# Optimization (agent-tuned)
TOTAL_BATCH_SIZE = 32768
DEVICE_BATCH_SIZE = 16
MATRIX_LR = 0.053 # ↑ from 0.04 (exp12→53→64)
EMBEDDING_LR = 0.59 # ↓ from 0.6 (exp52→58)
UNEMBEDDING_LR = 0.0028 # ↓ from 0.004 (exp13→89→90)
SCALAR_LR = 0.3 # ↓ from 0.5 (exp11)
WEIGHT_DECAY = 0.12 # ↓ from 0.2 (exp45→46→47→48)
ADAM_BETAS = (0.8, 0.95) # unchanged
WARMUP_RATIO = 0.0 # unchanged
WARMDOWN_RATIO = 0.47 # initially ↓ to 0.4, then ↑ to 0.47 (exp14→67→75)
FINAL_LR_FRAC = 0.0 # unchanged

Phase 1 — Conservative LR tuning (exp0–31): The agent started by testing learning rate and schedule changes at the default architecture. Most early experiments hurt performance — FineWeb-Edu punished aggressive LR increases that worked on climbmix. The agent learned that this dataset wants lower learning rates (SCALAR_LR 0.5→0.3, UNEMBEDDING_LR 0.004→0.003). Best after 31 experiments: 1.3977 — only 0.8% better than baseline.

Phase 2 — The architecture breakthrough (exp32–67): The pivotal discovery was ASPECT_RATIO reduction. Shrinking from 64→48→40→32 cut the model from 50M to a smaller, faster model that runs ×ばつ more gradient steps (565 vs 288) in the same 5-minute budget, while using only 15.5 GB instead of 26.1 GB. Exp34 alone (AR 40→32) produced the single biggest improvement of -0.036 val_bpb. The agent then systematically ground through weight decay (0.2→0.12) and fine-grained LR tuning.

Phase 3 — The plateau (exp68–88): 21 experiments yielded only 1 keep (WARMDOWN_RATIO 0.45→0.47, gaining 0.0003). The agent was clearly stuck at a local optimum around 1.342, trying ever-smaller perturbations with no payoff.

Phase 4 — The late breakthrough (exp89–100): On experiment 89, a seemingly tiny change — UNEMBEDDING_LR from 0.003 to 0.0029 — produced a massive 0.047 drop in val_bpb, shattering the plateau. This is remarkable: a 0.0001 LR delta caused a 3.5% improvement. Exp90 refined it to 0.0028, reaching the final best of 1.2951. The remaining 10 experiments couldn't improve further.

1. Architecture matters more than hyperparameters on this dataset. ASPECT_RATIO=32 was the biggest single lever, contributing 0.052 of the total 0.114 improvement (46%). The smaller model trades capacity for throughput — more gradient steps in the fixed time budget.

2. FineWeb-Edu wants lower learning rates. Unlike climbmix where MATRIX_LR went from 0.04→0.06 (×ばつ), here the optimal is 0.053 (×ばつ). SCALAR_LR dropped 40% (0.5→0.3). The dataset's educational text has more consistent structure, requiring less aggressive optimization.

3. The unembedding LR is a critical, non-linear parameter. The late breakthrough at exp89 shows a discontinuity: 0.0030→0.0029 changed val_bpb by 0.047, while 0.0029→0.0028 only changed it by 0.0002. This suggests a phase transition in how the model maps hidden states to token predictions on educational text.

4. Weight decay optimal is dataset-dependent. Climbmix preferred WD=0.02 (90% below default). FineWeb-Edu prefers WD=0.12 (40% below default) — the model needs more regularization on this structured educational text.

5. Memory efficiency is a side benefit. The optimal architecture uses only 15.5 GB (24% of 64 GB) vs 26.1 GB for climbmix's optimal. This leaves headroom for larger batch sizes or deeper models in future experiments.

The run naturally split into three memory regimes as the architecture evolved:

The optimal model configuration is dataset-dependent. Climbmix (web crawl data) optimizes best with a wide 50M-parameter model at the default architecture. FineWeb-Edu (educationally-scored text) optimizes best with a narrow model at half the width, running ×ばつ more gradient steps. The hyperparameter preferences also diverge significantly — learning rate directions, weight decay magnitude, and warmdown scheduling all differ.

This validates the multi-dataset experiment framework: you cannot assume hyperparameters found on one dataset transfer to another.

Two experiments crashed (2.0% crash rate):

• exp3 (TOTAL_BATCH_SIZE reduction) — Invalid configuration
• exp9 (DEPTH=12) — OOM or architecture incompatibility

Both were early exploratory moves. The agent avoided similar failures afterward.

uv run convert_dataset.py fineweb-edu --num-shards 10 --num-source 3
uv run prepare.py
uv run dashboard.py --agent --tag fineweb-edu --max 100

See Multi-Dataset Experiments for dataset conversion details and TUI Dashboard for agent mode documentation.

• Branch: feature/multi-dataset
• Dataset: HuggingFaceFW/fineweb-edu (sample/10BT)
• Agent: Claude Sonnet (autonomous via TUI dashboard)
• Date: March 17, 2026