FineWeb Edu High Agent Run Mar 21 2026

Autonomous experiment run on FineWeb-Edu-High — FineWeb-Edu filtered to educational quality score ≥ 4 (vs ≥ 3 for standard FineWeb-Edu). The agent ran 101 experiments on the Apple M5 Max (64 GB).

This run was designed to answer a specific question: is FineWeb-Edu's hyperparameter divergence data-dependent or path-dependent? The original FineWeb-Edu run started from full defaults, while all other datasets started from pre-optimized baselines and converged to nearly identical configs. By starting FineWeb-Edu-High from the same pre-optimized baseline, we can isolate the variable.

The answer: data-dependent. Starting from the same pre-optimized config as Climbmix, Cosmopedia-v2, and SlimPajama, FineWeb-Edu-High still diverged — finding 20 keeps and 1.97% improvement by systematically moving every parameter away from the cluster values. Educational text genuinely needs different hyperparameters.

This run was specifically designed to test the hypothesis from the Cross-Dataset Comparison:

Hypothesis: FineWeb-Edu's divergent hyperparameters were path-dependent — an artifact of starting from full defaults while other datasets started from pre-optimized baselines.

Result: Hypothesis rejected. FineWeb-Edu-High started from the exact same pre-optimized baseline (AR=32, MATRIX_LR=0.0435, EMBEDDING_LR=0.4, SCALAR_LR=0.4, WEIGHT_DECAY=0.15, WARMDOWN_RATIO=0.5) and systematically moved every single parameter to a different value:

Every single parameter moved. This is not noise — 20 keeps across 101 experiments, each incrementally optimizing in consistent directions. Educational text requires a fundamentally different optimization regime.

Both FineWeb-Edu variants diverge from the cluster, but they also differ from each other:

Both FineWeb-Edu variants share lower weight decay (0.10–0.12 vs 0.15) and lower warmdown ratio (0.47–0.48 vs 0.5). But the learning rate ratios differ significantly — the original FineWeb-Edu pushed MATRIX_LR much higher (0.053) while FineWeb-Edu-High found a more moderate value (0.047). This suggests the original run's high MATRIX_LR was partially path-dependent (overcorrecting from the default 0.04), even though the overall divergence direction is genuine.

The agent moved through five distinct phases:

Phase 1 — Learning Rate Exploration (exp1–34): Early batch size experiments crashed (exp1–3, 5–6). The agent then found MATRIX_LR=0.05 worked (exp4), spent 30 experiments exploring other LRs, and eventually fine-tuned MATRIX_LR to 0.047 (exp35). 5 crashes, 2 keeps.

Phase 2 — Learning Rate Ratios (exp35–56): Systematically adjusted SCALAR_LR (0.4→0.39), EMBEDDING_LR (0.4→0.375), and UNEMBEDDING_LR (0.0033→0.0032). This phase showed the data pulling all learning rates to different values than the cluster. 6 keeps.

Phase 3 — Regularization (exp57–66): WEIGHT_DECAY reduced from 0.15 to 0.10 in careful increments. Architecture experiments (DEPTH, ASPECT_RATIO, HEAD_DIM) all failed. 3 keeps.

Phase 4 — Schedule & Momentum (exp67–93): Two major breakthroughs — FINAL_LR_FRAC (0.0→0.085) and ADAM_BETAS β1 (0.8→0.66). These parameters hadn't been explored in any previous dataset run. 8 keeps.

Phase 5 — Architecture Fine-tuning (exp94–100): Final attempt at model changes. HEAD_DIM and ASPECT_RATIO experiments failed, but MLP_RATIO=4.25 (exp99) gave the last improvement. 1 keep.

# Architecture
ASPECT_RATIO = 32 # Same as cluster (hardware-determined)
DEPTH = 8 # Standard depth for M5 Max
MLP_RATIO = 4.25 # Wider than default (data-determined)
# Learning Rates (ALL different from cluster)
MATRIX_LR = 0.047 # Cluster: 0.0435
EMBEDDING_LR = 0.375 # Cluster: 0.4
SCALAR_LR = 0.39 # Cluster: 0.4
UNEMBEDDING_LR = 0.0032 # Cluster: 0.0033
# Regularization (different from cluster)
WEIGHT_DECAY = 0.10 # Cluster: 0.15
WARMDOWN_RATIO = 0.48 # Cluster: 0.5
FINAL_LR_FRAC = 0.085 # Cluster: 0.0
ADAM_BETAS = (0.66, 0.95) # Default: (0.8, 0.95)
# Performance
# val_bpb: 1.3463 | mem: 13.7 GB | steps: 525 | MFU: 17.7%

The consistent direction of divergence reveals a pattern:

1. Lower regularization (WEIGHT_DECAY 0.10 vs 0.15): Educational text is information-dense with diverse vocabulary. The model needs more freedom to fit complex patterns.

2. Non-zero final learning rate (FINAL_LR_FRAC 0.085): The model benefits from continuing to learn at the end of training — educational content has long-tail knowledge that benefits from continued gradient updates.

3. Lower momentum (β1 0.66 vs 0.8): Educational text has more diverse gradients (varied topics, technical vocabulary). Lower momentum prevents the optimizer from over-smoothing these diverse signals.

4. Shifted learning rate ratios: MATRIX_LR moves up while EMBEDDING_LR and SCALAR_LR move down. This suggests educational text benefits from relatively stronger updates to the attention/MLP weight matrices versus embeddings — possibly because the vocabulary is standard but the relationships between concepts are more complex.

See also: Cross-Dataset Comparison | FineWeb-Edu (Mar 17) | Climbmix (Mar 19) | Cosmopedia-v2 (Mar 20) | SlimPajama (Mar 20)