Five datasets tested on the same hardware (Apple M5 Max, 64 GB) with autonomous LLM-driven optimization. All runs converge to AR=32 architecture. Three of five converge to the exact same hyperparameters — and both FineWeb-Edu variants diverge, proving that educational text genuinely needs different optimization, not just a different starting point.

Cross-Dataset Comparison

Note on baselines: Climbmix, Cosmopedia-v2, SlimPajama, and FineWeb-Edu-High all started from pre-optimized baselines (AR=32, EMBEDDING_LR=0.4, SCALAR_LR=0.4 — carried from prior branch runs). Only the original FineWeb-Edu started from full defaults.

The key comparison: FineWeb-Edu-High started from the same pre-optimized baseline as the other three, yet still found 20 keeps and 1.97% improvement — proving the divergence is data-dependent.

All five datasets converge to AR=32. This is the defining finding of the multi-dataset comparison.

At AR=32, the model drops from ~50M to ~25M parameters, halving memory and doubling throughput. The ×ばつ more gradient steps in the 5-minute budget more than compensate for reduced per-step capacity. This is a hardware-optimal configuration for the M5 Max at this model scale — not dataset-specific.

The autoresearch loop is time-budgeted (5 minutes). The tradeoff:

• AR=64 (default): ~50M params, ~30K tok/sec, ~288 steps, 26.1 GB
• AR=32 (optimal): ~25M params, ~58K tok/sec, ~530 steps, 15.5 GB

The narrower model runs ×ばつ more gradient updates. For all five data distributions — raw web crawl, educational text (two quality tiers), synthetic textbooks, and deduplicated web mix — the extra gradient steps outweigh the reduced model width. This finding is hardware-specific and would likely differ on GPUs with different compute/memory ratios.

The headline finding: two distinct clusters emerge. Three datasets converge to one configuration, and both FineWeb-Edu variants form a second cluster — confirming a genuine data-dependent split.

Bold values highlight the dominant cluster. *Starred FineWeb-Edu values were not explored (the agent focused on other parameters from the defaults starting point).

Cluster A — "General text" (Climbmix, Cosmopedia-v2, SlimPajama): These three datasets — despite being radically different (raw web crawl, synthetic textbooks, deduplicated web mix) — converge to nearly identical configs. The optimization landscape for general text is flat and hardware-determined.

Cluster B — "Educational text" (FineWeb-Edu, FineWeb-Edu-High): Both FineWeb-Edu variants diverge from the cluster in consistent directions: lower weight decay, lower warmdown ratio, and shifted learning rate ratios. FineWeb-Edu-High additionally found lower momentum (β1=0.66) and non-zero final LR (0.085) — parameters the original FineWeb-Edu run didn't explore due to its different optimization path.

The original cross-dataset comparison raised this question:

Would FineWeb-Edu converge to the same config if re-run from the pre-optimized baseline?

Answer: No. FineWeb-Edu-High started from the exact same baseline as Climbmix, Cosmopedia-v2, and SlimPajama — and moved every single parameter away from the cluster values. The 20% keep rate (vs 1–4% for cluster datasets) confirms the optimization landscape is genuinely different for educational text.

However, the specific values are partially path-dependent. The two FineWeb-Edu variants found different learning rate ratios (MATRIX_LR 0.053 vs 0.047, EMBEDDING_LR 0.59 vs 0.375), showing that different starting points lead to different local optima within the "educational text" basin.

The consistent divergence patterns across both FineWeb-Edu runs reveal:

1. Lower regularization (WEIGHT_DECAY 0.10–0.12 vs 0.15–0.2): Educational text is information-dense with diverse vocabulary. The model needs more freedom to fit complex patterns.

2. Shorter warmdown (WARMDOWN_RATIO 0.47–0.48 vs 0.5): The model benefits from maintaining higher learning rates longer — educational content has diverse topics that benefit from continued exploration.

3. Lower momentum (β1 0.66 vs 0.8, from FineWeb-Edu-High): Educational text produces more diverse gradients. Lower momentum prevents over-smoothing these signals.

4. Non-zero final LR (FINAL_LR_FRAC 0.085, from FineWeb-Edu-High): Long-tail knowledge in educational content benefits from continued gradient updates at the end of training.

Starting from the pre-optimized baseline, the agent systematically moved every parameter — MATRIX_LR up, then all other LRs down, then WEIGHT_DECAY down, then FINAL_LR_FRAC up, then ADAM_BETAS down, then WARMDOWN_RATIO down, then MLP_RATIO up. Each change was small and validated, producing 20 keeps across 5 optimization phases. This is the highest keep rate of any pre-optimized run — ×ばつ higher than the next (Cosmopedia at 3.9%).

Starting from full defaults, FineWeb-Edu had the most room to improve. The agent found 17 keeps across three phases: optimization tuning → architecture breakthrough → fine-tuning. The AR 64→32 reduction at exp34 was the largest single improvement in any run (−0.036 val_bpb).

Climbmix (1 keep), Cosmopedia-v2 (4 keeps), and SlimPajama (3 keeps) all started from pre-optimized baselines and confirmed they were already near-optimal. The low keep rates reflect the quality of the starting config, not easier optimization landscapes.

FineWeb-Edu and FineWeb-Edu-High have nearly identical compressibility (1.342 vs 1.346) despite the higher quality threshold. The stricter filter (score ≥ 4 vs ≥ 3) selects for more information-dense text but doesn't fundamentally change the compression difficulty. Both are harder to compress than raw web crawl (Climbmix) — the quality filter selects information-dense documents with diverse vocabulary, while raw web crawl includes repetitive boilerplate that's easier to predict.

SlimPajama remains the hardest to compress. Despite being a cleaner, deduplicated subset of RedPajama, its source diversity (CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange) creates a broad distribution that's harder to model than any single-source dataset.

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0041, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.2
WARMDOWN_RATIO = 0.5 # mem: 15.5 GB | steps: 530 | MFU: 17.9%

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0033, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.15
WARMDOWN_RATIO = 0.5 # mem: 15.5 GB | steps: 543 | MFU: 18.3%

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0033, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.15
WARMDOWN_RATIO = 0.5 # mem: 13.7 GB | steps: 501 | MFU: 16.9%

ASPECT_RATIO = 32, MATRIX_LR = 0.053, EMBEDDING_LR = 0.59
UNEMBEDDING_LR = 0.003, SCALAR_LR = 0.3, WEIGHT_DECAY = 0.12
WARMDOWN_RATIO = 0.47 # mem: 15.5 GB | steps: ~530 | MFU: ~18%

ASPECT_RATIO = 32, MATRIX_LR = 0.047, EMBEDDING_LR = 0.375
UNEMBEDDING_LR = 0.0032, SCALAR_LR = 0.39, WEIGHT_DECAY = 0.10
WARMDOWN_RATIO = 0.48, FINAL_LR_FRAC = 0.085, ADAM_BETAS = (0.66, 0.95)
MLP_RATIO = 4.25 # mem: 13.7 GB | steps: 525 | MFU: 17.7%

Note: Cosmopedia-v2 and SlimPajama have identical optimal hyperparameters despite being fundamentally different data sources (synthetic textbooks vs. deduplicated web mix). Both FineWeb-Edu variants diverge from the cluster but also from each other — indicating multiple local optima within the "educational text" optimization basin.

1. Architecture is hardware-determined. AR=32 is optimal across all five datasets on M5 Max. This finding is robust across general text, educational text, synthetic text, and code-adjacent content.

2. Most optimization hyperparameters are hardware-determined for general text. MATRIX_LR, EMBEDDING_LR, SCALAR_LR, and WARMDOWN_RATIO converge to identical values across 3/5 datasets (the non-educational ones).

3. Educational text is genuinely different. Both FineWeb-Edu variants diverge from the cluster — the original from defaults and the high-quality variant from pre-optimized baselines. The divergence direction is consistent: lower regularization, lower momentum, non-zero final LR, shifted learning rate ratios.

4. The path-dependence question is resolved. FineWeb-Edu's divergence is data-dependent (educational text needs different HPs), but the specific values are partially path-dependent (the two variants found different learning rate ratios). The direction of divergence is real; the exact magnitude depends on starting point.

5. FineWeb-Edu-High discovered new parameters. ADAM_BETAS β1 (0.66), FINAL_LR_FRAC (0.085), and MLP_RATIO (4.25) were not explored by other runs. These parameters may represent additional optimization opportunities for educational text — or for all datasets if tested.

6. Keep rate is a signal of optimization landscape. Educational text: 19–20% keep rate. General text: 1–4% keep rate. The educational text optimization landscape has more meaningful gradients from the cluster starting point, confirming it occupies a different basin.

7. Dataset compressibility is independent of optimal hyperparameters. val_bpb ranges from 0.961 to 1.526 across datasets, but three of five converge to the same config. The model adapts to data difficulty through learned weights, not training hyperparameters — except for educational text, which requires a fundamentally different training regime.

These results were all generated by Claude Sonnet 4.0. A parallel series using Claude Sonnet 4.6 has completed two datasets:

• ClimbMix: Sonnet 4.6 found 8 keeps vs Sonnet 4.0's 1, with ×ばつ more improvement from baseline.
• FineWeb-Edu: Both models converge to nearly identical val_bpb (1.3416 vs 1.3424) but via completely different configurations — 8 of 10 key parameters differ. This proves FineWeb-Edu's optimization landscape has multiple near-equivalent optima.

Read the full Cross-LLM Comparison →

See individual run pages: Sonnet 4.6 FineWeb-Edu (Mar 22) | Sonnet 4.6 ClimbMix (Mar 22) | FineWeb-Edu-High (Mar 21) | SlimPajama (Mar 20) | Cosmopedia-v2 (Mar 20) | Climbmix (Mar 19) | FineWeb-Edu (Mar 17) | Climbmix (Mar 16, legacy)