Three datasets tested on the same hardware (Apple M5 Max, 64 GB) with autonomous LLM-driven optimization. All runs converge to AR=32 architecture but diverge on optimization hyperparameters — confirming that architecture is hardware-determined while optimization is data-determined.

Cross-Dataset Comparison

Note on baselines: Climbmix and Cosmopedia-v2 started from pre-optimized baselines (AR=32, EMBEDDING_LR=0.4, SCALAR_LR=0.4 — carried from prior branch runs), while FineWeb-Edu started from full defaults. The low improvement rates for climbmix and cosmopedia reflect near-optimal starting points, not easier optimization landscapes.

All three datasets converge to AR=32. This is the defining finding of the multi-dataset comparison.

At AR=32, the model drops from ~50M to ~25M parameters, halving memory and doubling throughput. The ×ばつ more gradient steps in the 5-minute budget more than compensate for reduced per-step capacity. This is a hardware-optimal configuration for the M5 Max at this model scale — not dataset-specific.

The autoresearch loop is time-budgeted (5 minutes). The tradeoff:

• AR=64 (default): ~50M params, ~30K tok/sec, ~288 steps, 26.1 GB
• AR=32 (optimal): ~25M params, ~58K tok/sec, ~530 steps, 15.5 GB

The narrower model runs ×ばつ more gradient updates. For all three data distributions — raw web crawl, educational text, and synthetic textbooks — the extra gradient steps outweigh the reduced model width. This finding is hardware-specific and would likely differ on GPUs with different compute/memory ratios.

While architecture converged, optimization hyperparameters tell three different stories:

EMBEDDING_LR is the clearest dataset signal. Climbmix and Cosmopedia both converge to 0.4 (33% below default), while FineWeb-Edu stays near default at 0.59. Web crawl and synthetic text both benefit from conservative embedding updates, while educationally-filtered text does not.

UNEMBEDDING_LR splits the other way: climbmix pushes it up (0.0041), while both FineWeb-Edu (0.003) and Cosmopedia (0.0033) push it down. The unembedding layer was the source of late breakthroughs on both FineWeb-Edu and Cosmopedia — it appears to be a consistently underexplored parameter that rewards fine-grained search.

WEIGHT_DECAY shows a three-way split: climbmix keeps the default (0.2), FineWeb-Edu drops to 0.12, and Cosmopedia finds 0.15. Web crawl's high entropy needs full regularization; curated and synthetic text benefit from modest reductions.

WARMDOWN_RATIO is the most stable parameter — only FineWeb-Edu adjusted it (0.47), while both climbmix and Cosmopedia kept the default 0.5.

Climbmix (raw web crawl) and Cosmopedia-v2 (synthetic textbooks) share nearly identical optimal configs despite being radically different data sources:

The only clear divergence is UNEMBEDDING_LR (opposite direction from default) and a slight WEIGHT_DECAY difference. This suggests that many optimization hyperparameters are more hardware-dependent than data-dependent — the compute/memory/step-count tradeoff dominates.

Starting from full defaults, FineWeb-Edu had the most room to improve. The agent found 17 keeps across three phases: optimization tuning → architecture breakthrough → fine-tuning. The AR 64→32 reduction at exp34 was the largest single improvement in any run (−0.036 val_bpb).

With only 1 keep in 101 experiments, the climbmix run served primarily to validate that the inherited config was optimal. The agent exhaustively tested every parameter category and found only UNEMBEDDING_LR 0.004→0.0041 worth keeping.

The agent spent 35 experiments on MATRIX_LR alone, systematically sweeping from 0.028 to 0.055 — all discarded. This "optimization trap" consumed a third of the experiment budget on a dead end. The actual improvements came late from WEIGHT_DECAY (exp88) and UNEMBEDDING_LR (exp98–102).

Counter-intuitively, educationally-scored web text is harder to compress than raw web crawl. FineWeb-Edu's quality filter selects for information-dense documents with diverse vocabulary, making next-token prediction more challenging. Raw web crawl includes repetitive boilerplate, ads, and formulaic content that's easier to model.

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0041, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.2
WARMDOWN_RATIO = 0.5 # mem: 15.5 GB | steps: 530 | MFU: 17.9%

ASPECT_RATIO = 32, MATRIX_LR = 0.053, EMBEDDING_LR = 0.59
UNEMBEDDING_LR = 0.003, SCALAR_LR = 0.3, WEIGHT_DECAY = 0.12
WARMDOWN_RATIO = 0.47 # mem: 15.5 GB | steps: ~530 | MFU: ~18%

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0033, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.15
WARMDOWN_RATIO = 0.5 # mem: 15.5 GB | steps: 543 | MFU: 18.3%

1. Architecture is hardware-determined. AR=32 is optimal across all three datasets on M5 Max. The compute/memory/step-count tradeoff at this model scale favors narrower, faster models regardless of data source.

2. Most optimization hyperparameters are also hardware-determined. MATRIX_LR, EMBEDDING_LR, SCALAR_LR, and WARMDOWN_RATIO converge to similar or identical values across datasets. The dominant factor is the step count achievable within the time budget, not the data distribution.

3. UNEMBEDDING_LR and WEIGHT_DECAY are the data-sensitive parameters. These show genuine divergence: UNEMBEDDING_LR moves in opposite directions for different data, and WEIGHT_DECAY ranges from 0.12 (FineWeb-Edu) to 0.2 (climbmix). These should be dataset-specific tuning targets.

4. Pre-optimized baselines leave almost nothing to find. Keep rates of 1% (climbmix) and 3.9% (cosmopedia) vs 19.3% (FineWeb-Edu from defaults) confirm that starting from a good config makes exhaustive search a validation exercise rather than a discovery exercise.

5. Late breakthroughs are real. Both FineWeb-Edu (exp34) and Cosmopedia (exp98–102) had their most important improvements in the final third of the run. Running only 50 experiments would miss these.

See individual run pages: Climbmix (Mar 19) | FineWeb-Edu (Mar 17) | Cosmopedia-v2 (Mar 20) | Climbmix (Mar 16, legacy)