Four datasets tested on the same hardware (Apple M5 Max, 64 GB) with autonomous LLM-driven optimization. All runs converge to AR=32 architecture, and three of four converge to the exact same hyperparameters — confirming that most optimization parameters are hardware-determined, not data-determined.

Cross-Dataset Comparison

Note on baselines: Climbmix, Cosmopedia-v2, and SlimPajama started from pre-optimized baselines (AR=32, EMBEDDING_LR=0.4, SCALAR_LR=0.4 — carried from prior branch runs), while FineWeb-Edu started from full defaults. The low improvement rates for the pre-optimized runs reflect near-optimal starting points, not easier optimization landscapes.

All four datasets converge to AR=32. This is the defining finding of the multi-dataset comparison.

At AR=32, the model drops from ~50M to ~25M parameters, halving memory and doubling throughput. The ×ばつ more gradient steps in the 5-minute budget more than compensate for reduced per-step capacity. This is a hardware-optimal configuration for the M5 Max at this model scale — not dataset-specific.

The autoresearch loop is time-budgeted (5 minutes). The tradeoff:

• AR=64 (default): ~50M params, ~30K tok/sec, ~288 steps, 26.1 GB
• AR=32 (optimal): ~25M params, ~58K tok/sec, ~530 steps, 15.5 GB

The narrower model runs ×ばつ more gradient updates. For all four data distributions — raw web crawl, educational text, synthetic textbooks, and deduplicated web mix — the extra gradient steps outweigh the reduced model width. This finding is hardware-specific and would likely differ on GPUs with different compute/memory ratios.

The headline finding from adding SlimPajama: three out of four datasets converge to the exact same configuration. Only FineWeb-Edu diverges.

Bold values highlight the dominant cluster. Cosmopedia-v2 and SlimPajama are identical on all six parameters. Climbmix matches on 4/6, diverging only on UNEMBEDDING_LR and WEIGHT_DECAY. FineWeb-Edu is the outlier, differing on 5/6.

This data strongly suggests two classes of hyperparameters:

Hardware-determined (same across 3+ datasets):

• MATRIX_LR = 0.0435 (3/4 exact match)
• EMBEDDING_LR = 0.4 (3/4 exact match)
• SCALAR_LR = 0.4 (3/4 exact match)
• WARMDOWN_RATIO = 0.5 (3/4 exact match)

Data-sensitive (varies by dataset):

• UNEMBEDDING_LR: ranges from 0.003 (FineWeb-Edu) to 0.0041 (climbmix)
• WEIGHT_DECAY: ranges from 0.12 (FineWeb-Edu) to 0.2 (climbmix)

FineWeb-Edu was the only run starting from full defaults (AR=64, all default HPs). Its agent had to discover the entire optimal configuration from scratch, making different gradient-descent-style decisions at each step. The other three runs started from pre-optimized baselines (AR=32, EMBEDDING_LR=0.4, etc.), meaning the agent was doing fine-tuning rather than exploration.

This raises an interesting question: would FineWeb-Edu converge to the same config if re-run from the pre-optimized baseline? The evidence suggests yes — FineWeb-Edu's divergence may be path-dependent rather than data-dependent.

Cosmopedia (synthetic textbooks), SlimPajama (deduplicated web mix), and Climbmix (raw web crawl) are radically different data sources yet converge to nearly identical configurations. This is strong evidence that the compute/memory/step-count tradeoff of the M5 Max at this model scale dominates over data distribution effects.

Starting from full defaults, FineWeb-Edu had the most room to improve. The agent found 17 keeps across three phases: optimization tuning → architecture breakthrough → fine-tuning. The AR 64→32 reduction at exp34 was the largest single improvement in any run (−0.036 val_bpb).

With only 1 keep in 101 experiments, the climbmix run served primarily to validate that the inherited config was optimal. The agent exhaustively tested every parameter category and found only UNEMBEDDING_LR 0.004→0.0041 worth keeping. The 10.9% crash rate came from aggressive batch size experiments.

The agent spent 35 experiments on MATRIX_LR alone, systematically sweeping from 0.028 to 0.055 — all discarded. This "optimization trap" consumed a third of the experiment budget on a dead end. The actual improvements came late from WEIGHT_DECAY (exp88) and UNEMBEDDING_LR (exp98–102).

The cleanest run of all four datasets. The agent found the optimal config in just 3 experiments (all DEVICE_BATCH_SIZE reductions), then spent 97 experiments confirming nothing else works. Zero crashes, zero late breakthroughs — the pre-optimized baseline was essentially already optimal for this data. The agent explored MATRIX_LR micro-perturbations (20+ experiments), EMBEDDING_LR from 0.3 to 1.0, and WEIGHT_DECAY from 0.05 to 0.2, all without improvement.

SlimPajama is the hardest to compress. Despite being a cleaner, deduplicated subset of RedPajama, it achieves the highest (worst) val_bpb. This may reflect its source diversity — combining CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, and StackExchange creates a broad distribution that's harder to model than any single-source dataset.

Counter-intuitively, educationally-scored web text is harder to compress than raw web crawl (FineWeb-Edu > Climbmix). The quality filter selects information-dense documents with diverse vocabulary. Raw web crawl includes repetitive boilerplate and formulaic content that's easier to predict.

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0041, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.2
WARMDOWN_RATIO = 0.5 # mem: 15.5 GB | steps: 530 | MFU: 17.9%

ASPECT_RATIO = 32, MATRIX_LR = 0.053, EMBEDDING_LR = 0.59
UNEMBEDDING_LR = 0.003, SCALAR_LR = 0.3, WEIGHT_DECAY = 0.12
WARMDOWN_RATIO = 0.47 # mem: 15.5 GB | steps: ~530 | MFU: ~18%

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0033, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.15
WARMDOWN_RATIO = 0.5 # mem: 15.5 GB | steps: 543 | MFU: 18.3%

ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0033, SCALAR_LR = 0.4, WEIGHT_DECAY = 0.15
WARMDOWN_RATIO = 0.5 # mem: 13.7 GB | steps: 501 | MFU: 16.9%

Note: Cosmopedia-v2 and SlimPajama have identical optimal hyperparameters despite being fundamentally different data sources (synthetic textbooks vs. deduplicated web mix).

1. Architecture is hardware-determined. AR=32 is optimal across all four datasets on M5 Max. The compute/memory/step-count tradeoff at this model scale favors narrower, faster models regardless of data source.

2. Most optimization hyperparameters are also hardware-determined. MATRIX_LR, EMBEDDING_LR, SCALAR_LR, and WARMDOWN_RATIO converge to identical values across 3/4 datasets. The dominant factor is the step count achievable within the time budget, not the data distribution.

3. UNEMBEDDING_LR and WEIGHT_DECAY are the data-sensitive parameters. These show genuine variation: UNEMBEDDING_LR ranges from 0.003 to 0.0041, and WEIGHT_DECAY from 0.12 to 0.2. However, even these show clustering — three datasets agree on the same values.

4. FineWeb-Edu's divergence may be path-dependent. As the only run from full defaults, FineWeb-Edu took a different optimization path. The question of whether it would converge to the same config from a pre-optimized baseline remains open.

5. Pre-optimized baselines leave almost nothing to find. Keep rates of 1.0% (climbmix), 3.0% (SlimPajama), and 3.9% (cosmopedia) vs 19.3% (FineWeb-Edu from defaults) confirm that starting from a good config makes exhaustive search a validation exercise.

6. Late breakthroughs are real but not universal. FineWeb-Edu (exp34) and Cosmopedia (exp98–102) had important late improvements, but Climbmix and SlimPajama found their best results early. Running 100 experiments is worthwhile for discovery runs but may be excessive for validation runs.

7. Dataset compressibility spans a wide range. From 0.961 (Cosmopedia, synthetic) to 1.526 (SlimPajama, multi-source), the best achievable val_bpb varies by 60% — yet the optimal training configuration barely changes. The model adapts to data difficulty through its learned weights, not through its training hyperparameters.

See individual run pages: SlimPajama (Mar 20) | Cosmopedia-v2 (Mar 20) | Climbmix (Mar 19) | FineWeb-Edu (Mar 17) | Climbmix (Mar 16, legacy)