Cosmopedia v2 Agent Run Mar 20 2026

Autonomous experiment run on Cosmopedia-v2 — a synthetic textbook and blogpost dataset generated by Mixtral, sourced from the SmolLM-Corpus. The agent ran 103 experiments on the Apple M5 Max (64 GB).

Starting from a pre-optimized baseline (AR=32, EMBEDDING_LR=0.4, SCALAR_LR=0.4 from prior branch runs), the agent found only 0.35% improvement — confirming the configuration was already near-optimal for this data.

Cosmopedia-v2 Agent Run Results

# Architecture (pre-optimized from prior runs)
DEPTH = 8
ASPECT_RATIO = 32 # ↓ from 64 (prior runs)
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
# Optimization (agent-tuned)
MATRIX_LR = 0.0435 # ↑ from 0.04 (prior runs)
EMBEDDING_LR = 0.4 # ↓ from 0.6 (prior runs)
UNEMBEDDING_LR = 0.0033 # ↓ from 0.004 (exp98, exp102)
SCALAR_LR = 0.4 # ↓ from 0.5 (prior runs)
WEIGHT_DECAY = 0.15 # ↓ from 0.2 (exp88)
ADAM_BETAS = (0.8, 0.95) # Default
WARMUP_RATIO = 0.0 # Default
WARMDOWN_RATIO = 0.5 # Default
FINAL_LR_FRAC = 0.0 # Default
# Performance
# Peak memory: 15.5 GB | Steps: 543 | tok/sec: 59,234 | MFU: 18.3%

This run demonstrates what happens when the agent starts from an already well-tuned configuration. Across 103 experiments testing every parameter category, only 4 produced improvements — and all were small (largest: −0.0014 val_bpb from batch size reduction).

The 3.9% keep rate is the lowest of any run in the project, confirming the pre-optimized baseline left almost no room for improvement.

The agent spent 35 consecutive experiments (exp4–exp43, minus a few diversions) systematically sweeping MATRIX_LR from 0.0435 down to 0.028. Every single one was discarded — many came close to the best (0.9653–0.9687 vs best 0.9625) but none beat it. This is a case where the agent correctly identified a promising direction but couldn't overcome the pre-optimized baseline.

Tested MATRIX_LR values: 0.055, 0.048, 0.045, 0.0425, 0.042, 0.041, 0.04075, 0.0405, 0.04025, 0.040, 0.0395, 0.039, 0.0385, 0.038, 0.0375, 0.037, 0.0365, 0.036, 0.0355, 0.035, 0.0345, 0.034, 0.0335, 0.033, 0.0325, 0.032, 0.031, 0.030, 0.029, 0.028 — 30 values tested, 0 kept.

The meaningful improvements came late:

• exp88 (of 103): WEIGHT_DECAY 0.2→0.15 — first non-batch-size keep after 85 fruitless experiments
• exp98–102: UNEMBEDDING_LR 0.0041→0.0035→0.0033 — two consecutive keeps in the final 5 experiments

This mirrors the FineWeb-Edu pattern where UNEMBEDDING_LR produced a late breakthrough. The unembedding layer appears to be a consistently underexplored parameter across datasets.

The best val_bpb of 0.9606 is dramatically lower than climbmix (1.296) or FineWeb-Edu (1.342). This is expected: Cosmopedia-v2 is synthetic textbook data generated by Mixtral with consistent style, vocabulary, and structure. The model can exploit these regularities for much better compression. Real web crawl has far more entropy.

• Branch: autoresearch/mar19-cosmopedia-v2
• Agent: Claude Sonnet (autonomous via headless runner)
• Date: March 19–20, 2026