SlimPajama Agent Run Mar 20 2026

Autonomous experiment run on SlimPajama — a deduplicated, cleaned version of RedPajama (627B tokens from CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange). The agent ran 101 experiments on the Apple M5 Max (64 GB).

Starting from a pre-optimized baseline (AR=32, EMBEDDING_LR=0.4, SCALAR_LR=0.4 from prior branch runs), the agent found only 1.0% improvement — all from reducing DEVICE_BATCH_SIZE to maximize gradient steps. 97 subsequent hyperparameter experiments were all discarded.

# Architecture (pre-optimized from prior runs)
ASPECT_RATIO = 32 # Half model width, ×ばつ throughput
DEPTH = 8 # Standard depth for M5 Max
# Optimization (unchanged from pre-optimized baseline)
MATRIX_LR = 0.0435
EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.0033
SCALAR_LR = 0.4
WEIGHT_DECAY = 0.15
WARMDOWN_RATIO = 0.5
# Key finding
DEVICE_BATCH_SIZE = 1 # Only change that mattered
# Performance
# mem: 13.7 GB | tok/sec: 54,593 | MFU: 16.9% | steps: 501

This is the cleanest result of any dataset run. The agent found the optimal configuration in just 3 experiments — all DEVICE_BATCH_SIZE reductions — then spent 97 more experiments confirming nothing else improves performance.

Several experiments came close to but couldn't beat exp3's 1.5259:

Weight decay around 0.135 and MATRIX_LR around 0.04352 showed tantalizing signal but couldn't overcome the noise floor.

The agent systematically swept EMBEDDING_LR from 0.3 to 1.0:

The current 0.4 is at the exact optimum. Higher values showed a non-monotonic pattern with a curious secondary dip around 0.8 — interesting but not actionable.

Like Cosmopedia-v2, the agent fell into fine-grained MATRIX_LR search, testing 0.041–0.05 plus ultra-fine steps around 0.0431–0.04354:

• 20+ experiments on MATRIX_LR perturbations
• Best near-miss: exp75 at 0.04352 (1.5309) — 0.3% worse than baseline
• The parameter is at a sharp local optimum at 0.0435

SlimPajama is the only dataset to complete a full run with zero crashes. The agent never attempted extreme configurations that would cause OOM or NaN — the pre-optimized baseline's stability may have discouraged risky experiments.

SlimPajama's best configuration is identical to Cosmopedia-v2's optimal config:

Three out of four datasets converge to the exact same configuration. Only FineWeb-Edu diverges. This strongly suggests the shared config is hardware-optimal, not dataset-optimal.

See also: Cross-Dataset Comparison | Cosmopedia-v2 (Mar 20) | Climbmix (Mar 19) | FineWeb-Edu (Mar 17)