Multi Dataset Experiments

The multi-dataset framework enables running the full autoresearch experiment loop across different training datasets to answer a key question: do optimal hyperparameters transfer across datasets?

Early results show they do not — the optimal architecture for climbmix (ASPECT_RATIO=64, 26 GB memory) is completely different from FineWeb-Edu (ASPECT_RATIO=32, 15.5 GB memory). Each dataset has its own performance landscape.

This work was inspired by karpathy/autoresearch PR #303 by Dean Sharon, which provides a guide for evaluating experiment results at scale. The noise floor estimation and Pareto efficiency concepts from that PR are directly applicable to cross-dataset comparison. The guide has been adapted for Apple Silicon context in docs/evaluating-results.md.

convert_dataset.py # Download + convert any dataset to shard format
run_suite.py # Orchestrate multi-dataset sweep
compare_datasets.py # Cross-dataset analysis + charts
docs/
 evaluating-results.md # Guide for analyzing results at scale
results/
 climbmix/results.tsv
 fineweb-edu/results.tsv
 cosmopedia-v2/results.tsv
 ...

Data isolation: Each dataset gets its own profile in ~/.cache/autoresearch/profiles/<name>/ containing both data shards and a BPE tokenizer trained on that dataset's text. Switching between datasets is an instant copy — no re-downloading or re-tokenizing.

Results isolation: Each dataset's experiments go into results/<dataset>/results.tsv. Cross-contamination is impossible.

Fair comparison: The val_bpb metric (bits per byte) is tokenizer-independent, so results are comparable across datasets. Each dataset uses the same 5-minute time budget, same model defaults, and same evaluation harness.

# Check status of all datasets
uv run run_suite.py --status
# Prepare a single dataset (download + tokenize, no experiments)
uv run run_suite.py --dataset cosmopedia-v2 --prepare-only
# Run the full suite
uv run run_suite.py --max-experiments 80
# Run a single dataset
uv run run_suite.py --dataset slimpajama
# Skip datasets that already have results
uv run run_suite.py --skip-completed
# Switch active dataset manually
uv run run_suite.py --load-profile fineweb-edu
# Compare results across datasets
uv run compare_datasets.py
uv run compare_datasets.py --summary # text only

Key insight: FineWeb-Edu's educational text benefits from a completely different architecture — a smaller, faster model (ASPECT_RATIO=32) that can fit nearly 2x the gradient steps in the same time budget. This smaller model also uses less than half the memory. The optimal learning rates, weight decay, and scheduling are all different.

This strongly suggests that hyperparameters do not transfer across datasets and each dataset needs its own optimization sweep.

• Available on: master (merged from feature/multi-dataset)
• Date: March 17, 2026