History

Complete 4-model ClimbMix comparison — Opus finishes 3rd, Haiku wins Final ranking: Haiku (1.2953) > S4.6 (1.3093) > Opus (1.3569) > S4.0 (1.3588) Architecture discovery correlates perfectly with results, not model cost. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Update Opus 4.6 analysis to 70/100 exp — highest keep rate, still architecturally blind Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Opus 4.6 early analysis (23/100 exp) — 26% crash rate, "bigger is better" bias Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add S4.0 + S4.6 stock ClimbMix analysis — complete 3-model comparison (Haiku wins) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Sonnet 4.0 stock baseline ClimbMix: 1.3588 — architecture blindness vs Haiku's 1.2953 S4.0 never changed AR/depth/window in 100 experiments. Haiku's architecture discoveries (AR=32, depth reduction, LLLL) tripled throughput and won decisively. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Final Haiku 4.5 ClimbMix results: Haiku wins (1.2953) — beats both Sonnets from stock defaults Late comeback via β1+MATRIX_LR synergy at exp90-91. 10 keeps, 3 crashes, 8.04% improvement from stock AR=64 baseline. Full 100-experiment analysis. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Haiku 4.5 ClimbMix analysis (67/100 exp) — decision-making comparison across 3 models Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Retract tainted Haiku results, add stock baseline methodology for fair cross-model comparison Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Haiku 4.5 early results — ClimbMix 22/101 experiments, new absolute best 1.2924 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Complete cross-LLM comparison: Sonnet 4.6 wins 3-2 across all 5 datasets Final two datasets complete: - SlimPajama: S4.0 wins (1.5259 vs 1.5267), both models find it near-impervious - FineWeb-Edu-High: S4.6 wins (1.3345 vs 1.3463), spectacular beta2 walk (5 consecutive keeps: 0.98->0.975->0.970->0.968->0.966->0.964) Overall: S4.6 crashes 2.1x less, explores 60% more parameters, discovered AR=21 architecture. S4.0 wins where it had pre-optimized baseline advantage. New: SlimPajama + FineWeb-Edu-High Sonnet 4.6 run pages Updated: Cross-LLM-Comparison (complete 5-dataset analysis), Cross-Dataset-Comparison, Home.md, chart (5 datasets) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Sonnet 4.6 SlimPajama results — S4.0 wins, score now 2-1-1 SlimPajama completed: 101 experiments, 2 keeps, best 1.5267 (vs S4.0's 1.5259). First dataset where Sonnet 4.0 clearly wins — flat optimization landscape where both models barely improve from baseline. Sonnet 4.6 struggled with reproducibility (exp36-49 couldn't replicate baseline). Series score: Sonnet 4.6 leads 2-1-1 with FineWeb-Edu-High remaining as the tiebreaker. New: Sonnet-4.6-SlimPajama-Agent-Run-Mar-24-2026.md Updated: Cross-LLM-Comparison.md (4 datasets), Cross-Dataset-Comparison.md, Home.md, cross-llm-comparison.png chart (4 datasets) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Sonnet 4.6 Cosmopedia-v2 results — AR=21 breaks AR=32 consensus Sonnet 4.6 completed 101 experiments on Cosmopedia-v2: 16 keeps (16.0%), 0 crashes, best val_bpb 0.9549 (vs Sonnet 4.0's 0.9606). The key discovery is ASPECT_RATIO=21, breaking the universal AR=32 finding from all five Sonnet 4.0 datasets. Every parameter differs between the two models' best configurations. New: Sonnet-4.6-Cosmopedia-v2-Agent-Run-Mar-24-2026.md Updated: Cross-LLM-Comparison.md (3 datasets), Cross-Dataset-Comparison.md, Home.md, cross-llm-comparison.png chart (3 datasets) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Sonnet 4.6 FineWeb-Edu results and update cross-LLM comparison New: Sonnet 4.6 FineWeb-Edu run page (100 experiments, 18 keeps, best 1.3416) Key finding: Both models converge to nearly identical val_bpb (1.3416 vs 1.3424) via completely different configurations — proving multiple near-equivalent optima exist in FineWeb-Edu's optimization landscape. Updated: Cross-LLM Comparison with full FineWeb-Edu analysis Updated: Cross-Dataset Comparison with Sonnet 4.6 FineWeb-Edu reference Updated: Home page with new run entry New: Cross-LLM comparison chart Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Sonnet 4.6 ClimbMix run, Cross-LLM Comparison page New pages: - Sonnet 4.6 ClimbMix Agent Run (119 experiments, 8 keeps, 1.63% improvement) - Cross-LLM Comparison (Sonnet 4.0 vs 4.6 analysis framework) Updated: - Home: new Cross-LLM section, Sonnet 4.6 run table, LLM column - Cross-Dataset Comparison: link to Cross-LLM page Key finding: Sonnet 4.6 demonstrates compositional multi-parameter optimization (3 synergistic HPs) vs Sonnet 4.0's single-param tweaks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add FineWeb-Edu-High run (Mar 21), update cross-dataset comparison to 5 datasets

Make comparison chart clickable to view full-size image

Add SlimPajama run, update cross-dataset comparison to 4 datasets Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Update cross-dataset comparison chart with Cosmopedia-v2 All 6 panels now show 3 datasets (climbmix, FineWeb-Edu, cosmopedia-v2). Architecture panel confirms AR=32 convergence. HP ratios show climbmix/cosmopedia similarity vs FineWeb-Edu divergence. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add Cosmopedia-v2 run results, update cross-dataset comparison to 3 datasets New page: Cosmopedia-v2-Agent-Run-Mar-20-2026 (103 experiments, 4 keeps, best val_bpb 0.9606). Updated cross-dataset comparison with architecture convergence finding (all 3 datasets → AR=32) and climbmix/cosmopedia config similarity. Updated Home.md with new run row. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update cross-dataset comparison with Mar 19 climbmix results Major revision: both datasets converge to AR=32 (architecture convergence, not divergence). Updated hyperparameter tables with accurate climbmix config (EMBEDDING_LR=0.4, SCALAR_LR=0.4). Added near-optimal plateau analysis and crash rate correlation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add Climbmix Mar 19 agent run results (101 experiments) Near-optimal baseline: only 0.08% improvement found across 101 experiments. 1 keep (UNEMBEDDING_LR), 89 discards, 11 crashes (all batch size). Best val_bpb: 1.2959. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add banner graphic to wiki Home page Dark-themed banner with convergence sparkline, chip badges showing best val_bpb per Apple Silicon chip, and project title. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update wiki for merged branches — remove stale branch references - TUI-Dashboard: branch → master, remove checkout instructions - Multi-Dataset-Experiments: branch → master, fix doc URLs - Home: FineWeb-Edu branch → master, fix evaluating-results URL Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move cross-dataset detail to dedicated wiki page Keep graphic and lead-in on Home page with link to full analysis. New Cross-Dataset-Comparison page has architecture divergence, hyperparameter divergence, improvement dynamics, best configs side-by-side, and implications. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add cross-dataset comparison section to wiki Home page New comparison section with 6-panel chart showing architecture divergence, hyperparameter divergence, improvement trajectories, and resource usage between climbmix and FineWeb-Edu. Renamed header to "Autoresearch — Characterization & Experimentation". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add FineWeb-Edu agent run results (101 experiments, Mar 17 2026) First multi-dataset experiment. 101 autonomous experiments on FineWeb-Edu achieving val_bpb=1.2951 (8.1% improvement). Key finding: optimal config is dataset-dependent — FineWeb-Edu prefers AR=32 (half width) with 2x more gradient steps vs climbmix's AR=64. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move References section to bottom of Home page Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add multi-dataset experiment framework wiki page Reference karpathy/autoresearch PR #303 (evaluating results at scale). Add Multi-Dataset-Experiments page with architecture, usage, and early climbmix vs FineWeb-Edu comparison results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add autonomous agent run results for Mar 16, 2026 81-experiment autonomous run on M5 Max using TUI dashboard agent mode. Best val_bpb: 1.3353 (exp72). 9 kept, 69 discarded, 3 crashes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Update TUI wiki: live screenshot, add troubleshooting section Replace simulated screenshot with one captured from actual training on M5 Max. Add troubleshooting section documenting the memory-pressure hang and one-step display delay. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>