Agent Run Mar 16 2026

This was the first fully autonomous experiment run using the TUI Dashboard agent mode. Claude Sonnet designed and executed 81 experiments over ~7 hours, iterating on hyperparameters to minimize val_bpb on the Apple M5 Max (64 GB).

The agent achieved a 1.3% improvement over the starting baseline, reducing val_bpb from 1.3529 → 1.3353 through a series of incremental optimizations to learning rates, weight decay, and warmdown scheduling.

Agent Run Results — Mar 16, 2026

The agent found 8 improvements beyond the baseline, each building on the previous best:

# Architecture (unchanged from defaults)
DEPTH = 8
ASPECT_RATIO = 64 # → model_dim=512, 4 heads
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
MLP_RATIO = 4.0
# Optimization (agent-tuned)
TOTAL_BATCH_SIZE = 32768
DEVICE_BATCH_SIZE = 16
MATRIX_LR = 0.06 # ↑ from 0.04 (exp2)
EMBEDDING_LR = 0.585 # ↓ from 0.6 (exp60, exp72)
UNEMBEDDING_LR = 0.0041 # ↑ from 0.004 (exp69)
SCALAR_LR = 0.5 # unchanged
WEIGHT_DECAY = 0.02 # ↓ from 0.2 (exp4→exp24→exp25)
ADAM_BETAS = (0.8, 0.95) # unchanged
WARMUP_RATIO = 0.0 # unchanged
WARMDOWN_RATIO = 0.4 # ↓ from 0.5 (exp48)
FINAL_LR_FRAC = 0.0 # unchanged

The agent's behavior shows three distinct phases:

Phase 1 — Broad exploration (exp0–exp25): The agent tested a wide range of parameters including architecture changes (HEAD_DIM, ASPECT_RATIO, DEPTH, WINDOW_PATTERN), batch sizes, learning rates, and regularization. Key discoveries: MATRIX_LR=0.06 and the weight decay reduction chain (0.2 → 0.1 → 0.05 → 0.02). Three crashes occurred from overly aggressive batch size reductions (exp5, exp6, exp15).

Phase 2 — Fine-tuning (exp26–exp55): With WEIGHT_DECAY=0.02 established, the agent explored finer-grained adjustments to Adam betas, scalar LR, matrix LR, and warmdown ratio. Most experiments were very close to the best (within 0.005 val_bpb). The key find was WARMDOWN_RATIO=0.4 (exp48).

Phase 3 — Precision tuning (exp56–exp80): The agent shifted to very small incremental changes (e.g., EMBEDDING_LR from 0.6 to 0.59, then 0.585; UNEMBEDDING_LR from 0.004 to 0.0041). Improvements became vanishingly small (~0.0002 per keep), suggesting convergence to a local optimum.

1. Weight decay was the biggest lever: Reducing from the default 0.2 down to 0.02 across three experiments (exp4→exp24→exp25) contributed the most cumulative improvement. However, going further to 0.01 hurt (exp26), establishing 0.02 as optimal.

2. Learning rate tuning at the margins: MATRIX_LR=0.06 (1.5x default) was the single biggest single-experiment improvement. Embedding and unembedding LR changes in the ±0.01 range produced measurable but tiny gains.

3. Warmdown ratio matters: Reducing from 0.5 to 0.4 (exp48) was a meaningful improvement, but going lower to 0.3 (exp49) or higher to 0.6 (exp44) both hurt. The model benefits from a slightly shorter cooldown phase.

4. Architecture changes didn't help: HEAD_DIM=160, ASPECT_RATIO=80, reduced DEPTH, and alternative WINDOW_PATTERN="SSL"/"SL" were all worse. The default architecture is well-suited for this chip.

5. Adam betas are robust: Testing β1 in {0.7, 0.75, 0.85, 0.9} and β2 in {0.92, 0.96} all performed worse than the default (0.8, 0.95).

6. Diminishing returns are real: After ~50 experiments, the agent was making sub-0.001 improvements. The last 30 experiments collectively improved val_bpb by only ~0.002.

Three experiments crashed (3.7% crash rate):

• exp5 (DEVICE_BATCH_SIZE=6) — OOM or incompatible batch size
• exp6 (DEVICE_BATCH_SIZE=7) — Same failure mode
• exp15 (TOTAL_BATCH_SIZE reduction) — Invalid configuration

All crashes were from aggressive batch size changes. The agent learned from these and avoided similar changes afterward.

Note: The Mar 15 run started from the default configuration with batch=131K, where the initial batch size reduction alone gave a 0.534 improvement. The Mar 16 run started from an already-optimized baseline (batch=32K), leaving much less room for improvement. The Mar 16 run demonstrates the agent's ability to find small, incremental gains even on a well-tuned configuration.

This run used the TUI Dashboard in agent mode:

uv run dashboard.py --agent --tag mar16-agent --max 100

The dashboard provided real-time monitoring with:

• Live training metrics (loss, tok/sec, MFU, progress bar)
• Hardware memory polling (RSS every 2s)
• Experiment status panel (keep/discard/crash counts)
• Activity log with full experiment history
• Automatic git branching and commit/revert per experiment

See TUI Dashboard for more details.

• Branch: autoresearch/mar16-agent
• Agent: Claude Sonnet (autonomous via TUI dashboard)
• Date: March 16, 2026