# Architecture
ASPECT_RATIO = 64
HEAD_DIM = 128
MLP_RATIO = 1.5 # narrow MLP (from 4x default)
WINDOW_PATTERN = "SSSL"
DEPTH = 6 # Pro tier default
# Optimization
TOTAL_BATCH_SIZE = 8192 # 8x smaller than Pro default (64K)
DEVICE_BATCH_SIZE = 4 # 4x smaller than Pro default (16)
EMBEDDING_LR = 0.6
UNEMBEDDING_LR = 0.004
MATRIX_LR = 0.04
SCALAR_LR = 0.5
WEIGHT_DECAY = 0.2
ADAM_BETAS = (0.8, 0.95)
WARMUP_RATIO = 0.0
WARMDOWN_RATIO = 0.7 # longer cooldown (from 0.5)
FINAL_LR_FRAC = 0.0

*M1 Max mar11 step count is estimated from their total_tokens/batch_size.

1. No memory swapping: The M1 Max baseline used 94 GB on a 64 GB machine (heavy swap). The M4 Pro baseline used 32 GB on 24 GB (also swapping). But the optimized M4 Pro config uses only 4.5 GB — fully in-memory with zero swap overhead.

2. Smaller batch = more steps: M4 Pro went further with batch 8K (vs M1 Max at 16K). The M4 Pro's faster per-core performance (~40% faster than M1) made each step faster, so 8K batch worked where M1 Max found it "too noisy" — the extra steps compensated for gradient noise.

3. Step speed dominates: On M4 Pro, the critical factor is maximizing training steps within the 5-minute window. Every experiment that increased model size (depth, width, MLP ratio) failed because fewer steps outweighed the per-step quality gains.

• Branch: autoresearch/mar14
• Agent: Claude Opus 4.6 (autonomous experiment runner)
• Date: March 14, 2026