forked from karpathy/autoresearch
-
Notifications
You must be signed in to change notification settings - Fork 0
Experiment Results Mar 15 2026 M5 Max
Dave Graham edited this page Mar 15, 2026
·
1 revision
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Est. bf16 TFLOPS | ~34.0 (40 cores x 0.85 TFLOPS/core) |
| Auto-detected Tier | Max (depth=8, device_batch=32, total_batch=128K) |
| Metric | Value |
|---|---|
| val_bpb | 1.319639 |
| Improvement over baseline | 36.4% (2.075 -> 1.320) |
| Peak Memory | 26,742 MB (40.8% of 64 GB) |
| MFU | 23.35% |
| Training Steps | 312 |
| Total Tokens | 10.2M |
# Architecture
DEPTH = 8 # Max tier default (unchanged)
MLP_RATIO = 4.0 # Full width (unchanged from default)
ASPECT_RATIO = 64 # → model_dim=512, 4 heads
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
# Optimization
TOTAL_BATCH_SIZE = 32768 # 4x smaller than Max default (128K)
DEVICE_BATCH_SIZE = 16 # 2x smaller than Max default (32)
EMBEDDING_LR = 0.6
UNEMBEDDING_LR = 0.006 # 1.5x default (from 0.004)
MATRIX_LR = 0.06 # 1.5x default (from 0.04)
SCALAR_LR = 0.5
WEIGHT_DECAY = 0.1 # halved (from 0.2)
ADAM_BETAS = (0.8, 0.95)
WARMUP_RATIO = 0.0
WARMDOWN_RATIO = 0.6 # slightly longer cooldown (from 0.5)
FINAL_LR_FRAC = 0.0
| Exp | val_bpb | Delta | Steps | Description |
|---|---|---|---|---|
| exp0 | 2.075 | — | 15 | Baseline (batch 131K, depth 8, MLP 4x) |
| exp1 | 1.541 | -0.534 | 152 | Batch 131K->65K |
| exp2 | 1.335 | -0.206 | 299 | Batch 65K->32K, dev_batch 32->16 |
| exp7 | 1.327 | -0.008 | 301 | Matrix LR 0.04->0.06 |
| exp9 | 1.322 | -0.005 | 308 | Weight decay 0.2->0.1 |
| exp12 | 1.320 | -0.002 | 310 | Warmdown ratio 0.5->0.6 |
| exp14 | 1.320 | -0.000 | 312 | Unembedding LR 0.004->0.006 |
| # | val_bpb | mem_gb | tok/sec | mfu | steps | status | description |
|---|---|---|---|---|---|---|---|
| exp0 | 2.075 | 94.2 | 1,400 | 1.2% | 15 | baseline | Default hyperparams (depth=8, batch=131K, dev=32, MLP=4x) |
| exp1 | 1.541 | 51.7 | 30,800 | 21.7% | 152 | keep | TOTAL_BATCH=65536 (2^16) |
| exp2 | 1.335 | 26.1 | 31,600 | 22.2% | 299 | keep | TOTAL_BATCH=32768, DEV_BATCH=16 |
| exp3 | 1.379 | 26.1 | 25,400 | 18.4% | 250 | discard | WARMDOWN_RATIO=0.7 — fewer steps, lower MFU |
| exp4 | 1.599 | 38.1 | 15,500 | 19.6% | 153 | discard | DEPTH=10 — too slow (2.1s/step), fewer steps |
| exp5 | 1.363 | 23.8 | 28,100 | 16.7% | 269 | discard | MLP_RATIO=2.5 — less capacity (44M vs 50M) |
| exp6 | 1.356 | 13.2 | 26,500 | 18.3% | 488 | discard | TOTAL_BATCH=16384, DEV_BATCH=8 — batch too small |
| exp7 | 1.327 | 26.1 | 32,200 | 22.3% | 301 | keep | MATRIX_LR=0.06 (from 0.04) |
| exp8 | 1.328 | 26.1 | 31,900 | 23.0% | 310 | discard | MATRIX_LR=0.08 — plateau, no improvement |
| exp9 | 1.322 | 26.1 | 32,300 | 22.9% | 308 | keep | WEIGHT_DECAY=0.1 (from 0.2) |
| exp10 | 1.322 | 26.1 | 32,000 | 23.1% | 308 | discard | EMBEDDING_LR=0.8 — no improvement over 0.6 |
| exp11 | 1.344 | 26.1 | 33,000 | 23.1% | 310 | discard | WARMUP_RATIO=0.05 — wastes steps at lrm=0.00 |
| exp12 | 1.320 | 26.1 | 33,400 | 23.2% | 310 | keep | WARMDOWN_RATIO=0.6 (from 0.5) |
| exp13 | 1.323 | 26.1 | 33,400 | 23.2% | 310 | discard | WARMDOWN_RATIO=0.7 — too much cooldown |
| exp14 | 1.320 | 26.1 | 33,800 | 23.4% | 312 | keep | UNEMBEDDING_LR=0.006 (from 0.004) |
| exp15 | 1.434 | 35.1 | 20,500 | 23.8% | 200 | discard | DEPTH=9 — 81M params too slow (1.6s/step) |
| exp16 | 1.329 | 26.1 | 34,500 | 23.1% | 312 | discard | SCALAR_LR=0.7 — worse than 0.5 |
| Metric | M1 Max (mar11) | M4 Pro (mar14) | M5 Max (mar14-m5max) |
|---|---|---|---|
| Best val_bpb | 1.621 | 1.429 | 1.320 |
| Memory | 64 GB | 24 GB | 64 GB |
| GPU Cores | 32 | 16 | 40 |
| Est. bf16 TFLOPS | ~16 | ~11 | ~34 |
| Peak Memory Used | 11.3 GB | 4.5 GB | 26.1 GB |
| Training Steps | ~210 | 751 | 312 |
| Params | 39.8M | 21.9M | 50.3M |
| TOTAL_BATCH_SIZE | 16,384 | 8,192 | 32,768 |
| DEVICE_BATCH_SIZE | 8 | 4 | 16 |
| DEPTH | 8 | 6 | 8 |
| MLP_RATIO | 1.5 | 1.5 | 4.0 |
| MATRIX_LR | 0.04 | 0.04 | 0.06 |
| WEIGHT_DECAY | 0.2 | 0.2 | 0.1 |
| WARMDOWN_RATIO | 0.7 | 0.7 | 0.6 |
| MFU | ~18% | ~13.5% | 23.4% |
| Total Experiments | 25 | 15 | 17 |
-
2x compute throughput: The M5 Max has ~34 TFLOPS bf16 vs ~16 TFLOPS on M1 Max and ~11 TFLOPS on M4 Pro. This translates directly to faster step times, allowing more gradient steps within the 5-minute budget.
-
Full architecture retained: Unlike M1 Max and M4 Pro which needed to narrow the MLP (4x->1.5x) to gain speed, M5 Max keeps the full 50M-param model at MLP=4x. The extra compute handles the larger model without sacrificing step throughput.
-
Higher LR + lower regularization: M5 Max benefits from a 1.5x higher Muon LR (0.06 vs 0.04) and halved weight decay (0.1 vs 0.2). This is consistent with the model having more capacity and more steps to converge — it can handle more aggressive optimization.
-
Batch 32K sweet spot: M5 Max's optimal batch (32K) is between M1 Max (16K) and the default (128K). The faster GPU means each step is fast enough that 32K provides the right balance between gradient quality and step count (~310 steps vs ~300 at 32K, ~488 at 16K but with much lower MFU).
| Dimension | M1 Max Finding | M4 Pro Finding | M5 Max Finding |
|---|---|---|---|
| MLP ratio | Narrow (1.5x) | Narrow (1.5x) | Full width (4x) |
| Batch strategy | Aggressive reduction | Extreme reduction | Moderate reduction |
| LR tuning | Default optimal | Default optimal | Higher LR helps |
| Weight decay | Default optimal | Default optimal | Lower WD helps |
| Warmdown | 0.7 | 0.7 | 0.6 |
| Key lever | Throughput | Step count | Optimization tuning |
| Parameter | Values Tested | Optimal |
|---|---|---|
| TOTAL_BATCH | 16384, 32768, 65536, 131072 | 32768 |
| MATRIX_LR | 0.04, 0.06, 0.08 | 0.06 |
| WEIGHT_DECAY | 0.1, 0.2 | 0.1 |
| WARMDOWN_RATIO | 0.5, 0.6, 0.7 | 0.6 |
| UNEMBEDDING_LR | 0.004, 0.006 | 0.006 |
| EMBEDDING_LR | 0.6, 0.8 | 0.6 |
| SCALAR_LR | 0.5, 0.7 | 0.5 |
| WARMUP_RATIO | 0.0, 0.05 | 0.0 |
| DEPTH | 8, 9, 10 | 8 |
| MLP_RATIO | 2.5, 4.0 | 4.0 |
| DEVICE_BATCH_SIZE | 8, 16, 32 | 16 |
-
Branch:
autoresearch/mar14-m5max - Agent: Claude Opus 4.6 (autonomous experiment runner)
- Date: March 15, 2026