glm53-flash-mlx

MLX (Apple Silicon) runtime and quantization tooling for zai-org/GLM-5.3-Flash — 320B-A18B: 34 Kimi-Delta linear-attention layers interleaved with 11 DeepSeek-sparse-attention layers (NoPE MLA behind a lightning indexer), 288-expert MoE, manifold-constrained hyper-connections (model_type: glm5_next).

Published builds: pipenetwork/GLM-5.3-Flash MLX (8-bit, 6-bit, mixed 4/8-bit, 4-bit; see Measurements).

mlx-vlm merged glm5_next on 2026年08月26日 (PR #2030); no release carries it. Reviewing that port line by line against transformers 5.16, and then checking it numerically at tiny scale, found the hyper-connection, KDA, MLA and indexer math all correct — and two things wrong that fluent output would never reveal:

Plus two epsilon mismatches (MLA low-rank norms use rms_norm_eps = 1e-5, not 1e-6; the indexer LayerNorm uses 1e-6, not 1e-5) and bf16 router logits where the reference uses float32.

glm53_flash_mlx/glm5_next/ is the mlx-vlm package with those fixed: a ClampedSwiGLU for the routed experts, shared expert and dense layers; a float32 router; the epsilons; a cast_predicate that protects the mHC arrays and KDA A_log/dt_bias, and a sanitize that heals checkpoints where they were already cast.

./scripts/run_tests.sh

A random tiny model with swiglu_limit low enough to actually clip, randomised mHC arrays and KDA decay parameters, a sequence longer than index_topk so the indexer selects, and more experts than top-k, checked against transformers:

[1] full forward (KDA x3 + DSA with live indexer, mHC, clamped MoE)
 logits max|delta| 9.239e-07 (scale 5.863e-01) OK
[2] short sequence (T <= index_topk: dense-MLA bypass)
 logits max|delta| 8.643e-07 OK
[3] sanitize is idempotent; bf16-cast hc/KDA scalars come back as float32 OK
[4] control: unclamped SwiGLU -> logits move by 5.865e-01 OK
[5] token-by-token decode == single forward
 incremental vs single-shot max|delta| 2.086e-07 OK
 chunked prefill 5+7 vs single-shot max|delta| 0.000e+00 OK

tests/debug_modules.py is the teacher-forced, module-by-module comparison that located the clamp: every other block matched to ~1e-7 before the fix.

python scripts/quantize_stream.py --src GLM-5.3-Flash-BF16-src --dst out/GLM-5.3-Flash-MLX-4bit --bits 4
python scripts/quantize_stream.py --src ... --dst out/...-mixed-4_8bit --bits 4 --other-bits 8

One decoder layer at a time: the release stores each layer's 288 experts as separate tensors and the runtime stacks them, so the natural streaming unit is the layer (14.5 GB of experts in bf16), gathered lazily from whichever shards hold it. The 643 GB source never needs to fit. The FP8 release would also work (the port dequantizes weight_scale_inv blocks); the bf16 release was used so nothing is lost before quantization.

Recipe: routed experts (304B, 97%) at --bits; everything else quantizable (~9B) at --other-bits — uniformly, because the runtime fuses the six KDA input projections into one matmul at load and they must share a width; the lightning-indexer projections always at 8-bit; the router, mHC arrays (fp32), KDA A_log/dt_bias (fp32), convolutions and norms as stored; the vision tower in bf16; the MTP layer dropped.

scripts/ppl_corpus.py tokenizes wikitext-2 (test) once; scripts/ppl_large.py teacher-forces every build on the same 141 windows of 2048 tokens; scripts/ppl_compare.py does the paired bootstrap. The bf16 model does not fit a 512 GB machine, so 8-bit is the anchor.

REAP-pruned builds (scripts/reap_calibrate_full.py, scripts/prune_build.py): experts ranked by mean router_weight ×ばつ ‖expert_output‖ over 65,536 calibration tokens (wikitext-2 train, ten languages of Wikipedia, code; zero 32-gram overlap with the eval set), collected by running the full 8-bit build; two disjoint halves of the calibration set pick the same kept set 86.6% of the time at keep-50%. Pruning is applied to the already-quantized mixed 4/8-bit build. Cost on this model is steep and roughly linear in experts removed: REAP25 +18%, REAP37 +37%, REAP50 +70% perplexity over the unpruned mixed build — published because 96–139 GB is what fits a 128 GB Mac, with the numbers on the card. The uniform-4-bit prunes were built and measured too (4.4361 / 5.1057 / 6.3840) and are not published: the mixed prunes beat them at every ratio for ~4 GB more.