-
Notifications
You must be signed in to change notification settings - Fork 205
[Klaud Cold] MI325X MiniMax-M3 EAGLE3 nightly image and FP8 KV cache#1838
[Klaud Cold] MI325X MiniMax-M3 EAGLE3 nightly image and FP8 KV cache #1838cquil11 wants to merge 5 commits into
Conversation
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook
If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you
PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow
As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.
If additional help is needed, PR authors can reach out to core maintainers over Slack.
2 similar comments
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook
If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you
PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow
As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.
If additional help is needed, PR authors can reach out to core maintainers over Slack.
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook
If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you
PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow
As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.
If additional help is needed, PR authors can reach out to core maintainers over Slack.
3b2ffd4 to
ea746c7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ea746c7. Configure here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FP8 KV on gfx942
Medium Severity
This change adds --kv-cache-dtype fp8 on MI325X (gfx942), but the repo still treats that stack as lacking calibrated ROCm FP8 attention scales (fallback scale 1.0) on the same architecture in minimaxm3_fp8_mi300x_mtp.sh and minimaxm3_fp8_mi300x.sh, while the non-MTP MI325X script still omits FP8 KV. The removed comment documented that BF16 was kept deliberately to avoid corrupted accuracy.
Reviewed by Cursor Bugbot for commit ea746c7. Configure here.
ea746c7 to
12dbb7a
Compare
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27793866629
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27793866629
@functionstackx
functionstackx
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cquil11 codex agent, plz remove the EAGLE patch now that u upgraded to nightly image, it is now included vllm-project/vllm#45546
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27794191643
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27794191643
1 similar comment
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27794191643
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27794191643
Addressed in 1770e9c: removed the legacy in-place EAGLE3 patch and now rely on upstream vLLM #45546.
cquil11
commented
Jun 19, 2026
@functionstackx Removed the legacy in-place EAGLE3 patch in 1770e9c and now rely on upstream vLLM #45546. The rerun is green and MI325X MTP averages +3.1% versus the latest official baseline. Could you re-review/approve?
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27836183445
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27836183445
1 similar comment
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27836183445
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27836183445
Uh oh!
There was an error while loading. Please reload this page.
What changed
minimaxm3-fp8-mi325x-vllm-mtptovllm/vllm-openai-rocm:nightly-b53b1c7ffe7aebdafd0876350f30e51d1226c92a--kv-cache-dtype fp8chi-mi325x-pod2-120, which lacks the populated model cache required by the launcherWhy
Use the updated ROCm vLLM build and reduce KV-cache memory use for the MI325X MiniMax-M3 EAGLE3 sweep. The node exclusion prevents Pyxis mount failures on the incomplete node.
Validation
Note
Medium Risk
Benchmark numbers will shift with the new image and FP8 KV; correctness depends on the nightly shipping upstream EAGLE3 support that the removed patch provided.
Overview
Updates the MI325X MiniMax-M3 EAGLE3 (MTP) benchmark path to a pinned ROCm vLLM nightly and simplifies how the server is started.
minimaxm3-fp8-mi325x-vllm-mtpnow usesvllm/vllm-openai-rocm:nightly-b53b1c7ffe7aebdafd0876350f30e51d1226c92ainstead ofvllm/vllm-openai-rocm:minimax-m3. Config comments no longer describe a runtime SupportsEagle3 patch or BF16-only KV on gfx942.In
minimaxm3_fp8_mi325x_mtp.sh, the large idempotent Python in-place patch ofvllm/models/minimax_m3/amd/model.pyis removed (EAGLE3 support is expected in the new image).vllm servegains--kv-cache-dtype fp8, and header comments are updated to match (FP8 KV for headroom vs the old BF16-on-gfx942 accuracy caveat).launch_mi325x-amds.shadds--exclude=chi-mi325x-pod2-120...onsallocso jobs skip a node without a populated/raid/hf-hub-cache.perf-changelog.yamldocuments the image bump, FP8 KV, patch removal, and node exclusion for this config key.Reviewed by Cursor Bugbot for commit 1770e9c. Bugbot is set up for automated code reviews on this repo. Configure here.