-
Couldn't load subscription status.
- Fork 6.5k
[ci] don't run sana layerwise casting tests in CI. #12551
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM! I've observed that some other test_layerwise_casting_inference tests seem to fail in the CI as well. For example, on the CI job when PRX was merged, both Qwen Image (https://github.com/huggingface/diffusers/actions/runs/18701189489/job/53330184168):
FAILED tests/pipelines/qwenimage/test_qwenimage_edit.py::QwenImageEditPipelineFastTests::test_layerwise_casting_inference - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 14.74 GiB of which 14.03 GiB is free. Process 19357 has 724.00 MiB memory in use. Of the allocated memory 465.60 MiB is allocated by PyTorch, and 132.40 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED tests/pipelines/qwenimage/test_qwenimage_edit_plus.py::QwenImageEditPlusPipelineFastTests::test_layerwise_casting_inference - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 14.74 GiB of which 14.05 GiB is free. Process 19357 has 708.00 MiB memory in use. Of the allocated memory 484.53 MiB is allocated by PyTorch, and 97.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
and Sana (https://github.com/huggingface/diffusers/actions/runs/18701189489/job/53330184161):
FAILED tests/pipelines/sana/test_sana.py::SanaPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
FAILED tests/pipelines/sana/test_sana_controlnet.py::SanaControlNetPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
FAILED tests/pipelines/sana/test_sana_sprint.py::SanaSprintPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
FAILED tests/pipelines/sana/test_sana_sprint_img2img.py::SanaSprintImg2ImgPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
test_layerwise_casting_inference tests failed.
@dg845 please check now.
Regarding the Qwen failures, I am not too sure but OOMs in our CI can be also triggered by a prior test failure in the mix. I printed the torch.cuda.max_memory_allocated() for those concerned QwenImage tests in GB and it came out to be 0.4499330520629883 GB.
So, I suspect these OOMs are caused by previous test failures. But LMK if you have an another perspective.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM :)
What does this PR do?
Tests pass locally perfectly fine on different GPUs (RT 4090, A100, H100).
Example failures: