-
Notifications
You must be signed in to change notification settings - Fork 422
Vulkan: Select device with env variable, and skip initialize for unused devices. #629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Works for me... kind of 🙂 I don't think SD_VK_DEVICE can be a replacement for GGML_VK_VISIBLE_DEVICES:
SD_VK_DEVICE=0 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195 - Vulkan: Using device 0
SD_VK_DEVICE=1 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[WARN ] stable-diffusion.cpp:191 - Cannot find targeted vulkan device (1). Falling back to device 0.
[INFO ] stable-diffusion.cpp:195 - Vulkan: Using device 0
GGML_VK_VISIBLE_DEVICES=0 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none
[INFO ] stable-diffusion.cpp:195 - Vulkan: Using device 0
GGML_VK_VISIBLE_DEVICES=1 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195 - Vulkan: Using device 0
But it's still useful when used together:
GGML_VK_VISIBLE_DEVICES=0,1 SD_VK_DEVICE=1 ./sd (...)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195 - Vulkan: Using device 1
As I understand it, ggml excludes the iGPU by default, so it doesn't even show up on the device list unless it's explicitly enabled.
(aside: I really wish there was a way to specify the devices by a stable ID. Notice how the '0' device points to different GPUs depending on the context...)
Ok I finally compiled sdcpp with Rocm, and at least it's selecting device 0 by default (and it's running faster than I thought compared to Vulkan), but It makes me think having an option to select backend device when initializing context could be a nice idea for all backends.
I'm thining maybe passing --device ID via cli, and also a --list-devices argument to know which id corresponds to what device?
It could also be worth it to find a way to select device for model parts, like end text encoders and VAE to device 1 and diffusion model to device 0 for example?
Uh oh!
There was an error while loading. Please reload this page.
Introduces the
SD_VK_DEVICEenv variable that can be used to chose the device to run inference on.Also backend was initialized on every device for no apparent reason.
I know it was already possible to hide some devices from the Vulkan backend by setting the
GGML_VK_VISIBLE_DEVICESenv variable, but this isn't ideal, as it affects every program that uses Vulkan GGML.(Maybe the same should be done for CUDA and SYCL backends)