--offload-to-cpu

Nice. Unloading from VRAM is currently only controlled by the free_params_immediately context flag, right? So setting that flag to false (to avoid paying for the RAM->VRAM cost on each inference), together with a new unload_weights function, would pretty much work as the 'low-hanging fruit' I suggested above. The process could keep an sd_ctx_t for each "pre-loaded" model, and one (or a few) active; and call unload_weights when switching that active context.

The --offload-to-cpu option will automatically offload the weights from the GPU once the computation is completed.

I managed to accelerate model loading by using multithreading, achieving up to ×ばつ faster loading compared to master

https://github.com/rmatif/stable-diffusion.cpp/blob/ref-tensor-loading/model.cpp

@leejet I can open a PR if you think this can be merged

Cool. Ping me when you do, I can help reviewing it.

Regarding the SDXL use case: what the civitai folks are doing is keeping the model in RAM and loading it directly from RAM → VRAM. What I noticed is that we are severely CPU-bound in this process.

Note the bottleneck will likely depend on the CPU, backend and weight types. If the model's weights already have their final types, most of that CPU cost can vanish (even a straight conversion to gguf already speeds things up).

I tried using mmap from ggml here: https://github.com/rmatif/stable-diffusion.cpp/tree/add-mmap

Storing the model in a ramdisk and loading it into RAM takes ~4.3s while using a warm load with mmap takes ~1.1s.

The warm load is essentially memory-bound, which means the cpu overhead is actually huge. From my measurements with multithreading we can achieve ramdisk → RAM in ~1.3s, meaning the cpu overhead will be around ~200ms. With further optimizations such as avoiding regex I think we could potentially reduce this even further

Interesting. I saw you're using memcpy between the mapped file and the local buffers. It'd likely be a bigger, more invasive change, but for weights that don't need conversion, you could instead use the mapped area directly as the tensor buffer. This would help reducing memory pressure in cases where the same file may be loaded more than once - like Civitai's :-) And together with @leejet 's new offload-to-cpu code path, some models could then run directly from the page cache.

CUDA does indeed support asynchronous loading and it's actually doing great, the RAM -> VRAM (once the cpu overhead reduced) is not a bottleneck imo

Maybe not for plain CUDA :-) The RAM -> VRAM path on my end can take around 60% of the loading time.

Note the bottleneck will likely depend on the CPU, backend and weight types. If the model's weights already have their final types, most of that CPU cost can vanish (even a straight conversion to gguf already speeds things up).

Even with a good CPU and no conversion, it still takes around 4s to load from ramdisk → RAM

Interesting. I saw you're using memcpy between the mapped file and the local buffers. It'd likely be a bigger, more invasive change, but for weights that don't need conversion, you could instead use the mapped area directly as the tensor buffer. This would help reducing memory pressure in cases where the same file may be loaded more than once - like Civitai's :-)

Dropping memcpy would indeed be more effective. I also thought about adding a "pinged" option, where we explicitly request the model to be pinged and avoid being paged or moved. I didn’t pursue it further, since they mainly rely on safetensors and prefer having more explicit control over memory management using ramdisk directly rather than relying on mmap

And together with @leejet 's new offload-to-cpu code path, some models could then run directly from the page cache.

I had already considered this, but I’m skeptical in the case of SDXL. Its model arch isn’t linear, so I doubt it can be done easily, it would probably require a significant amount of work imo

Maybe not for plain CUDA :-) The RAM -> VRAM path on my end can take around 60% of the loading time.

On my testing it's about ~20% I think it's reasonable

Great! PR is welcome. Where does this speed improvement come from? Is it due to multithreaded memcpy?

Using more (f)reads in parallel, I presume. The same rules as to why reading mmapped files is faster.

Great! PR is welcome. Where does this speed improvement come from? Is it due to multithreaded memcpy?

It's a combinaison of parallelizing metadata processing, executing concurrent disk reads and hiding the latency by overlapping I/O and cpu work