-
Notifications
You must be signed in to change notification settings - Fork 893
Add local-ai community container with Nvidia GPU support via Cuda 12 #5795
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Fábio C. Barrionuevo da Luz <bnafta@gmail.com>
Signed-off-by: Fábio C. Barrionuevo da Luz <bnafta@gmail.com>
Thanks a lot @luzfcb! I just invited you to the repo to make collaboration easier :)
Hi @luzfcb please let me know if you need any help here :)
@szaimen My problem is just free time. Unfortunately I've been busier than I expected these past few days and haven't had time to continue yet, so it might take a little longer for me to confirm if it's ready or if it needs more improvements.
That said, I noticed that local-ai added support for API keys (comma-separated API Keys), however I couldn't figure out how AIO and AIO Local-AI handle this. https://localai.io/advanced/#api-flags . It probable when set an API Key on AIO admin frontend, than set the LOCALAI_API_KEY environment variable or the .env file on the local-ai container and restart it or restart the local-ai process
Another issue I had was that I couldn't get any response to any request to local-ai . I'm still investigating what might be going wrong.
@szaimen My problem is just free time. Unfortunately I've been busier than I expected these past few days and haven't had time to continue yet, so it might take a little longer for me to confirm if it's ready or if it needs more improvements.
I see. All good! Take the time you need :)
That said, I noticed that local-ai added support for API keys (comma-separated API Keys), however I couldn't figure out how AIO and AIO Local-AI handle this. https://localai.io/advanced/#api-flags . It probable when set an API Key on AIO admin frontend, than set the
LOCALAI_API_KEYenvironment variable or the .env file on the local-ai container and restart it or restart the local-ai process
I see. I think api keys are not enforced, are they? So you should be able to test even without api keys, no?
Another issue I had was that I couldn't get any response to any request to local-ai . I'm still investigating what might be going wrong.
I see... You could try to test it via curl from inside the containers and such...
Hi, any update here? :)
This is a fork of https://github.com/szaimen/aio-local-ai .
It basically uses
quay.io/go-skynet/local-ai:v2.24.2-aio-gpu-nvidia-cuda-12instead ofquay.io/go-skynet/local-ai:v2.24.2-aio-cpuas the base docker image, changes the start script a bit to show metadata about the GPUs/Nvidia GPUs and requires ~48GB to ~96GB of free disk space.