From this, IMO it only misses Linux+CUDA bundle to be useable as download & use.

If we want better packaging on Linux, we can also work on snap/bash installer when trying to use pre-built packages.

It’s high time HuggingFace to copy Ollama’s packaging and GTM strategy, but this time, give credit to llama.cpp. Ideally, we should retain llama.cpp as the core component.

Is the barrier the installation process, or the need to use a complex command line to launch llama.cpp?

For me the biggest thing is llama-server. That tool is fantastic... but it has very low discoverability. Until a few months ago I still thought it was just a demo because it lived in the examples/ folder (I just noticed it moved from there to the tools/ folder in May).

I'd love to see more emphasis placed on llama-server. It's really good! I think it's probably the most end-user appropriate way to interact with this project.

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux, which on Mac and Windows work like a desktop application: you get an icon you can use to launch the tool which start the server running and launch a window that shows that web UI.

Maybe include systray integration and a simple UI for selecting and downloading models too.

At that point llama-server would feel like an alternative to Ollama and LM Studio and Jan. I think it deserves that - it has most of what those tools offer implemented already, what's missing is an installer and a think desktop shell.

It will be cool if 'llama-server' would have auto configuration option to the machine/model like 'ollama' does it.

For windows maybe choco and windows store would be a good idea? 🤔

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - `brew install llama.cpp` works
2. For Linux (CUDA) - they need to clone and install directly from github

I created a rpm spec to manage installation though I think flatpaks might be more user friendly and distribution agnostic.

For Windows - winget (?)

The released Windows builds are available via Scoop.

Updates happen automatically. Old installed versions are kept, and current one symlinked into a folder „current" which provides the executables on the path.

is it feasible to have a single release for OS including all the backend?

For linux I just install the vulkan binaries and run the server from there. Maybe we can have a install script like ollama that detects the system and launches the server which can be controlled from an app as well as cli? The user then gets basic command line utillities like run start stop load list etc?

On Mac, the easiest way (also arguably the safest way) from a user's perspective is to find it in App Store, and install from there. Because of apps from App Store are in a sandbox, so from a user's point of view, installing or uninstalling is simple and clean. Creating a build and passing the App Store review might take some efforts (due to the sandbox constraint), but it should be a one-time thing.

Its my understanding that none of the automated installs support GPU acceleration. I might be wrong but its definitely the case for Windows, which makes it useless to install via winget.

To me the biggest advantage ollama currently has is that the optimal settings for a model are bundled, the gguf spec would allow for this to since its versatile enough to make this a metadata field inside the model. It would allow people to load the settings from a gguf and frontends can extract them and adapt them as they see fit. I think that part is going to be more valuable than obtaining the binary since downloading the binary from github is not that hard.

My personal wishlist

1. Llama-swap integration
2. Catchall compiled binaries for linux, win, osx
3. GUI improvements

I just went through this, so it's fresh in my mind. I had not used Linux since about 2012, when I changed our in-house servers to Windows server. Running local llm's I have used Text-Generation-WebUI, Ollama, and LM Studio. With the release of GPT-OSS-120b, most of the wrappers did not keep up with the new changes as quickly as I wanted. I run a 5090, and I also wanted to move to the newer torch release. I was doing some small training stuff that needed Triton anyway, so, Linux.

OP lists using winget, in Windows, but the recompiled llama.cpp is Vulcan. The binary distributions stop at 12.4, so if you need the newer copies of torch >12.4, it's source code. Having been away from Linux a long time, setting up WSL and creating everything for a build was a bit of a struggle. I actually had GPT-OSS write me a step by step walk through with copy and paste commands so I wouldn't screw it up.

Once everything was running, it's great, and a significant speedup. I moved my dev work to Linux now. Then I wrote a script (really me and GPT-OSS wrote a script) to fire up llama.cpp, so I could just pick models from an old fashioned script menu. It's crude, but it's never out of date. I posted the script creation script that GPT wrote below because I thought the idea of a self creating script was nice.

make_l-server.md

IMHO the UI and the packaging can be a dedicated separate repo under the ggml umbrella. Llama.cpp should be remain true to its manifesto of "inference at the edge" and continue to be a fast and feature-full backend for edge devices. The impedance mismatch of being both a developer and a consumer product is probably not easy to manage.

For Linux (CUDA) - they need to clone and install directly from github

llama.cpp is packaged well in Gentoo, no need to do anything fancy.

When will llama.cpp have the ability to search on the internet? Without the ability to search for information online and relying solely on existing training data, answering questions is always outdated or biased, and can only be considered a toy with no practical value

I keep my llama.cpp in a podman container. I've setup scripts to build it in a container as well. And I'm using cuda.

Before I picked up llama.cpp I used ollama, found it bit too excessive and non-suited for me. Also saw that most libraries just package up llama.cpp and run my load there. I ended up just using llama.cpp directly through C API with python bindings.

I figure it's nice to build llama.cpp but a bit difficult to pick up because there are so many options and things to look at. Non technical people might have difficulty picking it up just for that. But I personally am here for it.

I find llama.cpp could use improvement on variety of places. One major place is the chat format interfaces. I think the current way chat templates work is too generic because the way this works is that you have an append-only log and you're paying high cost if you splice it. Chat templates on other hand pretend you can add stuff in the middle and they're essentially for-loops over messages list. I'd suggest you would completely change this design.

• have a toplevel prompt.
• Then have message templates for system, user, assistant prompts.
• Likewise have messages for text in middle completion.
• Finally have an assistant-starts-talking-predule...
• ...and a way to parse the assistant output back to corresponding messages.
• and assistant-stops-talking -token sequence.
• Just have a simple templating format that includes&leaves stuff from message if it is present or not.

In practice that covers all message formats because they're forced to be constrained in same way as long as they generate tokens in sequence. Even if the constraint was lifted some day, I think this makes a good format.

I think we should focus on what it is now, non-tech users will use Ollama or LMStudio anyway.

Hey everyone!

I’ve been working on some scripts and tools to make it super easy to download and run llama-server on any Linux machine (not just Ubuntu) and for both aarch64 and x86_64.

Feature detection is done before downloading to reduce download time, and CUDA is supported as well, though not fully optimized yet. This is also the only way to provide optimized builds for aarch64.

Right now, models are downloaded with hf download, to make things as easy, like Ollama.
That means the hf CLI is required for now.

On Linux, you can try it with:

curl -s https://angt.github.io/installama.sh | sh -s -- qwen3-4b:Q4_0

This should start a server running the requested model.

I’d love to hear your thoughts on this kind of packaging approach for Linux.
I’m currently exploring macOS with Metal support as the next step.

For Linux (CUDA) - they need to clone and install directly from github

It would be nice to have binaries for Ubuntu with CUDA, even though the one with Vulkan works pretty well

It would be great if GGML_CPU_ALL_VARIANTS=ON binaries with CUDA support were included in this. I have an old Ivy Bridge machine for llama.cpp inference, and it would make my life so much easier if I could just download a ready-to-go binary that will work with Nvidia cards and not require AVX2 or FMA, but take advantage of AVX and F16C.

Hey!!

Apple M1 through M4 are now supported with Metal. It turned out to be much easier than expected.
As before, just run:

curl -s https://angt.github.io/installama.sh | sh -s -- qwen3-4b:Q4_0

and you’re good to go 🚀

Again hf is still required to download the models, i'm going to work on that.

in my nodejs project where have the llama.cpp in textgeneration module, i use like that

bash install.sh
ai | chat | generate | webgpt

working in ubuntu and alpine, maybe adding support to all distros can be a good option, direct and minimalistic

I just setup this build process for precompiled binaries of llama.cpp for Ubuntu 24.04 with NVidia 12.9 drivers.

Just in case it helps anyone else 😸

https://github.com/hybridgroup/llama-cpp-builder

Maybe interesting for someone who wants to build from source under Windows. I made a Powershell Script which installs all the necessary prerequisites for a CUDA build, pulls the repo and builds it.
https://github.com/Danmoreng/llama.cpp-installer

First thanks for this initiative - I think making it easier for more folk to use llama.cpp directly instead of using commercial middleware like LMStudio, ollama is a win-win for everyone.

I think the docs can use a lot of improvement to make them more accessible, organized, more complete, more examples etc. (this will in turn also help AI search) I started this a bit here: #15709

Happy to contribute more if it would be useful. Even asking AI to cleanup, reorganize etc. the current docs would be a net positive

As a downstream consumer, I would love to have a binary release of hip-radeon (ROCm) for ubuntu, as you already do for windows. In the same time, there is still a missing docker image for this backend ( #11913 ). So we are still forced to build manually the source at each release.