Kudos on all the incredible work you and the community did!
It's incredibly important to have an open source self hosted version of some of this progress

Thanks for your amazing work with both whisper.cpp, and llama.cpp. I've been hugely inspired by your contributions (especially with Whisper while working on Buzz), and I share your excitement about all the amazing possibilities of on-device inference.

Personally, I've been thinking a lot about multi-modal on-device inference, like an assistive device that can both capture image data and respond to voice and text commands offline, and I plan to hack something together soon. In any case, thanks again for all your work. I really do hope to continue contributing to your projects (and keep improving my C++ :)) and learning more from you in the future.

— Chidi

"This project will remain open-source" ♥️
You hear that, OpenAI?

As someone involved in a number of "Inference at the edge" projects which are features focused, the feature all of them truly needed was performance! Now with llama.cpp and its forks these projects can reach audiences they never could before.

These are good thoughts.

Thank you Georgi, and thank you everyone else who has contributed and is contributing no matter how big or small.

this core / edge thing appears to have a lot of potential and should at least be made very transparent by all. I.e. what exactly runs at the edge in the current commercial products? so that we know how to plan our resources and not get "frustrated" when, e.g., Microsoft Edge crashes on some of them, or when the claims don't hold because the edge is not at the expected level.

Before more countries embark on building AI capabilities (UK?), maybe one should get that clear. Will there be a national core or a national edge?

This is the true spirit of hacking technologies from the bottom up! 🥷💻

This project will remain open-source

The code has to remain simple and compact in order to allow for quick and easy modifications

Keep doing the good work! 🚀🤘

LLaMA.cpp works shockingly well. You've proved that we don't need 16 bits, only 16 levels! Thanks so much for what you're doing, it's because of people like you that open source / community / collaborative AI will catch up with the strongest commercial AI, or at least contribute substantially and remain valuable. I've noticed that open source software is generally of much higher quality (for security, dev and research purposes at least).

I really appreciate the incredible efforts you've put into this project. It's wonderful to know that you intend keep it open-source and are considering the addition of more models. One particularly intriguing model is FLAN-UL2 20B. Though its MMLU performance is inferior to LLaMA 65B, it has already been instruction fine-tuned and comes with an Apache 2.0 license. This could potentially enable a lot of interesting real-world use cases, such as question answering across a collection of documents. It is however an encoder-decoder architecture and might be more work to get up and running.

One important difference is that FLAN-UL2 20B has a 4x bigger context length with 2048 tokens. That makes it more suitable for some use-cases such as in-context learning and retrieval-augmented generation.

CTranslate2 could be good inspiration, it's a C++ inference engine for Transformer language models with a focus on machine translation. It performs very well on CPUs and uses hardware acceleration effectively.

Thanks for the great open-source work! I want to play around with llama.cpp this weekend.

@ggerganov About these AVX2 routines, I had an idea about possible micro-optimizations.
Take a look, the only changed functions are dotProductCompressed40 and compressRow40:
https://gist.github.com/Const-me/4d30e1fc767ab314596e16e90f53b6f4

I like your phrase "Bloating the software with the ideas of today will make it useless tomorrow."

saw this post on hackernews
someone added openai plugins support to llama.cpp
https://github.com/lastmile-ai/llama-retrieval-plugin
https://blog.lastmileai.dev/using-openais-retrieval-plugin-with-llama-d2e0b6732f14

https://news.ycombinator.com/item?id=35315542

• If this software is much faster at inference than other methods because of quantification, we can expect people who train models to train their models directly as quantified or easily quantifiable ones, in order to be able to move them to the edge (or to cheaper data-centre inference hardware).

• Therefore putting too many special case optimisations in the main body of code here is a bad idea, because we want model designers to have a clear common target for inference.

Btw, thank you very much for this code which runs beautifully on my 10 year old Macbook Pro. Edmund.

There are libraries for hardware-accelerated matrix options on the raspberry pi: https://github.com/Idein/qmkl6

Very slow hope iPhone

This is a great effort and hope it reaches somewhere. Right now, the cost to run model for inference in GPU is cost-prohibitive for most ideas, projects, and bootstrapping startups compared to just using chatgpt API. Once you are locked in the ecosystem the cost which seems low for tokens, can increase exponentially. Plus, llama licensing is also ambiguous.

Thanks for the incredible work!

Inference at the edge

Based on the positive responses to whisper.cpp, and more recently, llama.cpp, it looks like there is a strong and growing interest for doing efficient transformer model inference on-device (i.e. at the edge).

image

The past few days, I received a large number of requests and e-mails with various ideas for startups, projects, collaboration. This makes me confident that there is something of value in these little projects. It would be foolish to let this existing momentum go to waste.

Recently, I've also been seeing some very good ideas and code contributions by many developers:

• @ameenba suggested a way to improve Whisper Encoder evaluation at the cost of accuracy #137. Ultimately, this allowed to demonstrate semi efficent short voice command recognition on a Raspberry Pi 4 Twitter
• @wangchou et. al. demonstrated how to evaluate the Whisper Encoder efficiently using Apple Neural Engine #548
• A person on Twitter showed me how to improve llama.cpp efficiency by 10% with simple SIMD change Twitter
• @Const-me kindly provided efficient AVX2 quantization routines #27
• and many other examples

The AI field currently presents a wide range of cool things to do. Not all of them (probably most) really useful, but still - fun and cool. And I think a lot of people like to work on fun and cool projects (for now, we can leave the "useful" projects to the big corps :)). From chat bots that can listen and talk in your browser, to editing code with your voice or even running 7B models on a mobile device. The ideas are endless and I personally have many of them. Bringing those ideas from the cloud to the device, in the hands of the users is exciting!

Naturally, I am thinking about ways to build on top of all this. So here are a few thoughts that I have so far:

• This project will remain open-source
• I would like to explore the application of this approach to other models and build more examples to demonstrate it
• I would be really happy to see developers join in and help advance further the idea of "inference at the edge"
• The strongest points of the current codebase are it's simplicity and efficiency. Performance is essential
• It's early to build a full-fledged edge inference framework. The code has to remain simple and compact in order to allow for quick and easy modifications. This helps to explore ideas at a much higher rate. Bloating the software with the ideas of today will make it useless tomorrow
• The AI models are improving at a very high rate and it is important to stay on top of it. The transformer architecture in it's core is very simple. There is no need to "slap" complex things on top of it
• Hacking small tools and examples is a great way to drive innovation. We should not get lost into software engineering problems. Especially at the beginning, the goal is to prototype and not waste time in polishing products
• And most of all, it's important to have fun in the process!

I hope that you share the hacking spirit that I have and would love to hear your ideas and comments about how you see the future of "inference at the edge".

Edit: "on the edge" -> "at the edge"

Hi,

I made a flutter mobile app with your project, i am really interested in this idea!

The repo : Sherpa Github
Playstore link : Sherpa

I will soon update to the latest version of llama, but if there is an optimisation for low end devices, I am extremely interested.

we will look back on this in 20 years, when telling the tales of cloud vs edge...

I really appreciate your support for memory-mapped files. Having the ability to "cheat" and run models I shouldn't be able to due to your efficient programming is simply a miracle.

what do you think it is that happened that people might consider that they "shouldn’t be able" to run large models on small hardware?
i’m old, in the 90s we did everything in under 64k, huge worlds, raytracers. there are still today yearly competitions around who can optimize code the best to run on small and slow systems.

感谢您令人惊叹的项目!

You guys need a donation page. This project keeps me from allowing these global poor overseas who fix OpenAI, and Gemini, and allowing a different system to be developed. I would be more than happy to throw a few dollars in the tip jar!