llama.cpp
FreeThe foundation nearly every local LLM tool is built on
A large language model inference engine written in C/C++ that runs on CPUs and a wide range of GPUs, using quantization to squeeze big models onto ordinary hardware. The GGUF model format originated here, and this project sits underneath Ollama, LM Studio and much of the local AI ecosystem.
Why we picked it
If Ollama is the face of local LLMs, llama.cpp is the foundation. 123,874 stars, MIT-licensed.
Those .gguf model files you download in other tools use a format this project defined, and the
friendly GUIs — Ollama, LM Studio, Jan — lean heavily on it or its derivatives for inference.
Its central contribution is quantization: compressing model weights from 16-bit floats down to 4-, 5- or 8-bit integers, cutting size to a fraction while keeping accuracy loss acceptable. That step is what makes consumer hardware viable at all — without it a 7B model needs well over ten gigabytes of VRAM; with it, four to five will do.
The other distinguishing trait is breadth of hardware support. CUDA, ROCm, Metal, Vulkan and SYCL are all covered; it runs especially well on Apple silicon thanks to unified memory, and it runs on pure CPU too (slowly, but it runs). People have it working on Raspberry Pis and phones.
When to use it directly instead of Ollama:
- You want the newest model support. New architectures usually land here first and reach higher-level tools afterwards. Releases are extremely frequent — currently b10428, with builds most days.
- You want fine control over inference. Context length, GPU layer offloading, batch size and the full range of sampling parameters are all exposed.
- You want to embed it. It offers a C API and a server mode, and many products compile it straight in.
The barrier is real: this is a command-line, source-level tool. Prebuilt release packages exist, but getting your specific GPU accelerated often means picking the right build or compiling it yourself. If you would rather not, install Ollama or LM Studio — you will still be running llama.cpp, just with the layer wrapped for you.