llama.cpp
by ggerganovHigh-performance LLM inference in pure C/C++. Runs Llama-family models on CPUs, Apple Silicon, CUDA, ROCm, Metal — anywhere.
About llama.cpp
llama.cpp is a C++ open-source local llm runtimes project by ggerganov. High-performance LLM inference in pure C/C++. Runs Llama-family models on CPUs, Apple Silicon, CUDA, ROCm, Metal — anywhere. With 72,000 GitHub stars and 0forks, it's one of the Local LLM Runtimes tools worth knowing. You can download the source, browse it on GitHub, or read the full setup guide in the README below.
llama.cpp — guide
Inference without a datacentre
llama.cpp exists to answer one question: can a large language model run on the computer you already own? It is written in plain C/C++ with no Python runtime and no framework underneath, which is why it ends up everywhere — laptops, Apple Silicon, CUDA and ROCm cards, Raspberry Pis, phones. A great many "local AI" applications are a user interface with llama.cpp doing the actual work underneath.
GGUF and quantization, in plain terms
Models are distributed as GGUF files, and each comes in several quantization levels. Quantization stores the model's weights at lower precision — 4 bits instead of 16, say — which shrinks the file and the memory it needs by roughly a factor of four. It is a genuine trade, not free: heavier quantization degrades output quality, and the damage shows up first on reasoning and long-context tasks rather than on casual chat. The practical advice is to run the largest model your memory allows at a middling quantization, rather than a small model at full precision.
What it costs to run
Compile it or download a release binary, then fetch a GGUF file from Hugging Face and point the binary at it. There is no account and no API key. RAM is the real constraint — a 7B model at 4-bit needs roughly 5GB, and larger models scale from there. Everything stays on your machine.
When to pick something else
If you want a graphical application rather than a command line, several projects wrap this engine. If you need to call hosted models across providers instead, LiteLLM is the tool for that job.
From the project README
llama.cpp
[image: llama]
<div align="center">
<b>LLM inference in C/C++</b>
[[image: License: MIT]](https://opensource.org/licenses/MIT) [[image: Release]](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0) [[image: Nightly]](https://github.com/ggml-org/llama.cpp/releases?q=b) [[image: Server]](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml) [[image: Docker]](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml) [[image: Winget]](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
This is an excerpt from the project's own README, quoted here for reference. Read the full README on GitHub →
llama.cpp — FAQ
Do I need a GPU to run llama.cpp?
No. Running well on CPU is the reason the project exists, and it is particularly strong on Apple Silicon where CPU and GPU share memory. A CUDA or ROCm GPU makes it considerably faster, but it is an optimisation rather than a requirement.
What is a GGUF file?
GGUF is the model format llama.cpp uses — a single file containing the weights and the metadata needed to load them. You download one per model from Hugging Face, choosing a quantization level that fits your available memory.
Does quantization make the model worse?
Yes, measurably, though how much depends on the level. Light quantization is close to indistinguishable for everyday use; aggressive quantization degrades reasoning and long-context performance first. A larger model at moderate quantization usually beats a smaller one at full precision.
How much RAM do I need?
Roughly the size of the quantized model file plus overhead for context. A 7B model at 4-bit needs about 5GB; 13B roughly doubles that. If the file does not fit in memory, it will not load.