llama.cpp for Beginners: Run a Local Model from the Command Line

By Web Asset House Studio · October 28, 2026

Note: Some future links on this site may be affiliate links. We never claim hands-on testing; guides are researched from the public sources listed in each article.

How we wrote this: A summary of the official llama.cpp README and server documentation on GitHub (linked below). We have not benchmarked any hardware or model. The project changes quickly, and command names and defaults have changed before, so check the README before you copy a command.

llama.cpp is the open-source engine underneath many local AI apps. Tools such as LM Studio use it to run GGUF model files. Using it directly gives you the newest features first and full control, at the cost of working in a terminal.

What it is

The llama.cpp README describes the project as "LLM inference in C/C++". Its stated goal is to run large language models (and vision-language models) "with minimal setup and state-of-the-art performance on a wide range of hardware". Points from the README worth knowing:

• Plain C/C++ with no dependencies. It's MIT-licensed.
• Apple silicon is "a first-class citizen" through Metal and Accelerate.
• Many GPU back ends. CUDA for Nvidia, HIP for AMD, and Vulkan and SYCL, among others.
• Quantization from 1.5-bit to 8-bit for "faster inference and reduced memory use". Our guide to what a GGUF file is explains names like Q4_K_M.
• CPU+GPU hybrid inference, so a model larger than your graphics card's memory can still be partly accelerated.

Install it

The README lists several options:

1. An install script from the project's site (one for macOS/Linux shells, one for Windows PowerShell).
2. Pre-built binaries from the GitHub releases page.
3. Docker images.
4. Building from source by following the build guide.

Pre-built releases are the lowest-effort path. Building from source matters mainly if you need a specific GPU back end that isn't in a release.

Your first chat and your first server

At the time of writing, the README's quick start shows two commands that download a model straight from Hugging Face and run it:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The first opens an interactive chat in your terminal. The second launches what the README calls an "OpenAI-compatible API server", with a built-in web chat page. The example model is very small, so it's a quick, low-memory way to confirm everything works before you download something bigger. Older tutorials use separate programs named llama-cli and llama-server. Follow the README for the version you installed.

Before choosing a bigger model, check that it fits. Private AI Computer's memory calculator estimates RAM needs by model size and quantization.

The local server: defaults that matter

The server documentation lists every option. Three are worth understanding before you run it:

The server also exposes a POST /v1/chat/completions endpoint, documented as OpenAI-compatible, so many existing apps and scripts can point at it by changing their base URL.

Privacy

Running the model locally means your prompts are processed on your computer rather than on a cloud service. Downloads still come from the internet: the -hf option fetches the model file from Hugging Face. For what "private" can and can't mean in practice, see Private AI Computer's explainer on what private AI means.

When to choose something else

If you'd rather click than type, our LM Studio guide covers a desktop app. Our Ollama walkthrough covers a tool with a simpler command set and a model library.

Sources

• llama.cpp README (GitHub, ggml-org)
• llama.cpp server documentation
• llama.cpp releases