What Is a GGUF File? How to Read Names Like Q4_K_M Before You Download
Note: Some future links on this site may be affiliate links. We never claim hands-on testing; guides are researched from the public sources listed in each article.
How we wrote this: An explainer based on Hugging Face's GGUF documentation and the official llama.cpp, LM Studio and Ollama docs (linked below). We have not benchmarked quality or speed for any file type.
Browse a model page on Hugging Face and you'll often find a dozen files with names like model-Q4_K_M.gguf and model-Q8_0.gguf. Picking the right one decides whether the model loads, how fast it runs, and how good its answers are.
What GGUF is
Hugging Face's GGUF documentation describes GGUF as a binary format optimized for quickly loading and saving models, which makes it efficient for inference. It was developed by the creator of llama.cpp, the C/C++ inference engine that many desktop apps build on. Unlike tensor-only formats such as safetensors, a GGUF file also carries a standard set of metadata. Hugging Face lists llama.cpp, LM Studio, GPT4All and Ollama among the tools that use it.
Hugging Face has a built-in viewer for GGUF metadata and tensor info, and you can filter the model search to GGUF files.
What the letters and numbers mean
Most GGUF files are quantized: the model's numbers are stored at lower precision to shrink the file and memory use, at some cost in quality. LM Studio's download docs put it simply: the options are copies of the same model at varying fidelity, and you should choose a 4-bit option or higher if your machine can handle it.
From Hugging Face's quantization table:
| Label | What it is | Bits per weight (per HF docs) |
|---|---|---|
| F16 / BF16 | 16-bit floating-point formats | 16 |
| Q8_0 | 8-bit round-to-nearest, listed as a legacy type | not listed |
| Q6_K | 6-bit "k-quant" | 6.5625 |
| Q5_K | 5-bit k-quant | 5.5 |
| Q4_K | 4-bit k-quant | 4.5 |
| Q3_K | 3-bit k-quant | 3.4375 |
| Q2_K | 2-bit k-quant | 2.625 |
| IQ4_XS, IQ3_S, IQ2_XS … | "i-quants" using an importance matrix | 4.25, 3.44, 2.31 … |
Files often add a suffix such as _S or _M to mark a variant of a type. Private AI Computer's guide to choosing a quantization points out that a label such as Q4_K_M isn't exactly four bits for every byte, because files mix precisions and include metadata.
How to choose a file
- Start from your memory. The file has to fit in RAM or VRAM with room left for the conversation's context. Private AI Computer's memory calculator estimates this for common models.
- Prefer 4-bit or higher when it fits, as LM Studio's docs suggest. Go lower only if nothing else fits, and expect weaker answers.
- Test with your own tasks. Quality loss depends on the model and the job. Try a few real (non-sensitive) prompts before settling.
- Check the license of the base model, not just the uploader's page. See our license guide.
Using a GGUF file you've downloaded
- llama.cpp. Its README shows downloading and running a model straight from Hugging Face with
llama cli -hf <repo>, or starting an OpenAI-compatible server withllama serve -hf <repo>. It supports 1.5-bit to 8-bit quantization and Apple Silicon, NVIDIA, AMD and Vulkan backends. - Ollama. The import guide says to create a
ModelfilecontainingFROM /path/to/file.gguf, then runollama create my-model. Ollama doesn't quantize GGUF files during import, so pick the quantization you want before downloading. See our Ollama walkthrough. - LM Studio. Search for the model inside the app and pick a file. Our LM Studio guide walks through it.