Cut Ollama's Memory Use: Flash Attention and K/V Cache Quantization (q8_0, q4_0)
Note: Some future links on this site may be affiliate links. We never claim hands-on testing; guides are researched from the public sources listed in each article.
How we wrote this: A plain-English summary of the official Ollama FAQ, checked October 29, 2026. We haven't measured these settings ourselves. Effects depend on the model and task.
Long contexts eat memory because the model keeps a key/value (K/V) cache of everything in the conversation. Ollama offers two server settings to shrink it.
Flash Attention
The Ollama FAQ calls Flash Attention a feature of most modern models that can significantly reduce memory use as context grows. Ollama turns it on automatically when the backend and devices support it.
- Force it on:
OLLAMA_FLASH_ATTENTION=1 - Turn it off:
OLLAMA_FLASH_ATTENTION=0
K/V cache quantization
With Flash Attention on, the K/V cache itself can be quantized using OLLAMA_KV_CACHE_TYPE:
| Type | Memory vs. default | FAQ notes |
|---|---|---|
f16 |
Baseline (default) | High precision, highest memory |
q8_0 |
About half | Very small loss in precision, usually no noticeable impact. Recommended if not using f16 |
q4_0 |
About a quarter | Small-to-medium loss, may be more noticeable at long context |
Two caveats from the FAQ:
- It's a global setting, so every model uses the same cache type.
- Models with a high GQA count (the FAQ names Qwen2) may lose more precision from cache quantization.
The FAQ says you may need to experiment to find the right balance between memory and quality.
How to set it
These are server environment variables. Set them, then restart Ollama:
- Mac app:
launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0, then restart the app. - Linux (systemd):
systemctl edit ollama.service, addEnvironment="OLLAMA_KV_CACHE_TYPE=q8_0"under[Service], then runsystemctl daemon-reloadandsystemctl restart ollama. - Windows: quit Ollama, add the variable under Edit environment variables for your account, then start Ollama again.
When it helps
- You want a bigger context (see context length) without running out of VRAM.
- You're raising parallel requests (see concurrency), since each one adds context.
ollama psshows a model split between CPU and GPU and you'd like it fully on the GPU (see GPU support).
This is separate from the model's own quantization (for example a Q4_K_M GGUF). See what is a GGUF file and AI model memory calculator.