Ollama Concurrency: Running Several Models and Parallel Requests (NUM_PARALLEL, MAX_LOADED_MODELS)
Note: Some future links on this site may be affiliate links. We never claim hands-on testing; guides are researched from the public sources listed in each article.
How we wrote this: A plain-English summary of the official Ollama FAQ on concurrent requests, checked October 29, 2026. Defaults can change between versions, so check the FAQ for yours.
Two kinds of concurrency
The Ollama FAQ describes two levels:
- Several models loaded at once, if there's enough memory (system RAM for CPU inference, VRAM for GPU inference).
- Parallel requests to one model, if there was enough memory when the model loaded.
If a new model doesn't fit, new requests queue until it can load. Idle models are unloaded to make room, and queued requests run in order. On GPUs, a new model must fit completely in VRAM to load alongside the others.
The settings and their defaults
| Variable | What it does | Default (per the FAQ) |
|---|---|---|
OLLAMA_MAX_LOADED_MODELS |
Most models loaded at once, if they fit | 3 × the number of GPUs, or 3 for CPU |
OLLAMA_NUM_PARALLEL |
Parallel requests each model handles | 1 |
OLLAMA_MAX_QUEUE |
Requests queued before Ollama rejects more | 512 |
When the queue is full, the server responds with a 503 (overloaded) error.
The memory cost of parallelism
Parallel requests multiply the context. The FAQ's example: a 2K context with 4 parallel requests becomes an 8K context, with the extra memory that implies. Required RAM scales with NUM_PARALLEL × context length. Before raising OLLAMA_NUM_PARALLEL, check memory headroom with ollama ps, and consider a smaller context (see context length) or a quantized K/V cache (see Flash Attention and K/V cache).
Multiple GPUs
If a model fits on one GPU, Ollama loads it there, which the FAQ says usually performs best because it reduces data transfer across the PCI bus. Otherwise it spreads the model across all GPUs.
Windows with Radeon GPUs
The FAQ notes these currently default to 1 loaded model because of a ROCm memory-reporting limitation. You can raise it, but don't load more than fits in VRAM.
A sensible starting point
For a single user on a desktop, the defaults are fine. For a small team sharing one machine through Open WebUI, raise OLLAMA_NUM_PARALLEL to 2–4 only if ollama ps shows the model on 100% GPU with memory to spare. Set variables as described in keep_alive and server settings, and keep the server local-only unless you've secured your network.
Related: API usage metrics and errors.