How to Run Ollama in Docker: CPU, NVIDIA and AMD Setups

By Web Asset House Studio · October 28, 2026

Note: Some future links on this site may be affiliate links. We never claim hands-on testing; guides are researched from the public sources listed in each article.

How we wrote this: A summary of Ollama's official Docker documentation and FAQ (linked below), checked October 28, 2026. Commands change between releases, so compare with the current docs before you run them.

Ollama publishes an official container image, ollama/ollama, on Docker Hub. Running it in Docker keeps Ollama separate from the rest of your system, makes it easy to update or remove, and pairs naturally with other containers such as Open WebUI. Here is the setup from Ollama's Docker documentation.

CPU only

The simplest version runs on any machine with Docker:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

What each part does:

-d runs the container in the background.
-v ollama:/root/.ollama stores downloaded models in a named Docker volume called ollama, so they survive when you remove or update the container.
-p 11434:11434 publishes Ollama's API port. 11434 is the same port the regular desktop install uses.
--name ollama lets you refer to the container by name in later commands.

CPU-only inference works, but it is much slower than a supported GPU for anything beyond small models.

NVIDIA GPUs

Docker containers can't use an NVIDIA GPU until you install the NVIDIA Container Toolkit on the host. Ollama's docs link to NVIDIA's installation guide and list the steps for apt and yum/dnf systems. After installing it, configure Docker and restart it:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then start Ollama with access to all GPUs:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

On NVIDIA JetPack (Jetson) systems, the docs say Ollama can't detect the JetPack version automatically. Pass JETSON_JETPACK=5 or JETSON_JETPACK=6 as an environment variable.

AMD GPUs

For AMD GPUs, Ollama uses a separate image tag, rocm, and passes the GPU devices through:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

The docs also note that Vulkan support is bundled into the standard ollama/ollama image and is enabled by default when the container can reach the GPU devices. OLLAMA_VULKAN=0 turns it off.

Run a model

With the container running, run a model inside it:

docker exec -it ollama ollama run llama3.2

The first run downloads the model into the ollama volume. Browse other models in the Ollama library. Before you pick a large one, check that it fits your memory. Our guide to what to check in a laptop for local AI covers the basics.

Because the container publishes port 11434, apps on the same machine can call it exactly as they would a desktop install. Our Ollama local API guide shows the requests.

Settings that matter in a container

Ollama is configured with environment variables, which you pass to Docker with -e. A few from the Ollama FAQ:

Variable	What it does
`OLLAMA_CONTEXT_LENGTH`	Default context window for loaded models (see our context length guide)
`OLLAMA_KEEP_ALIVE`	How long a model stays loaded in memory after a request
`OLLAMA_NUM_PARALLEL`	Parallel requests per model (default 1). The FAQ says required RAM scales with this times the context length
`OLLAMA_MAX_LOADED_MODELS`	How many models can stay loaded at once, if they fit in memory

Be careful with port 11434

Outside Docker, the FAQ says Ollama binds to 127.0.0.1 (this computer only) by default. Docker's own port publishing documentation warns that publishing a port is "insecure by default": a plain -p 11434:11434 makes it reachable not only from the host but from other machines too. Docker says that adding the localhost address limits access to the host:

docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama

If other devices do need access, the Ollama FAQ describes putting Ollama behind a proxy server such as Nginx, which is where you can add authentication. Never forward the port from your router to the internet.

Update or remove

To update, pull the new image, then remove and recreate the container with the same command. Your models stay in the ollama volume:

docker pull ollama/ollama
docker stop ollama && docker rm ollama

To remove everything, including downloaded models, also delete the volume with docker volume rm ollama.