How to Run an AI Model on Your Own Computer with Ollama (Beginner Walkthrough)
Note: Some future links on this site may be affiliate links. We never claim hands-on testing; guides are researched from the public sources listed in each article.
How we wrote this: A how-to based on Ollama's official documentation (linked below). We have not benchmarked any hardware. Commands and model names change often, so check the official docs before you rely on a specific step.
Running an AI model on your own computer used to mean Python environments and command-line juggling. Tools like Ollama have made it a download-and-run job. This guide covers the first hour: installing it, running a model, knowing where the files go, and understanding why your hardware sets the limits.
Step 1: Install Ollama
Ollama is available for macOS, Windows and Linux from the official download page. On Windows, the system requirements list Windows 10 22H2 or newer, plus a recent NVIDIA driver if you have an NVIDIA card, or a ROCm or Vulkan-capable driver for AMD Radeon GPUs.
After installing, open a terminal and type ollama to start the guided setup. The quickstart notes that you can sign in for cloud models or choose a local model. If privacy is the reason you're here, choose local.
Step 2: Run your first model
The command reference uses this as its example:
ollama run gemma4
The first run downloads the model, which can be several gigabytes, and then opens a chat prompt. Other everyday commands from the same reference:
ollama pull <model>downloads a model without starting a chat.ollama lslists the models you've downloaded.ollama rm <model>deletes one to free disk space.
Browse available models in the Ollama library. Smaller models load faster and fit on more machines, and larger ones tend to answer better but need more memory.
Step 3: Know where the files live
Per the Ollama FAQ, models are stored here by default:
| System | Default model folder |
|---|---|
| macOS | ~/.ollama/models |
| Linux | /usr/share/ollama/.ollama/models |
| Windows | C:\Users\%username%\.ollama\models |
If your main drive is small, set the OLLAMA_MODELS environment variable to put models on another disk.
Step 4: Keep it local
Two settings from the FAQ matter if you care about privacy:
- Local-only mode. The FAQ says that when you run locally, Ollama doesn't see your prompts or data. You can also switch off the cloud features entirely, either by setting
"disable_ollama_cloud": truein~/.ollama/server.jsonor with the environment variableOLLAMA_NO_CLOUD=1. Restart Ollama afterwards. - Network exposure. By default Ollama listens only on
127.0.0.1port 11434, which means only your own computer can reach it. ChangingOLLAMA_HOSTto0.0.0.0opens it to your network. Don't do that on a shared or public network unless you know why you need it.
For a deeper look at what "private" does and doesn't cover, see What "private AI" really means on Private AI Computer.
Why your hardware sets the limits
A model has to fit in memory to run well. Two things drive how much memory you need:
- Model size and quantization. Quantized files (often in the GGUF format used by llama.cpp-based tools) trade a little quality for a much smaller memory footprint. Private AI Computer has a guide to choosing a quantization.
- Context length. Ollama's context-length docs say the default depends on your video memory: 4k tokens under 24 GiB of VRAM, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. Raising it increases memory use. More detail: context length and the KV cache.
On the GPU side, Ollama's GPU documentation lists support for NVIDIA cards with compute capability 5.0 or newer (with driver 550+), a set of AMD Radeon cards through ROCm, plus additional AMD support through Vulkan.
If you're not sure what your machine can handle, the Private AI Computer calculator estimates which models fit a given amount of memory.
Quick troubleshooting
- Very slow answers: the model may not fit in GPU memory and is running partly on the CPU. Try a smaller model or a smaller quantization.
- Disk filling up: run
ollama lsandollama rmthe models you don't use. - Weird squares in the Windows terminal: the Windows docs note that older terminal fonts on Windows 10 may not render Ollama's progress characters. Change the terminal font.