Running Local AI Models With Ollama: A Complete Setup Guide

I self-host Gemma and Qwen locally, and every time someone asks how to get started, I end up re-explaining the same handful of things that Ollama's own docs cover, but spread across separate pages for install, CLI, API, and Modelfiles. This is that walkthrough as one page: install, run a model, understand the commands you'll actually use, talk to it over the API, and customize it with a Modelfile.

Installing Ollama

Pick your platform.

macOS or Linux, one command:

curl -fsSL https://ollama.com/install.sh | sh

Windows, in PowerShell:

irm https://ollama.com/install.ps1 | iex

Prefer a GUI installer over a piped script? macOS and Windows both have one: ollama.com/download/Ollama.dmg and ollama.com/download/OllamaSetup.exe.

Docker, if you'd rather run it in a container:

docker pull ollama/ollama

Once installed, Ollama runs as a background service. You don't need to think about starting it again unless you've stopped it manually, in which case ollama serve brings it back up in the foreground.

Pulling and running your first model

Two ways to get a model onto your machine. ollama pull downloads it without running it:

ollama pull gemma4

ollama run does both: pulls the model if you don't have it yet, then drops you into an interactive chat with it.

ollama run gemma4

You can also skip the interactive session and pass a prompt directly:

ollama run gemma4 "Explain what a Modelfile is in two sentences"

Model names follow a name:tag format when you want a specific variant instead of the default, for example ollama run qwen3.5:14b to pull a specific parameter size rather than whatever the default tag resolves to. The full catalog of available models lives at ollama.com/library, and it's worth checking there for the exact tags a given model offers before you pull, since sizes and quantization levels vary by model family.

The CLI commands you'll actually reach for

This is the table Ollama's own docs make you assemble from several different pages:

ollama ps is the one people forget exists and then wonder why their machine's memory usage looks strange. A model stays loaded in memory for a few minutes after your last request to it, so if you're switching between several large models, checking ollama ps before pulling another one tells you what's actually occupying resources right now, not just what's installed.

Talking to it over the API instead of the CLI

Ollama runs an HTTP server on port 11434 by default, which is what makes it useful for anything beyond a terminal chat, like wiring it into your own scripts or tools. The two endpoints you'll use most:

/api/generate for a single prompt and response, no conversation history:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Why is the sky blue?"
}'

/api/chat when you want multi-turn conversation, passing the message history yourself with each request:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{
    "role": "user",
    "content": "Why is the sky blue?"
  }],
  "stream": false
}'

Both endpoints stream responses by default, token by token, which is the right behavior for a chat UI and the wrong behavior for a script expecting one clean JSON blob back. Set "stream": false when you want the whole response in a single response body, as in the example above.

If you're working in Python or JavaScript instead of raw HTTP, official client libraries wrap this for you: pip install ollama or npm i ollama, both of which talk to the same local server under the hood.

Customizing a model with a Modelfile

A Modelfile is how you turn a base model into something with its own default behavior baked in, a system prompt, generation parameters, or both, without having to pass that configuration on every single request. The format is a plain text file with one instruction per line:

FROM gemma4
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM """You are a terse assistant that answers technical questions in as few words as possible without losing accuracy."""

FROM is the only required instruction, and it points at a base model already pulled locally, a Safetensors model directory, or a .gguf file if you're bringing your own weights. PARAMETER sets runtime behavior: temperature controls randomness (lower means more deterministic, more repetitive; higher means more varied, less predictable), num_ctx sets how much context window the model gets to work with, and there are others worth knowing about, top_p, seed, stop, that tune generation in narrower ways. SYSTEM sets the default system prompt, so you don't have to send it with every API call or retype it at the start of every chat session.

Once the file is written, building the model is one command:

ollama create my-terse-assistant -f Modelfile

From there, ollama run my-terse-assistant behaves like any other model on your machine, just with your customizations already applied.

A few things worth knowing before you go further

GPU usage is automatic. If Ollama detects a supported GPU, it uses it without any configuration on your part; if it doesn't, it falls back to CPU, which works for smaller models but gets noticeably slower as parameter counts climb. If a model feels unreasonably slow, checking whether it's actually landing on the GPU is the first thing to verify, not the model choice itself.

Model size and quantization matter more than model name recognition when you're picking what to run locally. A smaller, more aggressively quantized version of a well-known model will often outperform a larger, less optimized one on a machine with limited memory, and the tag on ollama.com/library is where you find out which quantization you're actually pulling before you commit disk space and download time to it.

That's the whole loop: install, pull, run, talk to it over the API or the CLI, and customize it with a Modelfile once the defaults stop fitting what you need. Everything past this point is really just picking the right model for the job, which is a question the model library answers better than any guide could.