Getting Started with Ollama: Install, Run Models, and Build AI Apps Locally

<div className="tldr-box"> <h3>Key Takeaways</h3> <ul> <li>Ollama is a free, opensource tool that lets you run large language models locally on your computer.</li> <li>It wraps llama.cpp, provides an OpenAIcompatible API, and makes downloading and switching between models as simple as a single command.</li> <li>No cloud subscription needed Your data stays on your machine Works offline once models are downloaded Supports hundreds of opensource models Provides a local API endpoint compatible with OpenAI SDK Installation Windows Download the installer from $1 and run it.</li> <li>Ollama will install as a background service and be available in PowerShell or Command Prompt.</li> </ul> </div>

What is Ollama?

Ollama is a free, open-source tool that lets you run large language models locally on your computer. It wraps llama.cpp, provides an OpenAI-compatible API, and makes downloading and switching between models as simple as a single command.

Why Ollama?

• No cloud subscription needed
• Your data stays on your machine
• Works offline once models are downloaded
• Supports hundreds of open-source models
• Provides a local API endpoint compatible with OpenAI SDK

Installation

Windows

Download the installer from ollama.com and run it. Ollama will install as a background service and be available in PowerShell or Command Prompt.

# Verify installation
ollama --version

macOS

# Download from ollama.com or use Homebrew
brew install ollama

Linux

curl -fsSL https://ollama.com/install.sh | sh

Your First Model

Once installed, open a terminal and pull your first model:

# Pull Llama 3.1 8B (~4.7 GB download)
ollama pull llama3.1:8b

# Run it interactively
ollama run llama3.1:8b

You'll see a prompt where you can chat directly with the model. Type /bye to exit.

Running Inference

Interactive Mode

ollama run llama3.1:8b
>>> What is the capital of France?
The capital of France is Paris.

One-Shot Generation

ollama run llama3.1:8b "Write a haiku about programming"

API Mode (REST)

Ollama automatically starts an API server at http://localhost:11434 when running.

# Using curl
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Python SDK

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.1:8b",
        "prompt": "Explain quantum computing in one sentence",
        "stream": False
    }
)
print(response.json()["response"])

Model Management

List Installed Models

ollama list

Pull Specific Models

# Popular models
ollama pull llama3.3:70b     # 70B model (needs ~40 GB)
ollama pull mistral:7b       # Fast 7B model
ollama pull qwen2.5:72b      # Strong 72B model
ollama pull phi3:14b         # Efficient 14B model
ollama pull codellama:34b    # Code-focused model

Remove a Model

ollama rm llama3.1:8b

Custom Modelfiles

Modelfiles let you customize model behavior, system prompts, and parameters.

Basic Example

Create a file named Modelfile:

FROM llama3.1:8b

# Set system prompt
SYSTEM """You are a helpful coding assistant. Always provide code examples."""

# Set parameters
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

Build and Run

# Build the custom model
ollama create my-coder -f Modelfile

# Run it
ollama run my-coder

Advanced Modelfile

FROM qwen2.5:72b

# System prompt for a JSON-only API
SYSTEM """You are a JSON API. Respond only with valid JSON. Do not include any explanatory text."""

# Parameter tuning
PARAMETER temperature 0.1
PARAMETER top_p 0.95
PARAMETER num_ctx 16384
PARAMETER stop "```"

# Template override
TEMPLATE """{{ .System }}
User: {{ .Prompt }}
Assistant: """

Open WebUI Setup

Open WebUI provides a ChatGPT-like browser interface for Ollama.

Docker Installation

docker run -d -p 3000:8080 \
  --name open-webui \
  --restart always \
  -v open-webui-data:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Then visit http://localhost:3000.

Non-Docker Installation

# Clone the repository
git clone https://github.com/open-webui/open-webui.git
cd open-webui

# Install and run
pip install -r requirements.txt
python app.py

Performance Tips

| Setting | Impact | |---|---| | Model size | 7B models run on 8 GB RAM; 70B models need 40 GB+ | | Quantization | Q4_K_M reduces memory by ~75% with minimal quality loss | | Context length | Reduce num_ctx to 2048 if memory is tight | | Batch size | Set num_batch to 512 for faster prompt processing | | GPU offloading | Set num_gpu to offload layers to GPU |

Common Commands Reference

| Command | Purpose | |---|---| | ollama serve | Start the API server | | ollama create | Build model from Modelfile | | ollama show | Show model details | | ollama cp | Copy a model | | ollama pull | Download a model | | ollama push | Upload a model to registry | | ollama list | List installed models | | ollama rm | Remove a model | | ollama run | Run a model |

FAQ

Can I use OpenAI SDK with Ollama?

Yes — Ollama's API is OpenAI-compatible. Just change the base URL to http://localhost:11434/v1 and use any model name as the model ID.

How much storage do I need?

Models range from 3.8 GB (Phi-3 mini) to 40+ GB (Llama 3.3 70B). Allocate at least 50 GB for a starter setup.

Can I run Ollama without a GPU?

Yes — Ollama runs on CPU. Smaller models (7B) run at 5–10 tok/s on modern CPUs. Larger models are slower but usable for batch tasks.