Build an AI Chatbot Locally: Ollama + Open WebUI + RAG with Your Documents

<div className="tldr-box"> <h3>Key Takeaways</h3> <ul> <li>Overview You can build a fully private, locallyrun AI chatbot that rivals ChatGPT — with the added ability to answer questions from your own documents.</li> <li>Using Ollama for model inference, Open WebUI for the chat interface, and ChromaDB for RetrievalAugmented Generation (RAG), you'll have a system that: Works 100% offline Keeps all data on your machine Answers questions from your PDFs, Word docs, and code files Costs nothing beyond your existing hardware Architecture ┌──────────┐ ┌──────────────┐ ┌──────────┐ │ Your │───▶│ Open WebUI │───▶│ Ollama │ │ Browser │ │ (Frontend) │ │ (LLM) │ └──────────┘ └──────┬───────┘ └──────────┘ │ ┌────────▼────────┐ │ ChromaDB (RAG) │ │ + Document │ │ Pipeline │ └─────────────────┘ Step 1: Install Ollama Install Ollama from $1 or via command line: bash Windows: Download installer from ollama.com macOS: brew install ollama Linux: curl fsSL https://ollama.com/install.sh | sh Pull a Model Choose a model based on your hardware: bash For most setups (8 GB+ VRAM) ollama pull llama3.1:8b For stronger hardware (24 GB+ VRAM) — better quality ollama pull qwen2.5:32b For maximum quality (40 GB+ RAM) — best RAG results ollama pull llama3.3:70b Test the Model bash ollama run llama3.1:8b "Hello, tell me about yourself" Step 2: Install Open WebUI Open WebUI is a ChatGPTlike interface that connects to Ollama.</li> <li>Docker Installation (Recommended) bash docker run d p 3000:8080 \ name openwebui \ restart always \ v openwebuidata:/app/backend/data \ ghcr.io/openwebui/openwebui:main NonDocker Installation bash git clone https://github.com/openwebui/openwebui.git cd openwebui Backend cd backend pip install r requirements.txt python app.py First Login 1.</li> <li>Open http://localhost:3000 in your browser 2.</li> </ul> </div>

Overview

You can build a fully private, locally-run AI chatbot that rivals ChatGPT — with the added ability to answer questions from your own documents. Using Ollama for model inference, Open WebUI for the chat interface, and ChromaDB for Retrieval-Augmented Generation (RAG), you'll have a system that:

• Works 100% offline
• Keeps all data on your machine
• Answers questions from your PDFs, Word docs, and code files
• Costs nothing beyond your existing hardware

Architecture

┌──────────┐    ┌──────────────┐    ┌──────────┐
│  Your     │───▶│  Open WebUI  │───▶│  Ollama   │
│  Browser  │    │  (Frontend)  │    │  (LLM)    │
└──────────┘    └──────┬───────┘    └──────────┘
                       │
              ┌────────▼────────┐
              │  ChromaDB (RAG) │
              │  + Document     │
              │  Pipeline       │
              └─────────────────┘

Step 1: Install Ollama

Install Ollama from ollama.com or via command line:

# Windows: Download installer from ollama.com
# macOS: brew install ollama
# Linux: curl -fsSL https://ollama.com/install.sh | sh

Pull a Model

Choose a model based on your hardware:

# For most setups (8 GB+ VRAM)
ollama pull llama3.1:8b

# For stronger hardware (24 GB+ VRAM) — better quality
ollama pull qwen2.5:32b

# For maximum quality (40 GB+ RAM) — best RAG results
ollama pull llama3.3:70b

Test the Model

ollama run llama3.1:8b "Hello, tell me about yourself"

Step 2: Install Open WebUI

Open WebUI is a ChatGPT-like interface that connects to Ollama.

Docker Installation (Recommended)

docker run -d -p 3000:8080 \
  --name open-webui \
  --restart always \
  -v open-webui-data:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Non-Docker Installation

git clone https://github.com/open-webui/open-webui.git
cd open-webui

# Backend
cd backend
pip install -r requirements.txt
python app.py

First Login

1. Open http://localhost:3000 in your browser
2. Create an admin account (first user becomes admin)
3. Select your model from the dropdown in the chat interface
4. Start chatting!

Step 3: Set Up RAG (Document Q&A)

RAG lets your chatbot answer questions based on your own documents.

Method A: Built-in Open WebUI RAG (Easiest)

Open WebUI has built-in RAG support:

1. Go to Settings → Documents
2. Click Upload Documents
3. Select PDF, DOCX, TXT, or code files
4. Open WebUI automatically chunks, embeds, and indexes them
5. In any chat, toggle "Use RAG" or type / to reference documents

Settings to configure:

| Setting | Recommended | Description | |---|---|---| | Chunk size | 1,000 tokens | Size of each document chunk | | Chunk overlap | 200 tokens | Overlap between chunks for context | | Top-K results | 3–5 | Number of chunks to retrieve | | Similarity threshold | 0.75 | Minimum relevance score |

Method B: ChromaDB + Custom Python (Advanced)

For more control over the RAG pipeline:

import chromadb
from chromadb.utils import embedding_functions
import ollama
import os

# Initialize ChromaDB
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="my_documents",
    embedding_function=embedding_functions.OllamaEmbeddingFunction(
        model_name="nomic-embed-text"
    )
)

# Add documents
def add_document(file_path, file_id):
    with open(file_path, "r", encoding="utf-8") as f:
        text = f.read()
    
    # Simple chunking
    chunks = [text[i:i+1000] for i in range(0, len(text), 800)]
    
    collection.add(
        documents=chunks,
        ids=[f"{file_id}_chunk_{i}" for i in range(len(chunks))]
    )

# Query with RAG
def query_with_rag(question):
    # Retrieve relevant chunks
    results = collection.query(
        query_texts=[question],
        n_results=3
    )
    
    context = "\n\n".join(results["documents"][0])
    
    # Generate response with context
    prompt = f"""Answer the question based on the following context:
    
Context:
{context}

Question: {question}

Answer:"""
    
    response = ollama.chat(model="llama3.1:8b", messages=[
        {"role": "user", "content": prompt}
    ])
    
    return response["message"]["content"]

# Example
print(query_with_rag("What is our company policy on remote work?"))

Step 4: Advanced Chatbot Features

Custom System Prompts

In Open WebUI, go to Settings → Personalization → set a system prompt:

You are a helpful assistant for Acme Corp. You help employees with:
- Company policy questions
- Technical documentation
- Code review
- Meeting summaries

Always be professional, concise, and accurate. 
When unsure, say "I don't know" rather than guessing.

Multiple Model Support

Ollama supports switching between models on the fly:

# Pull additional models
ollama pull mistral:7b          # Fast, smaller
ollama pull llama3.3:70b       # Best quality
ollama pull qwen2.5-coder:14b  # Code specialist

In Open WebUI, select any model from the dropdown per conversation.

Conversation History

Open WebUI stores all conversations locally. Features:

• Search through past conversations
• Export conversations as JSON or markdown
• Share conversations via generated links
• Organize into folders

Step 5: Making It Accessible

Local Network Access

Access your chatbot from other devices on your network:

# When running Open WebUI with Docker, expose on all interfaces
docker run -d -p 0.0.0.0:3000:8080 \
  --name open-webui \
  -v open-webui-data:/app/backend/data \
  -e WEBUI_SECRET_KEY=your-secret-key \
  ghcr.io/open-webui/open-webui:main

Access from other devices at http://YOUR_IP:3000.

HTTPS with Self-Signed Certificate

For secure access:

# Generate self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

# Run with SSL
# Configure reverse proxy (nginx/Caddy) with the cert

Alternative: Lobe Chat

Lobe Chat is another open-source frontend that supports Ollama:

docker run -d -p 3210:3210 \
  --name lobe-chat \
  lobehub/lobe-chat

Features over Open WebUI: More UI themes, plugin system, voice input.

Use Cases

| Use Case | How It Works | |---|---| | Personal knowledge base | Upload notes, books, articles — ask questions | | Company policy bot | Upload HR policies, employee handbook | | Code documentation | Upload API docs, code comments | | Research assistant | Upload papers, extract methodologies | | Meeting notes bot | Upload transcripts, query past meetings |

FAQ

How much does this cost?

Zero — all software is free and open-source. Your only cost is electricity (~$0.05–$0.20/hour depending on hardware).

Can I access it from my phone?

Yes — Open WebUI is responsive and works in any mobile browser. Access it at your computer's IP address:port from your phone on the same network.

How do I keep my data private?

All data stays on your machine. No data is sent to any external service. For maximum privacy, run on a laptop that never connects to the internet after initial setup.

What's the best hardware for this?

• Minimum: 16 GB RAM, 4 CPU cores, 20 GB storage
• Recommended: 32 GB RAM, 8 GB VRAM GPU, 100 GB SSD
• Ideal: 64 GB RAM, RTX 4090 (24 GB), 1 TB NVMe