Ollama: Run LLMs Locally

How Ollama Works

Ollama wraps llama.cpp (a highly optimized C++ inference engine) with a user-friendly CLI and a model registry. When you ollama run a model, it:

1. Checks if the model is downloaded (if not, pulls it from the Ollama library)
2. Detects your GPU (NVIDIA CUDA, AMD ROCm, or Apple Metal)
3. Loads the model into GPU memory (or CPU RAM as fallback)
4. Starts a chat session or serves the API

Installation & First Run

Install

# macOS / Linux (one-line installer)
curl -fsSL https://ollama.com/install.sh | sh

# macOS (Homebrew)
brew install ollama

# Windows: download from https://ollama.com/download

Run your first model

# Download and start chatting (auto-downloads on first run)
ollama run llama3.1

# Smaller model for limited hardware
ollama run phi3

# Coding-focused model
ollama run deepseek-coder-v2

Model Management

# List downloaded models
ollama list

# Pull a model without starting it
ollama pull mistral

# Show model details (size, quantization, parameters)
ollama show llama3.1

# Remove a model to free disk space
ollama rm codellama

# Copy/rename a model
ollama cp llama3.1 my-custom-llama

Custom Models with Modelfile

A Modelfile is like a Dockerfile but for AI models. It lets you customize system prompts, temperature, context window, and even create derivative models from base models.

# Modelfile for a coding assistant
FROM llama3.1

# System prompt
SYSTEM """
You are a senior software engineer. You write clean, efficient code with
clear explanations. When asked to write code, always include comments
explaining the approach. Use modern best practices.
"""

# Parameters
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
PARAMETER top_p 0.9

# Create and run your custom model
ollama create coding-assistant -f Modelfile
ollama run coding-assistant

Key Modelfile instructions

API Integration

Ollama serves an OpenAI-compatible API at http://localhost:11434. Any tool that works with OpenAI's API works with Ollama.

Chat completion

# Standard OpenAI-compatible endpoint
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1",
    "messages": [
      {"role": "system", "content": "You are a Python expert."},
      {"role": "user", "content": "Write a function to flatten a nested list"}
    ],
    "temperature": 0.3
  }'

Python integration

from openai import OpenAI

# Point OpenAI client at Ollama
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # any string works
)

# Works exactly like the OpenAI API
response = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a Python expert."},
        {"role": "user", "content": "Explain list comprehensions"}
    ]
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Write a haiku about coding"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Ollama native API

# Ollama's own endpoint (simpler format)
curl http://localhost:11434/api/generate \
  -d '{"model": "llama3.1", "prompt": "Why is the sky blue?", "stream": false}'

# Embedding generation
curl http://localhost:11434/api/embeddings \
  -d '{"model": "llama3.1", "prompt": "Hello world"}'

Popular Integrations

Interview Questions