Local AI Core Concepts

Concept Map

Here's how the pieces fit together. You pick a model, quantize it to fit your hardware, load it into an inference engine, and interact through a CLI, API, or web UI.

1. Models

A model is a pre-trained neural network that generates text. Open-source models are released by companies and researchers for anyone to download and run. The landscape evolves fast, but these are the major families:

2. Parameters & Model Sizes

Parameters are the "knowledge weights" of the model. More parameters = more knowledge and better reasoning, but more memory required.

3. Quantization

Quantization compresses a model's weights from high-precision floats (16-bit) to lower-precision integers (8-bit, 4-bit). This is the magic that makes 70B models run on consumer GPUs.

GGUF is the standard file format for quantized models. It's used by llama.cpp, Ollama, and LM Studio. You'll find GGUF files on Hugging Face (the "GitHub of AI models").

4. Inference Engines

An inference engine loads the model into memory and generates text. It handles tokenization, GPU acceleration, and the API server.

5. Context Window

The context window is how much text the model can see at once — your system prompt, conversation history, and the current message all count. Measured in tokens (roughly 3/4 of a word).

# Check a model's context window in Ollama
ollama show llama3.1 --modelfile | grep num_ctx

6. OpenAI-Compatible API

Most local inference engines expose an OpenAI-compatible REST API. This means any app built for ChatGPT (code editors, chatbots, automation tools) can work with your local model by just changing the base URL.

# Same API format as OpenAI, but pointed at localhost
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1",
    "messages": [{"role": "user", "content": "Explain Docker in one sentence"}]
  }'

# Python with the official OpenAI library
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",  # Point to Ollama
    api_key="not-needed"                    # Local = no auth
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Explain Docker in one sentence"}]
)
print(response.choices[0].message.content)