Running AI models locally with Ollama: privacy, speed, and zero API costs

Every time you send a prompt to Claude, GPT-4, or Gemini, your data leaves your machine and goes to a third-party server. For most use cases that is fine. But for sensitive codebases, confidential documents, medical data, legal work, or simply not wanting to pay per token at scale — running a capable AI model locally changes the picture entirely. Ollama is the tool that makes this practical in 2026.

What Ollama is

Ollama is an open-source tool that makes running large language models on your own machine as simple as running a Docker container. It handles model downloading, quantisation, GPU/CPU routing, and exposes a local REST API (compatible with the OpenAI API format) so your existing code works without changes.

# Install Ollama (macOS)
brew install ollama

# Or download from ollama.com for Windows/Linux

# Pull and run a model
ollama run llama3.2

# That's it — you now have a local AI
>>> Hello! Tell me about yourself.
I'm Llama 3.2, a large language model made by Meta...

Which models run locally in 2026

The best local models have improved dramatically. Here is what runs well on consumer hardware:

Model	Size	RAM needed	Best for
Llama 3.2 3B	2GB	4GB	Fast responses, simple tasks
Llama 3.2 8B	5GB	8GB	General use, good quality/speed balance
Llama 3.1 70B	40GB	64GB	Near frontier quality, needs high-end machine
Mistral 7B	4GB	8GB	Strong coding, fast
Qwen2.5-Coder 7B	5GB	8GB	Code-focused tasks
Gemma 2 9B	6GB	8GB	Google’s local option, instruction-following
DeepSeek-R1 8B	5GB	8GB	Reasoning tasks, strong maths

A modern MacBook with 16GB RAM will run 7-8B models comfortably. With an M3 Max or M4 Pro with 32GB+ unified memory, 70B models are feasible. On Linux with an Nvidia GPU, models load into VRAM and run fast — an RTX 4090 (24GB VRAM) handles 70B models with quantisation.

Using Ollama from your code

Ollama exposes an OpenAI-compatible API at http://localhost:11434. Any code that works with the OpenAI SDK can point at Ollama instead:

from openai import OpenAI

# Point at local Ollama instead of OpenAI's servers
client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama'  # Required by the SDK but not used by Ollama
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Explain async/await in Python in 3 sentences"}
    ]
)

print(response.choices[0].message.content)

// Node.js / TypeScript — same pattern
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',
});

const response = await client.chat.completions.create({
  model: 'llama3.2',
  messages: [{ role: 'user', content: 'Write a SQL query to find duplicate emails' }],
});

console.log(response.choices[0].message.content);

What quantisation means

Language models are normally stored as 32-bit or 16-bit floating-point numbers. Quantisation reduces this — typically to 4-bit or 8-bit — which shrinks the model size and memory footprint at the cost of a small quality reduction. The tradeoff is usually worth it: a 4-bit quantised 70B model uses roughly the same memory as a full-precision 13B model, with significantly better quality than the 13B.

Ollama handles quantisation automatically — when you pull a model, you are getting an already-quantised version optimised for your hardware. You can specify quantisation level explicitly: ollama pull llama3.1:70b-instruct-q4_K_M for a specific 4-bit variant.

When local AI makes sense

• Privacy-sensitive work — legal documents, medical records, financial data, proprietary code you cannot send to external APIs
• High-volume automated tasks — processing thousands of documents where API costs would add up
• Offline or air-gapped environments — no internet required once the model is downloaded
• Development and testing — iterate fast against a local model, then switch to a frontier model for production

When cloud AI is still better

Local 7-8B models are capable but they are not Claude Opus or GPT-4o. For complex reasoning, nuanced writing, difficult coding tasks, or anything where quality is paramount, frontier models are still significantly better. The practical approach in 2026: use local models for tasks where privacy or cost drives the decision, use cloud models where quality is the priority.