Local development with Ollama

Ollama: the fastest way to local essential

Here's the moment this chapter is built for. You're mid-way through the course, the free-tier credits on your hosted model just ran dry, and you're on a train with no signal. Or your legal team just said "that customer data cannot leave our network." Or you simply want to hammer a model with 500 test prompts at 2am without watching a meter tick. In every one of those cases the answer is the same: run the model on your own machine. The problem has always been that "run a model locally" used to mean a weekend of CUDA drivers, Python environments, and downloading raw weight files you had no idea how to load. Ollama collapses all of that into one command — which is exactly why it earns the first spot in this section.

The analogy that makes it click: think of Ollama as Docker for models. Docker took "set up this app with all its dependencies" and turned it into docker run; Ollama takes "download a multi-gigabyte model, quantize it, load it onto your GPU or CPU, and expose it over HTTP" and turns it into ollama run. You name a model, it fetches the right build for your hardware, and it just works. Under the hood it's driving llama.cpp (the same engine you'll meet again in LM4), but you never have to see that machinery.

Two words in that sentence are load-bearing, so let's be honest about them. The first is quantized: the model Ollama pulls is not the full-precision original but a compressed version where each weight is stored in roughly 4 bits instead of 16. That's what makes an 8-billion-parameter model fit in ~5 GB of memory instead of ~16 GB — the difference between "runs on your laptop" and "needs a data-center GPU." You trade a small, usually-imperceptible slice of quality for the ability to run at all (LM4 unpacks the trade in full). The second is local development: Ollama is tuned for you, one person, iterating — not for serving thousands of concurrent users. It answers one request at a time, comfortably; it does not do the industrial-strength batching that a production server needs. Reach for it to experiment, build, test offline, and keep private data private; reach for vLLM (LM3) when you need throughput.

The common mistake to avoid from day one: assuming "the model runs on my laptop" means "the model is as good as the hosted API." It won't be. An 8B model quantized to 4 bits is a genuinely capable assistant, but it is not GPT-4 or Claude Opus — it will hallucinate more, follow complex instructions less reliably, and struggle with long, subtle reasoning. That is completely fine for what this chapter is for: developing and testing the plumbing of your app cheaply and offline, then swapping the endpoint to a frontier model for the parts that need it. Judge Ollama as a development tool, not as a replacement for the best models, and you'll never be disappointed by it.

Ollama is the simplest on-ramp: one command pulls and runs a quantized model on your laptop, with an OpenAI-compatible API. It's built for local development — experimenting, building, testing offline — not high-throughput production (that's vLLM, LM3).

run.sh# Install from ollama.com, then:
ollama pull llama3.1:8b        # downloads a 4-bit quantized model
ollama run llama3.1:8b         # interactive chat in your terminal

# It also serves an HTTP API on :11434 automatically.

Your code already speaks it essential

Imagine the alternative for a second, because it's what makes this section quietly wonderful. Every model provider could invent its own request format — its own field names, its own way of packaging a conversation, its own response shape. If they did, switching from a hosted model to your local one would mean rewriting every call site in your app, testing all of it again, and maintaining two code paths forever. That friction is exactly what stops most people from ever trying a local model. Ollama removes it with a single design decision: it speaks the same language your code already knows.

The concrete magic is that Ollama exposes an OpenAI-compatible endpoint. "OpenAI-compatible" doesn't mean it calls OpenAI — it means it accepts requests and returns responses in the exact shape the widely-used openai Python client expects. That client shape became a de-facto industry standard (the way USB became the standard plug), so tools all over the ecosystem — LangChain, LlamaIndex, your own earlier course labs — can point at Ollama and just work. The endpoint lives at http://localhost:11434/v1, and the only thing you change is the base_url: where the client mails its request. Same envelope, different address.

Here's the useful mental model: an API client is really just a very fancy way of building an HTTP request and parsing the reply. The base_url is the street address on the envelope. When you point it at OpenAI's servers, your letter crosses the internet, gets billed, and comes back. When you point it at localhost:11434, the letter never leaves your machine — it goes to the little server Ollama started in the background in Lab LM2.1, gets answered by the model in your own RAM, and comes back. The rest of your program can't tell the difference, and that's the whole point: write your app once, run it against local or hosted models by flipping one string.

Two gotchas that trip people, so you don't lose an afternoon to them. First, the api_key: the OpenAI client requires one to be present, but Ollama ignores its value entirely (there's no billing on your laptop), so you pass any non-empty dummy string like "ollama". Leave it out and the client itself will refuse to start — the error looks like an auth problem but isn't. Second, "compatible" means most things line up, not every exotic feature. The everyday path — chat, streaming, temperature, system prompts — works cleanly; some provider-specific extras won't, and different models expose tool-calling to varying degrees. For the plumbing you build in this course, the compatibility is more than enough, and it's the same recurring theme you'll see again in IC6.

Ollama exposes an OpenAI-compatible endpoint, so the same client from the rest of the course works with only a base_url change — the recurring theme of IC6.

client.pyfrom openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key ignored
resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Say hello from my laptop."}],
)
print(resp.choices[0].message.content)

Customize with a Modelfile intermediate

Picture the small daily annoyance this solves. You've decided your local model should always answer as a terse SRE assistant, always at low temperature, always capped to a few sentences. Right now you achieve that by pasting the same system prompt and the same parameters into every single call — in every script, every notebook, every teammate's copy. It works, but it's fragile: someone forgets the system prompt, someone fat-fingers the temperature, and now your "model" behaves differently depending on who called it. A Modelfile fixes that by baking those settings into the model itself, so the presets travel with the name instead of living scattered across your code.

The analogy that lands it: if a plain model is raw ingredients, a Modelfile is a saved recipe with the oven already preheated. You write down "start from this base model, always use this system prompt, always these parameters," run ollama create, and out comes a new named model — sre-bot — that carries all of that baggage built in. From then on ollama run sre-bot is already terse, already focused, already an SRE assistant, with nothing to remember. If Docker's analogy from the first section still helps: a Modelfile is to a base model what a Dockerfile is to a base image — a short, declarative recipe (FROM, then a few directives) that produces a reusable, named artifact.

Now the mistake this section exists to prevent, because it's the one everybody makes: assuming a Modelfile is fine-tuning. It is not, and the difference is fundamental. Fine-tuning actually rewrites the model's weights — it teaches the model new behavior by training it on examples, a slow, GPU-hungry process that changes what the model knows. A Modelfile changes nothing about the weights. It only wraps the existing model in default settings — a system prompt, a temperature, a context size — that get applied automatically at call time. Think of it as configuration, not education: you're setting the model's defaults, not its capabilities. If your model can't do something, a Modelfile won't teach it; it'll just stop you from re-typing the setup. (When you genuinely need to change behavior the model can't already produce, that's the Fine-tuning track, FT1–FT6.)

Why this is the right amount of power for local dev: most of the time you don't need to retrain anything — you just need the model to reliably adopt a role and a set of knobs. A Modelfile gives you exactly that, in seconds, with zero training cost, and the result is a shareable artifact your whole team can pull and run identically. It's the 90% solution that's instant, and it keeps you from reaching for the 10% solution (fine-tuning) before you actually need it.

A Modelfile bakes a system prompt and parameters into a named model — a lightweight way to ship a preset without fine-tuning.

Modelfile# Modelfile
FROM llama3.1:8b
PARAMETER temperature 0.2
SYSTEM """You are a terse SRE assistant. Answer in at most 3 sentences."""

# Build and run your customized model:
#   ollama create sre-bot -f Modelfile
#   ollama run sre-bot

# install from ollama.com, then:
ollama pull llama3.1:8b   # download once: family=llama3.1, size=8B, 4-bit quant
ollama run  llama3.1:8b   # interactive chat in the terminal
# Ollama also serves an HTTP API on :11434 automatically

import json
def chat_payload(model, user_msg):
    return {"model": model,
            "messages": [{"role": "user", "content": user_msg}]}

p = chat_payload("llama3.1:8b", "hello")
print(json.dumps(p, indent=2))   # exactly the body an OpenAI-style API expects

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(**chat_payload("llama3.1:8b", "hello"))
print(r.choices[0].message.content)

FROM llama3.1:8b
SYSTEM "You are a terse SRE assistant. Answer in one sentence."
PARAMETER temperature 0.2
PARAMETER num_ctx 4096

def parse_modelfile(text):
    mf = {"params": {}}
    for line in text.strip().splitlines():
        key, _, rest = line.strip().partition(" ")
        if key == "FROM":      mf["base"] = rest
        elif key == "SYSTEM":  mf["system"] = rest.strip('"')
        elif key == "PARAMETER":
            name, val = rest.split()
            mf["params"][name] = val
    return mf

MF = '''FROM llama3.1:8b
SYSTEM "You are a terse SRE assistant."
PARAMETER temperature 0.2'''
print(parse_modelfile(MF))
# then: ollama create sre-bot -f Modelfile  &&  ollama run sre-bot

def use_ollama(concurrent_users, needs_gpu_batching, offline_laptop):
    if offline_laptop and concurrent_users <= 1:
        return "Ollama -- local dev / single user, perfect"
    if concurrent_users > 4 or needs_gpu_batching:
        return "NOT Ollama -- use vLLM (LM3) for throughput/batching"
    return "Ollama is fine for light multi-use; watch throughput"

print(use_ollama(1, False, True))    # Ollama
print(use_ollama(50, True, False))   # vLLM

def route(task):
    if task.get("sensitive"):
        return "ollama-local"                # never leaves the machine
    if task.get("difficulty", "easy") == "hard":
        return "hosted-frontier"             # quality matters
    return "ollama-local"                    # cheap default for simple work

print(route({"difficulty": "easy"}))                 # ollama-local
print(route({"difficulty": "hard"}))                 # hosted-frontier
print(route({"difficulty": "hard", "sensitive": True}))  # ollama-local (privacy wins)

def smoke_test(server_up, models_available, wanted, sample_reply):
    checks = []
    checks.append(("server reachable on :11434", server_up))
    checks.append((f"model '{wanted}' pulled", wanted in models_available))
    checks.append(("sample completion non-empty", bool(sample_reply.strip())))
    passed = all(ok for _, ok in checks)
    for name, ok in checks:
        print(("PASS " if ok else "FAIL ") + name)
    return passed

ready = smoke_test(
    server_up=True,
    models_available=["llama3.1:8b", "qwen2.5:7b"],
    wanted="llama3.1:8b",
    sample_reply="Hello!",
)
print("READY" if ready else "NOT READY")

Knowledge check check yourself