Local development with Ollama
Ollama is the fastest on-ramp: one command runs a quantized model on your laptop with an OpenAI-compatible API, so course code works unchanged. Built for local dev, not high-throughput prod.
- Ollama installed (runs models locally, laptop-friendly)
Learning objectives
- Run an open model locally with Ollama in minutes.
- Use its OpenAI-compatible API so course code works unchanged.
- Customize a model with a Modelfile.
- Know when Ollama is the right tool (dev) vs not (high-throughput prod).
code/lm2-ollama/ in the course, with a README. Run the scripts or copy the configs directly.Ollama: the fastest way to local essential
Ollama is the simplest on-ramp: one command pulls and runs a quantized model on your laptop, with an OpenAI-compatible API. It's built for local development — experimenting, building, testing offline — not high-throughput production (that's vLLM, LM3).
run.sh# Install from ollama.com, then:
ollama pull llama3.1:8b # downloads a 4-bit quantized model
ollama run llama3.1:8b # interactive chat in your terminal
# It also serves an HTTP API on :11434 automatically.
This is the fastest possible way to run a real AI model on your own laptop — two commands, no cloud account, no API key. ollama is a free tool you install first (from ollama.com); these commands then fetch and run a model.
ollama pull llama3.1:8bdownloads the model file once.llama3.1is the model family (Meta's Llama),8bmeans the 8-billion-parameter size, and Ollama grabs a quantized (compressed) version so it fits on a normal machine.ollama run llama3.1:8bstarts a chat right in your terminal — you type, the model replies, all offline on your own computer.- The last comment is the important bit: Ollama also quietly starts a small web server on port 11434. That means other programs (like the Python code in the next lab) can talk to the model over HTTP, exactly like they'd talk to a cloud API.
What the output means: You get an interactive chat prompt, and in the background a local API at http://localhost:11434 that your code can call.
Try this: Swap 8b for a smaller tag like llama3.2:3b if your laptop is tight on memory — it downloads faster and runs lighter, at some quality cost.
Your code already speaks it essential
Ollama exposes an OpenAI-compatible endpoint, so the same client from the rest of the course works with only a base_url change — the recurring theme of IC6.
client.pyfrom openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") # key ignored
resp = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Say hello from my laptop."}],
)
print(resp.choices[0].message.content)
This shows the payoff of Ollama's built-in server: your normal Python code — the same OpenAI client used elsewhere in the course — talks to the local model with just one line changed. Nothing else about your code has to change.
from openai import OpenAIimports the standard client library. You are not calling OpenAI's cloud here — you're reusing their client shape.OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")is the one change that matters:base_urlpoints the client at the Ollama server running on your own machine. Theapi_keyis ignored (there's no billing locally), so any string works.client.chat.completions.create(model="llama3.1:8b", messages=[...])sends a chat request.messagesis the conversation — here oneuserturn asking the model to say hello.resp.choices[0].message.contentdigs the reply text out of the response object and prints it.
What the output means: The model's reply (a greeting) prints to your terminal — generated entirely on your laptop, with no network call.
Try this: Point base_url at this same address from any earlier course lab. Because the client shape is identical, most labs run unchanged against your local model.
Customize with a Modelfile intermediate
A Modelfile bakes a system prompt and parameters into a named model — a lightweight way to ship a preset without fine-tuning.
Modelfile# Modelfile
FROM llama3.1:8b
PARAMETER temperature 0.2
SYSTEM """You are a terse SRE assistant. Answer in at most 3 sentences."""
# Build and run your customized model:
# ollama create sre-bot -f Modelfile
# ollama run sre-bot
A Modelfile is a tiny recipe that bakes a personality and settings into a named model, so you don't have to repeat the system prompt every time. Think of it as saving a preset. It is not fine-tuning — it just wraps an existing model with defaults.
FROM llama3.1:8bsays "start from this base model" — the one you already pulled.PARAMETER temperature 0.2sets a default behaviour: low temperature means more focused, predictable answers (higher would be more creative/random).SYSTEM """You are a terse SRE assistant..."""bakes in a permanent system prompt — the role and rules the model always follows. The triple quotes just let the text span lines.- The two commented commands build the preset into a new named model (
ollama create sre-bot -f Modelfile) and then run it (ollama run sre-bot). From then onsre-botalways behaves like a terse SRE assistant.
What the output means: After create, you have a new local model sre-bot that answers tersely by default — no need to resend the system prompt each call.
Try this: Change the SYSTEM text to a different role (say a friendly tutor) and rebuild — you've made your own custom local model in seconds.
Exercise LM2.1 — Run a course lab locally
Context: Running a real course lab entirely offline — no key, no network — is the proof that local serving works end to end.
Your task: Install Ollama, pull an 8B model, and repoint an earlier course lab (a RAG query or an agent step) at localhost:11434.
Requirements:
- Install Ollama and pull an 8B model
- Repoint an existing lab at the local endpoint
- Confirm it runs fully offline — no API key, no network
- Verify the lab's behavior is unchanged
💡 Hint: Only the client's base_url (and a dummy key) should change — the lab code itself stays the same.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Ollama is the fastest on-ramp to a local model: one command pulls a quantized model, one runs it, and it serves an HTTP API on :11434 automatically — all offline, no API key.
Your task: Write the two commands (pull and run) and explain each token. (Needs Ollama installed.)
Requirements:
- Show the
ollama pullcommand and what it fetches (family/size/quant) - Show the
ollama runcommand and what it starts - Explain that pull downloads the quantized weights once
- Note that run also starts a background HTTP server on :11434
- Label it as needing Ollama installed
💡 Hint: pull fetches once; run starts an interactive chat plus a server other programs can call — name each token in the model tag.
Show solution
The fastest on-ramp. This is config for the runtime (needs Ollama installed):
# install from ollama.com, then:
ollama pull llama3.1:8b # download once: family=llama3.1, size=8B, 4-bit quant
ollama run llama3.1:8b # interactive chat in the terminal
# Ollama also serves an HTTP API on :11434 automaticallypull fetches the (quantized) weights once; run starts a local chat and a background server other programs can call. All offline, no API key.
Context: Ollama exposes an OpenAI-compatible endpoint, so existing course code works unchanged — only the base_url moves.
Your task: Model the chat request payload offline (stdlib), then show the one-line base_url swap the real client needs.
Requirements:
- Build the chat payload (model + messages) with the standard library, no server
- The payload matches the shape an OpenAI-style API expects
- Show the real call pointing an OpenAI client at
http://localhost:11434/v1 - Emphasize that only the
base_urlchanges; app code is untouched - Label the real call as needing Ollama running
💡 Hint: Because the API shape matches OpenAI's, the payload is identical and the only change is where the client points.
Show solution
First, build/validate the chat payload offline (runnable, no server):
import json
def chat_payload(model, user_msg):
return {"model": model,
"messages": [{"role": "user", "content": user_msg}]}
p = chat_payload("llama3.1:8b", "hello")
print(json.dumps(p, indent=2)) # exactly the body an OpenAI-style API expectsThe real call just points an OpenAI client at the local server (needs Ollama running):
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(**chat_payload("llama3.1:8b", "hello"))
print(r.choices[0].message.content)Because the API shape matches OpenAI's, only the base_url changes — your app code is untouched.
Context: A Modelfile bakes a base model, a system prompt, and parameters into a named model you can ollama create and reuse.
Your task: Write a Modelfile, then model a tiny parser (stdlib) that reads it back into a dict to prove the structure.
Requirements:
- Author a Modelfile with FROM, SYSTEM, and PARAMETER lines
- Parse it offline into a dict (base, system, params)
- Strip quotes from the SYSTEM value and collect PARAMETER name/value pairs
- Show the parsed structure
- Note the real build step is
ollama create
💡 Hint: Partition each line on the first space and branch on the keyword; the parser just confirms the file's shape, the build itself needs Ollama.
Show solution
The Modelfile is config (needs Ollama to build). First the file:
FROM llama3.1:8b
SYSTEM "You are a terse SRE assistant. Answer in one sentence."
PARAMETER temperature 0.2
PARAMETER num_ctx 4096Now parse it offline to confirm the shape (runnable):
def parse_modelfile(text):
mf = {"params": {}}
for line in text.strip().splitlines():
key, _, rest = line.strip().partition(" ")
if key == "FROM": mf["base"] = rest
elif key == "SYSTEM": mf["system"] = rest.strip('"')
elif key == "PARAMETER":
name, val = rest.split()
mf["params"][name] = val
return mf
MF = '''FROM llama3.1:8b
SYSTEM "You are a terse SRE assistant."
PARAMETER temperature 0.2'''
print(parse_modelfile(MF))
# then: ollama create sre-bot -f Modelfile && ollama run sre-botA Modelfile bakes the system prompt and defaults into a named model you can ollama create and reuse.
Context: Ollama is built for local development, not high-throughput production — that job belongs to vLLM (LM3). Knowing when a workload has outgrown it is the skill.
Your task: Write a decision helper that flags when a workload should move off Ollama.
Requirements:
- Recommend Ollama for single-user/offline/local development
- Recommend vLLM when concurrency is high or GPU batching is needed
- Take concurrent users and batching need as parameters
- Show a dev case (Ollama) and a production case (vLLM)
💡 Hint: Ollama shines for experimenting and offline building; once many users hit it concurrently, you need vLLM's batching.
Show solution
Match the tool to the load. Runnable:
def use_ollama(concurrent_users, needs_gpu_batching, offline_laptop):
if offline_laptop and concurrent_users <= 1:
return "Ollama -- local dev / single user, perfect"
if concurrent_users > 4 or needs_gpu_batching:
return "NOT Ollama -- use vLLM (LM3) for throughput/batching"
return "Ollama is fine for light multi-use; watch throughput"
print(use_ollama(1, False, True)) # Ollama
print(use_ollama(50, True, False)) # vLLMOllama shines for experimenting and building offline; once many users hit it concurrently, you need vLLM's batching.
Context: In dev you often want cheap local calls for simple work but a hosted frontier model for the hard cases — and privacy can override both.
Your task: Write a router that picks Ollama or a hosted API based on task difficulty and privacy.
Requirements:
- Sensitive tasks stay local on Ollama regardless of difficulty
- Hard tasks route to the hosted frontier model
- Simple, non-sensitive tasks default to local
- Show an easy case, a hard case, and a hard-but-sensitive case
💡 Hint: Check privacy first (it wins), then difficulty; the OpenAI-compatible API makes switching targets a one-liner.
Show solution
A hybrid router keeps dev cheap and private. Runnable:
def route(task):
if task.get("sensitive"):
return "ollama-local" # never leaves the machine
if task.get("difficulty", "easy") == "hard":
return "hosted-frontier" # quality matters
return "ollama-local" # cheap default for simple work
print(route({"difficulty": "easy"})) # ollama-local
print(route({"difficulty": "hard"})) # hosted-frontier
print(route({"difficulty": "hard", "sensitive": True})) # ollama-local (privacy wins)Local-first for simple/sensitive work, hosted for the hard cases — the OpenAI-compatible API makes swapping targets a one-liner.
Context: Before wiring an app to a local server, a readiness check catches the server-down / model-not-pulled / empty-completion failures up front instead of mid-request.
Your task: Model a smoke-test that checks the server is up, the model is pulled, and a trivial completion returns text.
Requirements:
- Check each condition: server reachable, wanted model available, sample completion non-empty
- Pass only if all checks pass
- Print a per-check PASS/FAIL line
- Model the decision offline; note the real version hits :11434
- Suggest gating app startup on this check
💡 Hint: The real version lists models via /api/tags then makes a tiny chat call; the logic is an all-of over the three checks so you fail fast.
Show solution
Model the smoke-test decision offline (the real version hits :11434). Runnable:
def smoke_test(server_up, models_available, wanted, sample_reply):
checks = []
checks.append(("server reachable on :11434", server_up))
checks.append((f"model '{wanted}' pulled", wanted in models_available))
checks.append(("sample completion non-empty", bool(sample_reply.strip())))
passed = all(ok for _, ok in checks)
for name, ok in checks:
print(("PASS " if ok else "FAIL ") + name)
return passed
ready = smoke_test(
server_up=True,
models_available=["llama3.1:8b", "qwen2.5:7b"],
wanted="llama3.1:8b",
sample_reply="Hello!",
)
print("READY" if ready else "NOT READY")Real version: GET /api/tags to list models, then a tiny /v1/chat call (needs Ollama running). Gate app startup on this so you fail fast, not mid-request.
✓ Checkpoint — you can move on when you can…
- Run an open model locally with Ollama.
- Use its OpenAI-compatible API from course code.
- Customize a model with a Modelfile.
- Say when Ollama fits (dev) and when it doesn't (scale).
Knowledge check check yourself
How can existing course code that uses the OpenAI client talk to a model running in Ollama, and what's the one change required?
Show answer
base_url to http://localhost:11434/v1 (the api_key is ignored). Most earlier labs then run unchanged against the local model.What is a Modelfile, and why is it not the same as fine-tuning?
Show answer
FROM a base model plus PARAMETER and SYSTEM settings) that bakes a system prompt and defaults into a new named model. It only wraps an existing model with presets so you don't resend the system prompt each call — it does not change the model's weights, so it isn't fine-tuning.