The open-weight landscape
Hosted APIs are the default; open weights win on data-residency, offline, control, and high-volume cost. This chapter maps the families, licenses, and the size-vs-hardware reality.
Besides paid APIs like Claude, there are open-weight models (Llama, Mistral, Qwen…) whose files you can download and run on your own hardware. Why bother? Keeping data in-house, working offline, or saving money at scale. This section takes you from "run a model on your laptop in one command" to "serve one properly for a team" — and how to decide honestly when it's worth it versus just using an API.
The words you'll hear (in plain terms):
| Term | What it actually means |
|---|---|
| open-weight model | a model whose files you can download and run yourself (Llama, Mistral, Qwen…). |
| Ollama | the easiest tool to run such a model locally — one command, works on a laptop. |
| vLLM | a production-grade server for running open models fast for many users. |
| GGUF / quantized | a compact model file format that runs even on CPU / Apple Silicon. |
| self-host vs API | run it yourself (control, fixed cost) vs pay a provider per token (easy, no ops). |
What you need before starting:
- The AWS section's cost-model idea is handy background, not required.
- Python basics; comfort running a command in a terminal.
- Any modern laptop works for the local labs; a GPU helps for the serving ones.
New to the topic? Read this box, then take the chapters in order — each section is tagged essential → expert so you always know the depth you're at.
Learning objectives
- Name the major open-weight model families and their licenses.
- Match model size to the hardware needed to run it.
- Explain 'open weights' vs 'open source' and why the license matters.
- Choose a starting model for a given constraint.
Why run open models at all essential
Hosted APIs (Claude, etc.) are the right default — managed, capable, no ops. But open-weight models earn their place when you need data to never leave your infra, offline/air-gapped operation, full control of the model, or cost predictability at high volume. This track makes running them practical.
This little flow is the whole decision this chapter is about: should you use a hosted API or run an open model yourself? Read it left to right.
- The left box (Your requirement) is the honest starting point — what does your situation actually demand? Data that can't leave your network? Offline use? Tight control? Low cost at huge volume?
- The middle box (Hosted API) is the default recommendation: a provider like Claude runs the model for you — nothing to manage. Most projects should start here.
- The right box (Open-weight self-host) is the alternative you pick only when your requirement demands it — you download the model's files and run them on your own hardware, trading convenience for control.
- The arrows read as "leads you to consider" — a requirement points you at the default first, then to self-hosting only if the default doesn't fit.
In short: Hosted API is the sensible default; open-weight self-host is what you reach for when data-residency, offline operation, control, or high-volume cost forces the issue.
The families essential
| Family | Sizes | Note |
|---|---|---|
| Llama (Meta) | 1B–405B | broad ecosystem; community license |
| Mistral / Mixtral | 7B–8x22B | strong small models; MoE options |
| Qwen (Alibaba) | 0.5B–72B+ | strong multilingual + coding |
| Gemma (Google) | 2B–27B | efficient, permissive-ish |
| Phi (Microsoft) | mini–medium | small, punch above weight |
Size vs hardware intermediate
The practical constraint is memory. A rough rule at common quant levels (from IC2):
| Model size | FP16 VRAM | 4-bit VRAM | Runs on |
|---|---|---|---|
| 7–8B | ~16 GB | ~5–6 GB | one consumer GPU / good laptop (4-bit) |
| 13B | ~26 GB | ~9 GB | one bigger GPU |
| 70B | ~140 GB | ~40 GB | multi-GPU (IC6 tensor-parallel) |
Exercise LM1.1 — Pick your model
Context: The model you choose under a real constraint drives the rest of the track — everything downstream assumes that pick.
Your task: For a real or hypothetical constraint (e.g. "data can't leave our network, one A10 GPU"), pick a family, size, and quant level and justify it against the license and the memory table.
Requirements:
- State the constraint explicitly (hardware, residency, commercial use)
- Pick a model family, size, and quantization level
- Justify the size against the VRAM/memory table
- Justify the choice against the license
💡 Hint: Work from the constraint inward: VRAM sets the size, residency/commercial-use sets the license, and both together pick the family.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Hosted APIs are the sane default; open weights earn their place only for a specific reason — data-residency, offline/air-gapped, full control, or high-volume cost.
Your task: Write a helper that names which of those four reasons a given requirement triggers (or says the hosted API is the right default).
Requirements:
- Map each of the four open-weights reasons to a triggering requirement
- Return the matching reason when a requirement fires one
- Fall back to "hosted API default" when none applies
- Show a triggering case and a non-triggering case
💡 Hint: A small requirement→reason table with a default; if nothing matches, the managed API is the cheaper, saner choice.
Show solution
Encode the four reasons the lesson gives. Runnable stdlib:
REASONS = {
"data must stay on our infra": "data-residency",
"must work offline / air-gapped": "offline",
"we need full control of the model": "control",
"very high token volume": "cost-at-scale",
}
def why_open(requirement):
for k, reason in REASONS.items():
if k in requirement.lower():
return f"open weights -> {reason}"
return "hosted API is the right default here"
print(why_open("data must stay on our infra")) # data-residency
print(why_open("summarize a few docs a day")) # hosted defaultOpen weights earn their place for a specific reason — if none applies, the managed API is the cheaper, saner default.
Context: A rough but load-bearing rule: FP16 weights need ~2 bytes per parameter. That one number sets the hardware floor for any open model.
Your task: Write a calculator that, given a parameter count in billions, estimates the VRAM in GB for the weights at FP16.
Requirements:
- Use ~2 bytes per parameter for FP16
- Convert bytes to GB correctly (1024-based)
- Report weights-only VRAM for a few sizes (e.g. 7B, 13B, 70B)
- Note this is the floor, not the total (activations/KV add more)
💡 Hint: params × 1e9 × 2 bytes, divided into GB; a 7B lands near 13 GB, a 70B near 130 GB.
Show solution
2 bytes/param at FP16. Runnable:
def vram_fp16_gb(params_billion):
bytes_per_param = 2 # FP16
return params_billion * 1e9 * bytes_per_param / (1024**3)
for b in [7, 13, 70]:
print(f"{b}B -> ~{vram_fp16_gb(b):.1f} GB (weights only, FP16)")
# 7B -> ~13.0 GB, 13B -> ~24.2 GB, 70B -> ~130.4 GBWeights alone: a 7B fits a 16GB card, a 70B needs multiple GPUs. Activations/KV cache add more — this is the floor, not the total.
Context: The size→VRAM wall is hard at FP16: given a GPU, only certain model sizes fit — and you must reserve headroom for the KV cache.
Your task: Write a selector that returns which model sizes fit at FP16 on a given GPU, leaving headroom.
Requirements:
- Reuse the FP16 weights estimate
- Reserve a fraction of VRAM for KV cache/activations before filtering
- Return the sizes that fit within the usable budget
- Show a small card and a large card fitting different sets
💡 Hint: Shrink the GPU's VRAM by a headroom fraction, then keep the sizes whose weights fit; quantization (LM4) is what moves the wall later.
Show solution
Reserve headroom for KV cache, then filter. Runnable:
def vram_fp16_gb(pb):
return pb * 1e9 * 2 / (1024**3)
def fits(gpu_gb, sizes=(7, 13, 34, 70), kv_headroom=0.30):
usable = gpu_gb * (1 - kv_headroom) # leave room for KV cache/activations
return [s for s in sizes if vram_fp16_gb(s) <= usable]
print("24GB card fits FP16:", fits(24)) # [7, 13]
print("80GB card fits FP16:", fits(80)) # [7, 13, 34, 70]Quantization (LM4) shrinks the weights so bigger models fit — but at FP16 the size->VRAM wall is hard.
Context: "Open weights" (downloadable) is not "open source" (permissive license). Whether you can ship a model is governed by its license, not by the download.
Your task: Write a checker that, given a license, says what you may do — commercial use? redistribute? restrictions?
Requirements:
- Cover permissive (Apache/MIT), community (open weights with clauses), and research-only licenses
- Report commercial-use and redistribute permissions per license
- Flag research-only as not shippable to production
- Handle an unknown license by telling the user to read it first
💡 Hint: A license→permissions table; the key lesson is that "you can download it" never means "you can ship it".
Show solution
The license, not the download, governs use. Runnable:
LICENSES = {
"apache-2.0": {"commercial": True, "redistribute": True, "note": "permissive OSS"},
"mit": {"commercial": True, "redistribute": True, "note": "permissive OSS"},
"llama-community": {"commercial": True, "redistribute": True,
"note": "open weights, with acceptable-use + scale clauses"},
"research-only": {"commercial": False, "redistribute": False,
"note": "weights available but NOT for production"},
}
def check(license_id):
L = LICENSES.get(license_id.lower())
if not L:
return "unknown license -- read it before shipping"
return f"commercial={L['commercial']} redistribute={L['redistribute']} ({L['note']})"
print(check("apache-2.0")) # fully permissive
print(check("research-only")) # cannot ship"You can download it" does not mean "you can ship it". Always check the license against your intended use.
Context: The starting model falls out of the axes together — VRAM, license, and deployment — not from picking the biggest name.
Your task: Given VRAM, whether commercial use is required, and whether offline is required, recommend a size tier and a posture.
Requirements:
- Pick the largest FP16-fitting size for the GPU (with headroom)
- Fall back to a quantized path when nothing fits at FP16
- Require a permissive license when redistribution/commercial use is needed
- Require self-hosting when offline is required
- Return a concrete recommendation combining all three
💡 Hint: Compute the size ceiling from VRAM first, then layer the license and deployment constraints on top; no FP16 fit points to a quantized GGUF (LM4).
Show solution
Fold size + license + deployment into one pick. Runnable:
def vram_fp16_gb(pb): return pb * 1e9 * 2 / (1024**3)
def recommend(gpu_gb, commercial, offline):
usable = gpu_gb * 0.7
biggest = max([s for s in (7, 13, 34, 70) if vram_fp16_gb(s) <= usable] or [0])
if biggest == 0:
return "no FP16 fit -- use a quantized GGUF (LM4) or a smaller model"
lic = "permissive (Apache/MIT) if you must redistribute" if commercial else "any"
deploy = "self-host required" if offline else "self-host optional"
return f"start ~{biggest}B; license: {lic}; {deploy}"
print(recommend(24, commercial=True, offline=True)) # ~13B, permissive, self-host
print(recommend(8, commercial=False, offline=False)) # quantized pathThe starting model falls out of hardware + license + deployment — not from picking the biggest name.
Context: Defending a model choice to a team means a repeatable filter (hardware + license) then a quality ranking — a shortlist, not a vibe.
Your task: Build a scorer over candidate models (size, license, quality tier, VRAM need) that filters by hardware and license, then ranks survivors by quality.
Requirements:
- Each candidate carries params, license, and a quality score
- Exclude candidates whose FP16 weights won't fit the GPU (with headroom)
- Exclude non-permissive licenses when commercial use is required
- Rank the survivors by quality, best first
- Show a run excluding a research-only model and ranking the rest
💡 Hint: Filter first (fit + license), then sort by quality; a research-only model should drop out even if it's the highest quality.
Show solution
Turn the landscape into a defensible ranked shortlist. Runnable:
CANDIDATES = [
{"name": "A-8B", "params": 8, "license": "apache-2.0", "quality": 6},
{"name": "B-70B", "params": 70, "license": "llama-community","quality": 8},
{"name": "C-13B", "params": 13, "license": "research-only", "quality": 7},
]
PERMISSIVE = {"apache-2.0", "mit", "llama-community"}
def vram_fp16_gb(pb): return pb * 1e9 * 2 / (1024**3)
def shortlist(cands, gpu_gb, need_commercial):
usable = gpu_gb * 0.7
ok = []
for c in cands:
if vram_fp16_gb(c["params"]) > usable:
continue # won't fit
if need_commercial and c["license"] not in PERMISSIVE:
continue # license blocks prod
ok.append(c)
return sorted(ok, key=lambda c: c["quality"], reverse=True)
for c in shortlist(CANDIDATES, gpu_gb=80, need_commercial=True):
print(c["name"], "quality", c["quality"])
# B-70B first (fits 80GB, commercial-OK), then A-8B; C-13B excluded (research-only)A repeatable filter (hardware + license) then a quality ranking is how you defend a model choice instead of arguing by vibes.
✓ Checkpoint — you can move on when you can…
- Name the major open-weight families and their sizes.
- Match model size to hardware at FP16 and 4-bit.
- Explain open-weights vs open-source and the license risk.
- Choose a starting model for a constraint.
Knowledge check check yourself
Why does "open weights" not mean "open source," and why is that a legal rather than a technical concern?
Show answer
Roughly what hardware does a 7–8B model need at FP16 versus 4-bit, and why does the chapter recommend starting at 7B/4-bit?