The open-weight landscape

Why run open models at all essential

Start with the mistake almost everyone makes first. You discover that models like Llama and Mistral are free to download, and the reflex is instant: "Why would I ever pay Claude or OpenAI per token when I can just run one myself?" So you spin up a GPU box, wrestle with drivers, get a model answering — and three weeks later you're maintaining an inference server, paying for a GPU that sits idle 90% of the day, and getting answers that are a notch behind the hosted model you were trying to avoid. The download was free; the operation was not. That gap between "free to download" and "free to run in production" is exactly what this section is here to make honest.

Here's the mental model to hold onto: think of a hosted API like hailing a taxi and self-hosting like buying a car. A taxi is the right choice almost always — no maintenance, no parking, no insurance, you pay only when you ride. Buying a car only pays off when you have a specific reason: you drive constantly (high volume), you need to go somewhere taxis won't (offline/air-gapped), you're carrying something you can't hand to a stranger (data residency), or you need to modify the vehicle itself (full control). Buying a car "because it's cheaper per mile" while ignoring insurance, fuel, and the driveway you need to park it in is the trap above. So the honest question is never "open or hosted?" in the abstract — it's "do I have one of those four specific reasons?"

The common trap in one line: people compare the API's per-token price against the GPU's per-token price and declare self-hosting cheaper — forgetting that the GPU costs the same whether it serves a million requests or zero. A hosted API scales its cost to your usage; a self-hosted GPU charges you rent around the clock. That's why the diagram below puts "your requirement" first and the hosted API in the middle as the default: you earn your way out of the default by pointing at a concrete need, not by doing napkin math on token prices alone. Read the four reasons that follow as a checklist — if you can't tick at least one honestly, the taxi wins.

Hosted APIs (Claude, etc.) are the right default — managed, capable, no ops. But open-weight models earn their place when you need data to never leave your infra, offline/air-gapped operation, full control of the model, or cost predictability at high volume. This track makes running them practical.

The families essential

The open-weight world looks chaotic until you realise it's really just a handful of "families." Newcomers get overwhelmed scrolling a model hub — thousands of names, each with cryptic suffixes like -instruct, -70B, -GGUF, -AWQ — and freeze, unsure which is "the real one." The calming truth: nearly all of them descend from a few base families, each published by one big lab, and the thousands of variants are just community re-packagings (different sizes, fine-tunes, or quantizations) of those originals. Learn the families and the chaos collapses into a short, readable menu.

A useful analogy: a family is like a car marque — Toyota, Honda, BMW. "Llama" or "Qwen" is the marque; "Llama 8B Instruct" is a specific model like a "Corolla," and the GGUF you download from some user is that same car with an aftermarket trim. Two things follow. First, models within a family share a lineage — training recipe, tokenizer conventions, prompt format — so once you can drive one Llama, the others feel familiar. Second, and this is the part people miss: the family determines the license, because the license is set by the lab that trained the base model, not by whoever re-uploaded a quantized copy. A Mistral fine-tune is still bound by Mistral's terms no matter whose account it lives under.

So how do you actually pick a family? Not by chasing whoever topped a leaderboard last week — those rankings churn constantly and the differences at the top are often within noise for your task. Pick on the axes that stay true: the size range it offers (can it go small enough for your hardware?), its strengths (Qwen leans multilingual and coding; Phi punches above its weight at tiny sizes; Mistral offers efficient mixture-of-experts options), the ecosystem around it (is it well-supported in the serving tools you'll meet in LM2–LM3?), and — decisively — its license, covered in the warning just below. The table maps the current major families to those axes. Treat the sizes and notes as a general orientation, not a spec sheet: labs ship new versions constantly, so the exact numbers drift, but the shape of the landscape — a few labs, broad size ranges, license as the real differentiator — is stable. Read across each row asking one question: "given my hardware and my legal constraints, could this family even be in the running?"

Size vs hardware intermediate

This is the section that saves you from the single most common beginner faceplant: downloading a 70B model onto a laptop and watching it either crash outright or crawl so slowly it's unusable. The instinct is to grab the biggest, best-sounding model — bigger must be better, right? — but a model isn't like a movie file you can stream in chunks. To run at all, essentially all of its weights must sit in fast memory at once. Ask for more than the machine has and it doesn't run slower; it doesn't run. Understanding the size-vs-hardware wall before you download is what turns "why won't this work?" into a five-second sanity check.

The core idea is refreshingly countable, so let's demystify the jargon. A model's "size" — 7B, 70B — is its number of parameters (7 billion, 70 billion), the individual numbers it learned during training. Each parameter has to be stored in memory, and how many bytes each one takes is set by the precision: FP16 (the common full-fidelity format) spends 2 bytes per parameter, so the arithmetic is almost embarrassingly simple — parameters × 2 bytes = memory for the weights. A 7B model is ~14 GB; a 70B is ~140 GB. That's it; that's the wall. (Quantization, the "4-bit" column, is the escape hatch — it stores each weight in roughly half a byte instead of two, shrinking the same model ~4×, which is why a 7B that needs 16 GB at FP16 slips into 5–6 GB at 4-bit. You'll go deep on that in IC2 and LM4.)

Two things trip people up even after they know the formula. First, "VRAM" versus "RAM": the numbers below refer to VRAM — the memory on a GPU, which is fast enough to run a model at a usable speed. Your machine's ordinary system RAM can technically hold a model too, but running there is often painfully slow — one important exception is Apple-Silicon Macs, whose unified memory the GPU can use directly, which is why a MacBook punches above its weight for local models. Second, and the classic mistake: the weights are the floor, not the ceiling. A running model also needs room for the KV cache (the growing memory of the conversation, from IC3) and activations, which is why you never fill a GPU to 100% of the weight size — you leave headroom, typically 20–30%, or you'll run fine on a short prompt and crash on a long one. So read the table below as "what fits with breathing room," not "the exact minimum." Once these numbers are instinct, choosing a model stops being aspirational and becomes arithmetic: your hardware sets a ceiling, and you pick the best model that lives under it.

The practical constraint is memory. A rough rule at common quant levels (from IC2):

REASONS = {
    "data must stay on our infra": "data-residency",
    "must work offline / air-gapped": "offline",
    "we need full control of the model": "control",
    "very high token volume": "cost-at-scale",
}
def why_open(requirement):
    for k, reason in REASONS.items():
        if k in requirement.lower():
            return f"open weights -> {reason}"
    return "hosted API is the right default here"

print(why_open("data must stay on our infra"))    # data-residency
print(why_open("summarize a few docs a day"))       # hosted default

def vram_fp16_gb(params_billion):
    bytes_per_param = 2               # FP16
    return params_billion * 1e9 * bytes_per_param / (1024**3)

for b in [7, 13, 70]:
    print(f"{b}B -> ~{vram_fp16_gb(b):.1f} GB (weights only, FP16)")
# 7B -> ~13.0 GB, 13B -> ~24.2 GB, 70B -> ~130.4 GB

def vram_fp16_gb(pb):
    return pb * 1e9 * 2 / (1024**3)

def fits(gpu_gb, sizes=(7, 13, 34, 70), kv_headroom=0.30):
    usable = gpu_gb * (1 - kv_headroom)     # leave room for KV cache/activations
    return [s for s in sizes if vram_fp16_gb(s) <= usable]

print("24GB card fits FP16:", fits(24))   # [7, 13]
print("80GB card fits FP16:", fits(80))   # [7, 13, 34, 70]

LICENSES = {
    "apache-2.0": {"commercial": True,  "redistribute": True,  "note": "permissive OSS"},
    "mit":        {"commercial": True,  "redistribute": True,  "note": "permissive OSS"},
    "llama-community": {"commercial": True, "redistribute": True,
                        "note": "open weights, with acceptable-use + scale clauses"},
    "research-only":   {"commercial": False, "redistribute": False,
                        "note": "weights available but NOT for production"},
}
def check(license_id):
    L = LICENSES.get(license_id.lower())
    if not L:
        return "unknown license -- read it before shipping"
    return f"commercial={L['commercial']} redistribute={L['redistribute']} ({L['note']})"

print(check("apache-2.0"))       # fully permissive
print(check("research-only"))    # cannot ship

def vram_fp16_gb(pb): return pb * 1e9 * 2 / (1024**3)

def recommend(gpu_gb, commercial, offline):
    usable = gpu_gb * 0.7
    biggest = max([s for s in (7, 13, 34, 70) if vram_fp16_gb(s) <= usable] or [0])
    if biggest == 0:
        return "no FP16 fit -- use a quantized GGUF (LM4) or a smaller model"
    lic = "permissive (Apache/MIT) if you must redistribute" if commercial else "any"
    deploy = "self-host required" if offline else "self-host optional"
    return f"start ~{biggest}B; license: {lic}; {deploy}"

print(recommend(24, commercial=True, offline=True))   # ~13B, permissive, self-host
print(recommend(8,  commercial=False, offline=False)) # quantized path

CANDIDATES = [
    {"name": "A-8B",  "params": 8,  "license": "apache-2.0",     "quality": 6},
    {"name": "B-70B", "params": 70, "license": "llama-community","quality": 8},
    {"name": "C-13B", "params": 13, "license": "research-only",  "quality": 7},
]
PERMISSIVE = {"apache-2.0", "mit", "llama-community"}

def vram_fp16_gb(pb): return pb * 1e9 * 2 / (1024**3)

def shortlist(cands, gpu_gb, need_commercial):
    usable = gpu_gb * 0.7
    ok = []
    for c in cands:
        if vram_fp16_gb(c["params"]) > usable:
            continue                                   # won't fit
        if need_commercial and c["license"] not in PERMISSIVE:
            continue                                   # license blocks prod
        ok.append(c)
    return sorted(ok, key=lambda c: c["quality"], reverse=True)

for c in shortlist(CANDIDATES, gpu_gb=80, need_commercial=True):
    print(c["name"], "quality", c["quality"])
# B-70B first (fits 80GB, commercial-OK), then A-8B; C-13B excluded (research-only)

Knowledge check check yourself