Quantized inference on your hardware

You don't always have a GPU intermediate

Here's the moment this chapter is built for. You've read the whole "run open models" track, you're excited to try Llama on your own machine — and then you check your laptop and there's no NVIDIA GPU in it. Maybe you're on a MacBook, maybe a cheap cloud VM, maybe a Raspberry-Pi-class edge box on a factory floor with no internet. The naive reaction is "well, I guess I can't run these models." That reaction is wrong, and this chapter is the proof. You can run a genuinely capable 8-billion-parameter model on hardware that cost a few hundred dollars and has no dedicated graphics card at all.

Why the "no GPU" wall feels so solid — and why it isn't. A model's weights are just a giant pile of numbers. Trained models store each number as a 16-bit float (fp16), and an 8B model in fp16 is roughly 16 GB of weights that all have to sit in fast memory. GPUs win because their VRAM is enormous and their arithmetic is massively parallel — so the whole assumption becomes "you need a big GPU to hold and multiply all those numbers." The escape hatch is to attack the size of the numbers themselves. If you store each weight in 4 bits instead of 16, the same 8B model shrinks to about a quarter of its footprint — roughly 4 GB — small enough to live in ordinary system RAM. That's quantization (the deep version is in IC2), and it's the single trick that turns "impossible on a laptop" into "runs fine."

Think of it like an audio file. A studio master is huge and pristine; an MP3 throws away detail your ear barely notices and lands at a tenth of the size. Quantization does the same to model weights — rounds each number to a coarser grid, loses a sliver of precision, and buys you a model that fits. The engine that plays these compressed files is llama.cpp, a lean C++ inference runtime, and the file format it reads is GGUF — a single self-contained file holding the quantized weights plus everything needed to run them. The pairing is deliberately humble: no CUDA, no Python model-loading dance, no gigabytes of framework — just a file and a runtime that runs anywhere, including on the CPU and on Apple Silicon's unified memory.

The common mistake is treating this as a lesser, "toy" path. It isn't a downgrade of GPU serving — it's a different tool for a different job. GPU serving (LM3) exists to push high throughput to many concurrent users; GGUF/llama.cpp exists to run a model at all where no GPU is present, or to keep everything on one private machine for offline and privacy-sensitive work. You'll pay for it in speed — a CPU generates tokens far slower than a GPU, and you'll feel it — but "slower and working" beats "fast and impossible" every time you're on the wrong hardware. Keep that framing as you read on: the rest of the chapter is really two decisions stacked together — which quant level fits your memory, and which runtime fits your hardware and load.

GPU serving (LM3) is ideal but not always available — laptops, edge devices, CPU-only servers. llama.cpp and the GGUF format (from IC2) run quantized models efficiently on CPU and Apple Silicon, so you can run real models with no GPU.

GGUF quant levels advanced

Now the decision that actually trips people up. You go to download a model and discover it doesn't ship as one file — it ships as a dozen, with cryptic names like Q4_K_M, Q5_K_M, Q8_0, Q3_K_S. Same model, same parameter count, wildly different file sizes. Which one do you grab? Pick blindly and you'll either download a file too big to load (and crash on start), or a file so aggressively compressed the model starts talking nonsense. This section is the one skill that makes the choice deliberate instead of a guess.

Read the name and it tells you everything. The number after the Q is the bits per weight: Q8 keeps 8 bits, Q4 keeps 4, Q3 keeps 3. Fewer bits means a coarser grid to round each weight onto — so a smaller file, less memory, faster math, and a little more rounding error baked into every number. That's the whole tradeoff in one digit: lower number = smaller and faster, but less accurate. The trailing letters (_K_M, _K_S, _0) are refinements — "K" marks the modern k-quant schemes that spend their bits more cleverly, and the S/M/L suffix is small/medium/large within that level. When in doubt, the _K_M ("medium") variants are the sweet spot the community has settled on.

The mental model that keeps you out of trouble is graceful degradation. Dropping from 16-bit down to 8-bit is nearly free — the quality loss is so small most people can't spot it in normal use. From 8 to 4 bits you lose a little more, but a good 4-bit model is still genuinely useful and this is where most laptops happily live. Below 4 bits the curve starts to bend: 3-bit is noticeably rougher, and 2-bit models often produce visibly worse, sometimes broken, output. Picture a photo you keep re-saving as a lower-quality JPEG — the first few saves look fine, then suddenly you see the blocky artifacts. Quantization has the same shape: quality holds up well, then falls off a cliff once you push too far. The art is stopping before the cliff.

So how do you actually choose? Work backwards from the memory you have, not the quality you want. Your ceiling is your RAM (or VRAM), minus a couple of gigabytes you must leave for the operating system, the KV-cache that grows with your context length, and general overhead. Whatever budget remains, pick the highest-quality quant that still fits under it — never the biggest file you can find, and never reflexively the smallest. On a typical 16 GB laptop a 7-8B model at Q4_K_M (~4 GB) leaves comfortable headroom; on an 8 GB machine you're choosing between a tight Q4 and a safer Q3; with 32 GB you can afford Q8_0 and near-original quality. The table below gives concrete sizes for a 7B model so you can do that arithmetic at a glance — and the practice ladder later turns exactly this "fit the best quant into a budget" logic into runnable code.

GGUF ships a model at many quant levels; the name encodes the tradeoff (e.g. Q4_K_M). Lower = smaller/faster/less accurate.

gguf.sh# Option A: via Ollama (uses GGUF under the hood) — easiest
ollama run llama3.1:8b-instruct-q4_K_M

# Option B: llama.cpp directly, with its OpenAI-compatible server:
pip install "llama-cpp-python[server]"
python -m llama_cpp.server \
  --model ./llama-3.1-8b-instruct.Q4_K_M.gguf \
  --n_gpu_layers 0        # 0 = pure CPU;  >0 offloads layers to a GPU if present

client.pyfrom openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="local-gguf",
    messages=[{"role": "user", "content": "Run entirely on my CPU, please."}])
print(r.choices[0].message.content)

# Option A -- via Ollama (uses GGUF under the hood), easiest:
ollama run llama3.1:8b-instruct-q4_K_M

# Option B -- llama.cpp directly, OpenAI-compatible server:
pip install "llama-cpp-python[server]"
python -m llama_cpp.server \
  --model ./llama-3.1-8b-instruct.Q4_K_M.gguf \
  --n_gpu_layers 0            # 0 = pure CPU; >0 offloads layers to a GPU if present

QUANT_7B = [                       # (name, approx GB, quality rank high->low)
    ("Q8_0",   7.5, 4),
    ("Q5_K_M", 5.0, 3),
    ("Q4_K_M", 4.0, 2),            # the common laptop default
    ("Q3_K_M", 3.3, 1),
]
def pick_quant(ram_gb, reserve=1.5):
    budget = ram_gb - reserve                  # leave RAM for the OS/app
    fitting = [q for q in QUANT_7B if q[1] <= budget]
    if not fitting:
        return "none fit -- use a smaller model"
    return max(fitting, key=lambda q: q[2])[0] # highest quality that fits

print(pick_quant(8))    # Q4_K_M (best that fits ~6.5GB budget)
print(pick_quant(16))   # Q8_0
print(pick_quant(4))    # Q3_K_M

def est_tps(total_layers, gpu_layers, cpu_tps=6.0, gpu_tps=60.0):
    gpu_layers = max(0, min(gpu_layers, total_layers))
    frac_gpu = gpu_layers / total_layers
    # harmonic-ish blend: the slow (CPU) part dominates
    if not frac_gpu:
        return round(cpu_tps, 1)
    cpu_frac = 1 - frac_gpu
    blended = 1 / (cpu_frac / cpu_tps + frac_gpu / gpu_tps)
    return round(blended, 1)

for gl in [0, 16, 32]:
    print(f"n_gpu_layers={gl:2} -> ~{est_tps(32, gl)} tok/s")
# more offloaded layers -> higher throughput; 0 = pure CPU baseline

def pick_stack(has_gpu, concurrent_users, edge_or_laptop):
    if edge_or_laptop or not has_gpu:
        return "llama.cpp + GGUF -- runs on CPU/Apple Silicon, no GPU needed"
    if concurrent_users > 4:
        return "vLLM -- GPU throughput serving (LM3)"
    return "either works; GGUF is simpler for light single-node use"

print(pick_stack(has_gpu=False, concurrent_users=1, edge_or_laptop=True))
print(pick_stack(has_gpu=True,  concurrent_users=50, edge_or_laptop=False))

QUANTS = [                       # name, GB(7B), quality(0-100)
    ("Q8_0",   7.5, 99),
    ("Q5_K_M", 5.0, 96),
    ("Q4_K_M", 4.0, 93),
    ("Q3_K_M", 3.3, 86),
]
def report(ram_gb):
    print(f"{'quant':8}{'GB':>6}{'quality':>9}{'fits?':>7}")
    for name, gb, q in QUANTS:
        fits = "yes" if gb <= ram_gb - 1.5 else "no"
        speed = round(7.5 / gb, 2)          # smaller file ~= faster, rough proxy
        print(f"{name:8}{gb:6.1f}{q:9}{fits:>7}   ~{speed}x")

report(ram_gb=8)

QUANTS = [("Q8_0", 7.5, 99), ("Q5_K_M", 5.0, 96),
          ("Q4_K_M", 4.0, 93), ("Q3_K_M", 3.3, 86)]

def fleet_quant(device_ram_gb, reserve=1.5):
    budget = min(device_ram_gb) - reserve       # constrained by the smallest
    fitting = [q for q in QUANTS if q[1] <= budget]
    if not fitting:
        return None, budget
    best = max(fitting, key=lambda q: q[2])      # highest quality that fits all
    return best, budget

fleet = [8, 16, 32]
best, budget = fleet_quant(fleet)
print(f"smallest device budget ~{budget}GB -> ship {best[0]} (quality {best[2]})")
# every device runs the same file; the 8GB box is the limiter

Knowledge check check yourself