Local & Open Models · Chapter LM4

Quantized inference on your hardware

No GPU? llama.cpp + GGUF run quantized models efficiently on CPU and Apple Silicon. Pick a quant level for your memory budget and run real models anywhere.

⏱️ ~1.5 hours🧪 2 labs🎯 Intermediate→Advanced

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

pip install "llama-cpp-python[server]" — runs on CPU / Apple Silicon, no GPU needed
Ollama installed (runs models locally, laptop-friendly)

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Run quantized models on CPU / Apple Silicon with llama.cpp + GGUF.
Pick a GGUF quant level for your memory budget and quality bar.
Understand where GGUF/llama.cpp fits vs GPU serving.
Run a model with no GPU at all.

▶ Runnable companionThe code in this lesson is also saved under code/lm4-quantized-local/ in the course, with a README. Run the scripts or copy the configs directly.

You don't always have a GPU intermediate

GPU serving (LM3) is ideal but not always available — laptops, edge devices, CPU-only servers. llama.cpp and the GGUF format (from IC2) run quantized models efficiently on CPU and Apple Silicon, so you can run real models with no GPU.

GGUF quant levels advanced

GGUF ships a model at many quant levels; the name encodes the tradeoff (e.g. Q4_K_M). Lower = smaller/faster/less accurate.

Quant	Size (7B)	Use
Q8_0	~7.5 GB	near-FP16 quality, if you have the RAM
Q5_K_M	~5 GB	strong quality/size balance
Q4_K_M	~4 GB	the common default — good on a laptop
Q3_K_M	~3.3 GB	tight memory; visible quality cost

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab LM4.1 · Run GGUF on CPU / Apple Silicon

gguf.sh# Option A: via Ollama (uses GGUF under the hood) — easiest
ollama run llama3.1:8b-instruct-q4_K_M

# Option B: llama.cpp directly, with its OpenAI-compatible server:
pip install "llama-cpp-python[server]"
python -m llama_cpp.server \
  --model ./llama-3.1-8b-instruct.Q4_K_M.gguf \
  --n_gpu_layers 0        # 0 = pure CPU;  >0 offloads layers to a GPU if present

▶ How this works

This lab is for machines with no GPU — a plain laptop or an Apple-Silicon Mac. It shows two ways to run a quantized model on the CPU. GGUF is just a compact model file format designed to run well without a GPU.

Option A — ollama run llama3.1:8b-instruct-q4_K_M is the easy path: Ollama already uses GGUF internally, and the q4_K_M tag picks a specific 4-bit quant level (small, laptop-friendly).
Option B — pip install "llama-cpp-python[server]" installs llama.cpp, the low-level engine that runs GGUF files, with its own OpenAI-compatible server built in.
python -m llama_cpp.server --model ./llama-3.1-8b-instruct.Q4_K_M.gguf starts that server, pointing at a GGUF file you downloaded.
--n_gpu_layers 0 is the key flag: 0 means "run entirely on the CPU". If you do have a GPU, a number above 0 offloads that many layers onto it for extra speed.

What the output means: A local OpenAI-compatible server running purely on your CPU — slower than a GPU, but it works on hardware that has no GPU at all.

Try this: Try a heavier quant like Q5_K_M for better quality if you have the RAM, or a lighter Q3_K_M if memory is tight — the quant-level table above shows the trade-offs.

Lab LM4.2 · Same client again

client.pyfrom openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="local-gguf",
    messages=[{"role": "user", "content": "Run entirely on my CPU, please."}])
print(r.choices[0].message.content)

▶ How this works

Same connect-with-one-line pattern again — this time against the CPU-only llama.cpp server. The recurring theme of the whole track: whatever runs the model, your client code barely changes.

OpenAI(base_url="http://localhost:8000/v1", api_key="none") aims the standard client at the local llama.cpp server on port 8000.
client.chat.completions.create(model="local-gguf", messages=[...]) sends the chat request; local-gguf is just the local model's name.
r.choices[0].message.content reads and prints the reply — no GPU, no cloud, no API key involved.

What the output means: The model answers from your CPU. Expect it to be noticeably slower than a GPU or a hosted API, but it runs anywhere.

Try this: Time how long the reply takes here versus a GPU or hosted run. That speed gap is the real cost of going GPU-free — and often an acceptable one for light, offline use.

When to use whichGGUF/llama.cpp: CPU, laptops, edge, single-user. vLLM (LM3): GPU, high-throughput, many users. Same open model, different runtime for different hardware and load.

Exercise LM4.1 — No-GPU inference

Context: The headline of the whole chapter: a quantized model runs on CPU at all — the speed cost is the tradeoff you're measuring.

Your task: Run a Q4_K_M GGUF model on CPU (via Ollama or llama.cpp), measure TPOT, and compare to a GPU or hosted run.

Requirements:

Run a Q4_K_M GGUF model with no GPU
Measure time-per-output-token (TPOT)
Compare against a GPU or hosted run
Note the speed cost — and that it works at all without a GPU

💡 Hint: Expect CPU TPOT to be markedly slower; the point is that it runs at all, and partial GPU offload would close much of the gap.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Run GGUF on CPU / Apple Silicon (config)Beginner

Context: No GPU? llama.cpp + GGUF run quantized models on CPU or Apple Silicon — GGUF is a compact format built to run well without a GPU.

Your task: Write the two ways from the lesson to run GGUF — via Ollama and via llama.cpp's OpenAI-compatible server. (Needs the runtime.)

Requirements:

Show the Ollama path (a quantized model tag)
Show the llama.cpp path (its OpenAI-compatible server) with the model file
Note --n_gpu_layers 0 for pure CPU (and that >0 offloads to a GPU if present)
Explain GGUF is built to run without a GPU
Label both as needing the runtime installed

💡 Hint: Ollama uses GGUF under the hood (the easy path); llama.cpp's server is the direct path — a plain laptop or Apple-Silicon Mac is enough.

Show solution

Two CPU paths (needs the runtime installed, no GPU required):

# Option A -- via Ollama (uses GGUF under the hood), easiest:
ollama run llama3.1:8b-instruct-q4_K_M

# Option B -- llama.cpp directly, OpenAI-compatible server:
pip install "llama-cpp-python[server]"
python -m llama_cpp.server \
  --model ./llama-3.1-8b-instruct.Q4_K_M.gguf \
  --n_gpu_layers 0            # 0 = pure CPU; >0 offloads layers to a GPU if present

GGUF is a compact file format built to run well without a GPU — a plain laptop or Apple-Silicon Mac is enough.

Exercise 2 · Pick a GGUF quant level for a memory budgetIntermediate

Context: GGUF quant names (Q8_0 down to Q3_K_M) encode the size/speed/accuracy tradeoff: lower means smaller and faster but less accurate.

Your task: Write a selector that picks the highest-quality quant that fits a RAM budget for a 7B model.

Requirements:

Use a table of quant levels with approximate sizes and a quality rank
Reserve some RAM for the OS/app before fitting
Return the highest-quality quant that fits the budget
Handle the case where none fits
Show picks at a few RAM sizes

💡 Hint: Filter to quants that fit the budget, then take the highest quality; Q4_K_M is the common laptop default, Q8_0 if RAM allows, Q3_K_M when tight.

Show solution

Use the lesson's size table and pick the best that fits. Runnable:

QUANT_7B = [                       # (name, approx GB, quality rank high->low)
    ("Q8_0",   7.5, 4),
    ("Q5_K_M", 5.0, 3),
    ("Q4_K_M", 4.0, 2),            # the common laptop default
    ("Q3_K_M", 3.3, 1),
]
def pick_quant(ram_gb, reserve=1.5):
    budget = ram_gb - reserve                  # leave RAM for the OS/app
    fitting = [q for q in QUANT_7B if q[1] <= budget]
    if not fitting:
        return "none fit -- use a smaller model"
    return max(fitting, key=lambda q: q[2])[0] # highest quality that fits

print(pick_quant(8))    # Q4_K_M (best that fits ~6.5GB budget)
print(pick_quant(16))   # Q8_0
print(pick_quant(4))    # Q3_K_M

Pick the highest-quality quant your RAM allows: Q4_K_M is the common default, Q8_0 if you have the memory, Q3_K_M when it's tight.

Exercise 3 · Estimate tokens/sec on CPU vs partial GPU offloadAdvanced

Context: llama.cpp can offload some layers to a GPU via n_gpu_layers; because the CPU portion is the bottleneck, offloading even part of the model helps a lot.

Your task: Model a rough throughput estimate: a CPU-only baseline that improves as more layers are offloaded to a GPU.

Requirements:

Take total layers and the number offloaded to the GPU
CPU-only is the slow baseline; full offload approaches the GPU rate
Blend so the slow (CPU) fraction dominates until most layers are offloaded
Clamp the offloaded count to the layer total
Show throughput at 0, partial, and full offload

💡 Hint: A blend where the CPU fraction dominates (harmonic-style) captures why partial offload gives an outsized speedup; set --n_gpu_layers as high as VRAM allows.

Show solution

Model the offload speedup (illustrative, runnable):

def est_tps(total_layers, gpu_layers, cpu_tps=6.0, gpu_tps=60.0):
    gpu_layers = max(0, min(gpu_layers, total_layers))
    frac_gpu = gpu_layers / total_layers
    # harmonic-ish blend: the slow (CPU) part dominates
    if not frac_gpu:
        return round(cpu_tps, 1)
    cpu_frac = 1 - frac_gpu
    blended = 1 / (cpu_frac / cpu_tps + frac_gpu / gpu_tps)
    return round(blended, 1)

for gl in [0, 16, 32]:
    print(f"n_gpu_layers={gl:2} -> ~{est_tps(32, gl)} tok/s")
# more offloaded layers -> higher throughput; 0 = pure CPU baseline

Offloading even part of the model to a GPU helps a lot because the CPU portion is the bottleneck — set --n_gpu_layers as high as VRAM allows.

Exercise 4 · Where does GGUF/llama.cpp fit vs GPU serving?Expert

Context: GGUF/llama.cpp is for no-GPU / edge / laptop; vLLM (LM3) is for GPU throughput. Routing a deployment to the right stack is the decision.

Your task: Write a decision helper that routes a deployment to llama.cpp+GGUF or vLLM.

Requirements:

Route to llama.cpp+GGUF when there's no GPU or it's edge/laptop
Route to vLLM when there's a GPU and real concurrency
Take has-GPU, concurrency, and edge/laptop as parameters
Show a no-GPU case and a high-concurrency GPU case

💡 Hint: GGUF fills the no-GPU gap; once you have GPUs and concurrency, vLLM's batching wins.

Show solution

Match the stack to the hardware and load. Runnable:

def pick_stack(has_gpu, concurrent_users, edge_or_laptop):
    if edge_or_laptop or not has_gpu:
        return "llama.cpp + GGUF -- runs on CPU/Apple Silicon, no GPU needed"
    if concurrent_users > 4:
        return "vLLM -- GPU throughput serving (LM3)"
    return "either works; GGUF is simpler for light single-node use"

print(pick_stack(has_gpu=False, concurrent_users=1, edge_or_laptop=True))
print(pick_stack(has_gpu=True,  concurrent_users=50, edge_or_laptop=False))

GGUF fills the gap where there's no GPU; once you have GPUs and real concurrency, vLLM's batching wins.

Exercise 5 · Quantization quality-vs-size tradeoff curveProfessional

Context: Lower quant is smaller and faster but lower quality — and making that curve explicit is how you defend a choice instead of guessing.

Your task: Build a table over quant levels showing size, a relative-speed proxy, and a rough quality score to justify a choice.

Requirements:

List quant levels with size (GB) and a quality score
Add a relative-speed proxy (smaller file ≈ faster)
Mark which levels fit a given RAM budget
Print an aligned table
Identify the knee of the curve

💡 Hint: The knee is usually Q4_K_M — near-full quality at roughly half the size; Q3 only when memory forces it, Q8 when you can afford it.

Show solution

Make the tradeoff explicit so a choice is defensible. Runnable:

QUANTS = [                       # name, GB(7B), quality(0-100)
    ("Q8_0",   7.5, 99),
    ("Q5_K_M", 5.0, 96),
    ("Q4_K_M", 4.0, 93),
    ("Q3_K_M", 3.3, 86),
]
def report(ram_gb):
    print(f"{'quant':8}{'GB':>6}{'quality':>9}{'fits?':>7}")
    for name, gb, q in QUANTS:
        fits = "yes" if gb <= ram_gb - 1.5 else "no"
        speed = round(7.5 / gb, 2)          # smaller file ~= faster, rough proxy
        print(f"{name:8}{gb:6.1f}{q:9}{fits:>7}   ~{speed}x")

report(ram_gb=8)

The knee is usually Q4_K_M: near-full quality at roughly half the size. Q3 only when memory forces it; Q8 when you can afford it.

Exercise 6 · Fit a fleet of edge devices to one quantIndustry scenario

Context: For a uniform edge fleet with varying RAM, the weakest device sets the quant — otherwise you fragment into per-device builds.

Your task: Given devices with varying RAM (e.g. 8/16/32 GB), pick the single GGUF quant that runs on ALL of them and report coverage.

Requirements:

The smallest-RAM device constrains the choice
Compute the budget from the minimum device RAM (with a reserve)
Pick the highest-quality quant that fits that budget
Report which quant ships and its quality
Note that one portable file avoids per-device builds

💡 Hint: Take the min over device RAM, then pick the best quant that fits it; GGUF's one portable file is what makes a uniform rollout possible.

Show solution

The weakest device sets the quant for a uniform fleet. Runnable:

QUANTS = [("Q8_0", 7.5, 99), ("Q5_K_M", 5.0, 96),
          ("Q4_K_M", 4.0, 93), ("Q3_K_M", 3.3, 86)]

def fleet_quant(device_ram_gb, reserve=1.5):
    budget = min(device_ram_gb) - reserve       # constrained by the smallest
    fitting = [q for q in QUANTS if q[1] <= budget]
    if not fitting:
        return None, budget
    best = max(fitting, key=lambda q: q[2])      # highest quality that fits all
    return best, budget

fleet = [8, 16, 32]
best, budget = fleet_quant(fleet)
print(f"smallest device budget ~{budget}GB -> ship {best[0]} (quality {best[2]})")
# every device runs the same file; the 8GB box is the limiter

For a uniform rollout you quantize to what the smallest device can hold — otherwise you fragment into per-device builds. GGUF makes one portable file possible (needs the runtime on each device).

✓ Checkpoint — you can move on when you can…

Run a quantized model on CPU / Apple Silicon with GGUF.
Pick a GGUF quant level for a memory budget.
Explain where llama.cpp fits vs vLLM.
Run a real model with no GPU.

Knowledge check check yourself

✓ Knowledge check

What do llama.cpp and the GGUF format let you do that GPU serving (vLLM) cannot, and what flag runs a model purely on CPU?

Show answer

They run quantized models efficiently on CPU and Apple Silicon, so you can run real models on hardware with no GPU at all (laptops, edge, CPU-only servers). In the llama.cpp server, --n_gpu_layers 0 means pure CPU; a value above 0 offloads that many layers to a GPU if present.

✓ Knowledge check

What does a GGUF quant tag like Q4_K_M encode, and how does it compare to Q8_0 and Q3_K_M for a 7B model?

Show answer

The tag encodes the size/quality tradeoff — lower numbers mean smaller, faster, and less accurate. Q4_K_M (~4 GB) is the common laptop default; Q8_0 (~7.5 GB) is near-FP16 quality if you have the RAM; Q3_K_M (~3.3 GB) fits tight memory with a visible quality cost.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

You don't always have a GPU intermediate

GGUF quant levels advanced

Exercise LM4.1 — No-GPU inference

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Local & open models