Inference & Cost · Chapter IC2

Quantization: GPTQ, AWQ, GGUF

Store weights in fewer bits — FP16→INT8→INT4 — and a model gets smaller and, because decode is memory-bound, faster. This chapter covers the formats and the accuracy tradeoff.

⏱️ ~2 hours🧪 1 lab🎯 Advanced

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

a GPU (QLoRA fits a 7B model on one consumer card) + pip install transformers peft trl bitsandbytes

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Explain how quantization shrinks a model and speeds memory-bound decode.
Distinguish the formats: GPTQ, AWQ, GGUF, and bitsandbytes.
Reason about the accuracy-vs-size tradeoff and pick a bit-width.
Load and run a quantized model.

▶ Runnable companionThe code in this lesson is also saved under code/ic2-quantization/ in the course, with a README. Run the scripts or copy the configs directly.

Why quantization works essential

A model's weights are numbers. Store them in fewer bits — FP16 → INT8 → INT4 — and the model gets smaller and, because decode is memory-bound (IC1), faster to serve: less data to move per token. A 4-bit model is ~4× smaller than FP16 and often fits on hardware the full model can't.

🗺️ How to read this diagram

This shows the recipe for quantizing a model: take its numbers, learn how big they get, then re-store them using far fewer bits. "Bits" are just how much space each number takes.

FP16 weights (16 bit) — the model as trained. Every weight is a high-precision number that eats 16 bits of memory. Accurate, but big and slow to move.
Calibrate (measure ranges) — the tool runs a little sample data through the model to see the typical range of each group of weights (how small and how large they get). Knowing the range lets it map them onto a small set of integers without wasting precision.
INT4/INT8 weights (4-8 bit) — the weights re-expressed as small integers. INT8 uses half the space of FP16; INT4 uses a quarter. Less data per token = faster memory-bound decode (IC1).
Read the arrows as the steps of a one-time conversion you do before serving; afterwards you just load the small version.

In short: Fewer bits = smaller and faster, at a small accuracy cost. The middle "calibrate" step is what keeps the accuracy loss small — it aims the limited integer values where the weights actually are.

Precision	Size vs FP16	Typical use
FP16/BF16	1×	training, high-accuracy serving
INT8	~0.5×	safe speedup, minimal quality loss
INT4	~0.25×	fit big models on small GPUs; small quality cost

The formats essential

You'll meet four names; they differ in how they quantize and where they run:

Format	Approach	Best for
GPTQ	post-training, calibration-based	GPU inference (vLLM/TGI)
AWQ	activation-aware weighting	GPU, strong accuracy at 4-bit
GGUF	llama.cpp format, CPU+GPU	local/laptop, Apple Silicon
bitsandbytes	on-the-fly 8/4-bit	quick tuning + QLoRA (see FT3)

This connects to two other tracksGGUF is what you'll run locally in LM4; bitsandbytes 4-bit is the foundation of QLoRA in FT3. Quantization is the thread that ties inference, local models, and fine-tuning together.

Run a quantized model intermediate

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab IC2.1 · Load a 4-bit model

quantize.pyimport torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # normal-float 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,  # compute in bf16, store in 4-bit
)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    quantization_config=quant, device_map="auto",
)
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

ids = tok("Explain quantization in one sentence.", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))

▶ How this works

This loads a real 7-billion-parameter model in 4-bit form so it fits on a modest GPU, then asks it a question. The bitsandbytes library does the shrinking for you as the model loads.

BitsAndBytesConfig(load_in_4bit=True, ...) is the knob-box: it says "store the weights in 4 bits." bnb_4bit_quant_type="nf4" picks the nf4 number format (a 4-bit layout tuned for neural-net weights).
bnb_4bit_compute_dtype=torch.bfloat16 is a key trick: weights are stored in 4 bits to save memory, but the actual math is done in wider 16-bit numbers for accuracy.
from_pretrained("mistralai/Mistral-7B-Instruct-v0.3", quantization_config=quant, device_map="auto") downloads the model and applies that config; device_map="auto" lets it place the model on your GPU automatically.
The last three lines are ordinary use: tok(...) turns text into token IDs, model.generate(..., max_new_tokens=60) writes up to 60 new tokens, and tok.decode(...) turns the tokens back into readable text to print.

What the output means: A one-sentence explanation of quantization, produced by a model small enough to run on a single consumer GPU — something the full 16-bit version often can't do.

Try this: Change load_in_4bit=True to load_in_8bit=True (and drop the 4bit options). It uses more memory but is usually a touch more accurate — that's the size-vs-quality tradeoff in one line.

Quantization is lossy — measure itLower bits = smaller/faster, but accuracy degrades, and how much is task-dependent. Never ship a quantized model without running your evals (Ch 5) against the full-precision baseline. INT8 is usually safe; INT4 needs a quality check.

Exercise IC2.1 — Quantify the tradeoff

Context: The bit-width you ship is a per-task tradeoff you can only settle by measuring size, speed, and quality on your own eval — the size/speed/quality table makes the choice concrete.

Your task: Load a model at FP16, INT8, and INT4; for each, measure model size, TPOT, and accuracy on a small eval set of your own, then build the table and pick the bit-width you'd actually ship.

Requirements:

Load the model at all three precisions
Measure size, TPOT (from IC1), and accuracy on your own eval
Assemble a size/speed/quality table
Pick the bit-width you'd ship for your task
Base the pick on measured numbers, not defaults

💡 Hint: The Lab IC2.1 load path (BitsAndBytesConfig nf4 / bf16-compute) is the loader; extend it across the three precisions and reuse your IC1 TPOT probe.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Model size at each precisionBeginner

Context: Quantization stores weights in fewer bits — INT8 is ~half FP16 and INT4 ~a quarter — shrinking the model and speeding memory-bound decode.

Your task: For a 7B-parameter model, compute the weight memory (GB) at FP16, INT8, and INT4.

Requirements:

Size = params × bytes-per-param
Use 2 / 1 / 0.5 bytes for FP16 / INT8 / INT4
Print all three GB figures
Show INT4 moves a 16 GB-card model onto an 8 GB card

💡 Hint: It's one multiply per precision — the only thing that changes is bytes-per-param.

Show solution

Size is just params x bytes-per-param:

params = 7_000_000_000
bytes_per = {"fp16": 2, "int8": 1, "int4": 0.5}
for name, b in bytes_per.items():
    gb = params * b / 1024**3
    print(f"{name}: {gb:.1f} GB")
# fp16: 13.0 GB   int8: 6.5 GB   int4: 3.3 GB

INT4 turns a model that needs a 16 GB card into one that fits comfortably on 8 GB — the whole point of quantization for local/single-GPU serving.

Exercise 2 · Does it fit on this GPU?Intermediate

Context: A model's weights aren't the only thing on the card — the KV-cache and activations need headroom too, so a fit check must add overhead.

Your task: Write a checker that, given a GPU's VRAM and a ~20% overhead for KV-cache and activations, reports which precisions of a model fit; test a 13B model on a 24 GB card.

Requirements:

Multiply each precision's weight size by a ~1.20 headroom factor
Return per-precision required GB and a fits boolean
Test a 13B model on 24 GB
Show FP16 overflows once overhead is added while INT8 fits

💡 Hint: Compute the weight size per precision, pad it by the overhead factor, then compare against the card — the headroom is what tips FP16 over.

Show solution

Combine the size math with a headroom factor:

def fits(params_b, vram_gb, overhead=1.20):
    out = {}
    for name, bpp in {"fp16":2, "int8":1, "int4":0.5}.items():
        need = params_b * bpp / 1024**3 * overhead
        out[name] = (round(need,1), need <= vram_gb)
    return out

for name,(need,ok) in fits(13_000_000_000, 24).items():
    print(f"{name}: needs {need} GB -> {'FITS' if ok else 'no'}")
# fp16: needs 29.1 GB -> no
# int8: needs 14.5 GB -> FITS
# int4: needs 7.3 GB  -> FITS

FP16 overflows a 24 GB card once you add KV-cache overhead; INT8 is the safe pick here with minimal quality loss.

Exercise 3 · Quantization error vs bit-widthAdvanced

Context: Fewer bits cost accuracy: quantizing to a coarse integer grid introduces error that grows fast as bit-width falls, which is why INT4 is usually the floor.

Your task: Quantize a small array of weights to N-bit integers over their min…max range, measure mean absolute error, and show error growing from 8 to 4 to 2 bits.

Requirements:

Compute levels = 2^bits − 1 and a scale over the value range
Round each weight to the nearest grid point and back
Report mean absolute error vs the originals
Show MAE rising as bits fall (error climbs fast below 4-bit)
Motivate INT4 as the usual floor, plus calibration

💡 Hint: Halving the bits roughly quadruples the grid spacing — the round-to-grid step is where the error enters.

Show solution

A faithful toy of round-to-grid quantization (stdlib only):

def quantize(weights, bits):
    lo, hi = min(weights), max(weights)
    levels = (1 << bits) - 1                 # e.g. 4 bits -> 15 levels
    scale = (hi - lo) / levels or 1.0
    q = [round((w - lo) / scale) * scale + lo for w in weights]
    mae = sum(abs(a-b) for a,b in zip(weights,q)) / len(weights)
    return round(mae, 4)

w = [0.11, -0.4, 0.9, -0.05, 0.33, -0.72, 0.5, 0.02]
for bits in (8, 4, 2):
    print(f"{bits}-bit MAE: {quantize(w, bits)}")
# 8-bit MAE: ~0.0016   4-bit: ~0.017   2-bit: ~0.12

Halving the bits roughly quadruples the grid spacing, so error climbs fast below 4-bit — which is why INT4 is the usual floor and calibration (aiming the levels where weights actually cluster) is what keeps 4-bit usable.

Exercise 4 · Pick the format from the deploymentExpert

Context: The four quantization formats target different hardware — GPTQ/AWQ for GPU inference, GGUF for CPU/Apple Silicon/local, bitsandbytes for on-the-fly QLoRA.

Your task: Write a selector that maps a deployment description to the right quantization format.

Requirements:

Return bitsandbytes when doing QLoRA
Return GGUF for laptop/CPU/Apple-Silicon targets
For a GPU server, return AWQ when 4-bit accuracy matters, else GPTQ
Default to GGUF as the safe fallback
Match the format to where it runs

💡 Hint: Branch on the target hardware first, then refine the GPU case by whether 4-bit accuracy is the priority (AWQ) or calibration throughput (GPTQ).

Show solution

Decision logic straight from the lesson's format table:

def pick_format(target, need_accuracy_4bit=False, doing_qlora=False):
    if doing_qlora:
        return "bitsandbytes — on-the-fly 4-bit for QLoRA (FT3)"
    if target in ("laptop", "cpu", "apple-silicon"):
        return "GGUF — llama.cpp, runs CPU+GPU locally"
    if target == "gpu-server":
        return "AWQ — activation-aware, strong 4-bit accuracy" if need_accuracy_4bit \
               else "GPTQ — calibration-based, great on vLLM/TGI"
    return "GGUF — safe local default"

print(pick_format("laptop"))
print(pick_format("gpu-server", need_accuracy_4bit=True))
print(pick_format("gpu-server", doing_qlora=True))

There is no universal best format — GGUF wins on laptops, AWQ/GPTQ on GPU servers, bitsandbytes when you are training. Match the format to where it runs.

Exercise 5 · Quantize + load a model (real, needs GPU/libs)Professional

Context: Loading a quantized model changes almost nothing in your code — the only quantization-specific part is which checkpoint you load; the API is unchanged.

Your task: Show the real AWQ load path with transformers so a teammate can run it, using no invented APIs and labelling it as needing a GPU + libraries.

Requirements:

Load a pre-quantized AWQ repo with from_pretrained(…, device_map="auto")
Build the prompt via apply_chat_template
Generate and decode as usual
Show the API is identical to any model — only the checkpoint differs
Note the libs: transformers autoawq accelerate

💡 Hint: There's no special quantized-inference call — pick the AWQ checkpoint and the normal generate path just works on the 4-bit weights.

Show solution

Correct transformers shape — needs GPU + pip install transformers autoawq accelerate:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"   # a pre-quantized AWQ repo
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",          # place on available GPU(s)
)

msgs = [{"role": "user", "content": "Explain quantization in one line."}]
inputs = tok.apply_chat_template(msgs, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

Nothing here is quantization-specific except the repo you load — a pre-quantized AWQ checkpoint. The 4-bit weights make it fit and speed memory-bound decode; the API is unchanged.

Exercise 6 · Approve or reject a quantized model for productionIndustry scenario

Context: The production decision is a gate, not a vibe: pick the smallest precision that stays within an accuracy-drop tolerance AND fits the VRAM budget.

Your task: Given task-eval accuracy for FP16 vs INT8 vs INT4 plus a VRAM budget, write the gate that picks the smallest precision within tolerance and fitting VRAM, and print the decision + reason.

Requirements:

Take FP16 accuracy as the baseline
Set an accuracy-drop tolerance and a VRAM budget
Keep only candidates within tolerance AND fitting VRAM
Pick the smallest-VRAM survivor
Gate on a task eval, not on vibes

💡 Hint: Filter on both the accuracy-drop and the VRAM fit, then take the smallest survivor — the aggressive precision usually fails the accuracy check.

Show solution

The real ship/no-ship gate — smallest that is both accurate enough and fits:

candidates = [
    # name, accuracy, vram_gb
    ("fp16", 0.812, 13.0),
    ("int8", 0.809, 6.5),
    ("int4", 0.771, 3.3),
]
baseline = candidates[0][1]     # fp16 accuracy
TOL = 0.01                      # allow <=1 point absolute drop
VRAM = 8.0

ok = [c for c in candidates if baseline - c[1] <= TOL and c[2] <= VRAM]
pick = min(ok, key=lambda c: c[2]) if ok else None   # smallest that qualifies
print("qualify:", [c[0] for c in ok])   # ['int8']  (int4 drops 4.1 pts, fp16 too big)
print("PICK:", pick[0] if pick else "none -- relax VRAM or tolerance")

INT4 saves the most memory but drops 4 accuracy points — outside tolerance. INT8 fits the 8 GB card and is statistically even with FP16, so it ships. Always gate quantization on a task eval, not vibes.

✓ Checkpoint — you can move on when you can…

Explain why fewer bits speed up memory-bound decode.
Match GPTQ/AWQ/GGUF/bitsandbytes to their best use.
Load and run a 4-bit model.
Judge a quantized model against a full-precision eval baseline.

Knowledge check check yourself

✓ Knowledge check

Why does storing weights in fewer bits (FP16 to INT4) speed up serving, given that decode is memory-bound?

Show answer

Decode's bottleneck is moving weights out of memory per token, not the arithmetic. A 4-bit model is ~4x smaller than FP16, so there's far less data to move each token, making memory-bound decode faster — and it fits on hardware the full model can't.

✓ Knowledge check

Match each format to its best use: GPTQ, AWQ, GGUF, bitsandbytes.

Show answer

GPTQ: post-training calibration-based, for GPU inference (vLLM/TGI). AWQ: activation-aware weighting, strong accuracy at 4-bit on GPU. GGUF: llama.cpp format for CPU+GPU local/laptop and Apple Silicon (LM4). bitsandbytes: on-the-fly 8/4-bit, used for quick tuning and QLoRA (FT3).

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Why quantization works essential

The formats essential

Run a quantized model intermediate

Exercise IC2.1 — Quantify the tradeoff

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Inference & cost