Quantization: GPTQ, AWQ, GGUF

Why quantization works essential

Start with the wall you'll hit. You find the perfect open model, you write the loader, you hit run — and the process dies with CUDA out of memory. A 13-billion-parameter model in its native FP16 wants ~26 GB just for weights, and your card has 24. The model is right there, you just can't fit it. Quantization is the lever that gets it onto the hardware you actually have — and, as a bonus, makes it faster too. This section is about why that bonus is real and not a free-lunch fantasy.

Here's the analogy that makes it click. Think of each weight as a colour you're storing. A 16-bit weight is a photographer's RAW file — millions of shades, gorgeous, huge. An 8-bit weight is a good JPEG — you genuinely can't see the difference, at half the size. A 4-bit weight is a GIF with 16 colours — noticeably coarser if you look closely, but a quarter the size and still perfectly recognizable. Quantization is choosing how many "shades" each number gets. Fewer shades, less space — and the whole art is spending your limited shades where the picture actually needs them.

Now the part people get backwards. The obvious guess is "fewer bits = faster because the math is simpler." That's not the main reason. As you saw in IC1, generating each token is memory-bound: the GPU spends its time moving the weights out of memory, not multiplying them. The arithmetic units sit half-idle waiting for data. So the thing that actually speeds decode up is having less data to move per token — and a 4-bit model is roughly a quarter the bytes of FP16, so there's a quarter as much to haul across the memory bus. Smaller weights ride the bottleneck; that's the whole trick.

The common mistake is treating quantization as pure upside and reaching straight for the smallest bit-width "to be safe on memory." It isn't free — squeezing a continuous number onto a coarse integer grid throws information away, and that shows up as small errors in every weight. INT8 is almost always invisible; INT4 is usually fine but task-dependent; below 4-bit the errors compound fast (you'll measure exactly this in Exercise 3). The right instinct is "use the fewest bits that still passes my eval," not "use the fewest bits, full stop." Keep that tension in mind as you read on: everything below is about buying memory and speed without spending more accuracy than you can afford.

A model's weights are numbers. Store them in fewer bits — FP16 → INT8 → INT4 — and the model gets smaller and, because decode is memory-bound (IC1), faster to serve: less data to move per token. A 4-bit model is ~4× smaller than FP16 and often fits on hardware the full model can't.

The formats essential

Here's where quantization stops being one idea and starts being an alphabet soup. You go looking for a quantized model and the download page throws GPTQ, AWQ, GGUF, and bitsandbytes at you, plus cryptic suffixes like Q4_K_M and -4bit-128g. It's easy to assume one is simply "the best" and the rest are legacy. They're not. They're different tools for different machines — and picking the wrong one means a model that won't load, runs on the wrong device, or leaves accuracy on the table.

The analogy that untangles it: these are like video codecs. H.264, HEVC, ProRes — they all compress video, but you'd never hand a ProRes edit master to a phone or stream H.264 into a Hollywood grading suite. The content (your model) is the same; the container and the trade-offs differ by where it plays. So the two questions that actually decide the format aren't "which is smartest" but where does this run (GPU server? laptop? Apple Silicon?) and what am I doing (serving? fine-tuning?).

Two axes cut through the confusion. First, when the quantizing happens: GPTQ and AWQ are done ahead of time — a one-off calibration pass bakes a smaller checkpoint you download and serve, so there's zero conversion cost at load. bitsandbytes, by contrast, quantizes on the fly as the model loads, which is why it's the one you reach for during fine-tuning (QLoRA in FT3) — no separate pre-quantized artifact needed. Second, where it runs: GPTQ/AWQ are GPU-first, while GGUF is the llama.cpp format built to run beautifully on CPU and Apple Silicon, which is exactly why it dominates laptop/local use (LM4).

The mistake to sidestep: grabbing a GGUF file and trying to serve it through vLLM on a datacenter GPU (wrong runtime — it's a llama.cpp format), or hunting for a "GPTQ laptop build" that runs well on CPU (that's GGUF's job). Match the format to the machine and the task first; the exact bit-width and calibration knobs are a second-order tune. Read the table below as "same idea, four delivery vehicles," not "four competitors where one wins."

You'll meet four names; they differ in how they quantize and where they run:

Run a quantized model intermediate

Here's the reassuring surprise, and it's worth saying up front: running a quantized model is almost boringly ordinary. If you were bracing for a special "quantized inference API" — a different generate call, a manual dequantize step, some ritual to unpack the 4-bit weights — relax. There isn't one. You tell the loader "store this in 4 bits," and from that point every line of your code looks exactly like running any other model. The quantization lives inside the weights; your calling code never sees it.

The mental model: think of it like opening a zipped file. You don't rewrite your program to read a .zip — the library transparently decompresses as it reads, and your code just sees bytes. Here, the quantized weights are the compressed form; the library unpacks each small integer back into a usable number on the fly as the math needs it, in tiny local batches. That's the point of the one config knob you'll set below — bnb_4bit_compute_dtype: store the weights tiny (4-bit) to save memory and bandwidth, but compute in a wider 16-bit type so the arithmetic stays accurate. Small on disk and on the bus; roomy in the ALU. Best of both.

The mistake to head off is expecting a quantized model to be faster at everything. It shrinks memory and speeds memory-bound decode (the token-by-token generation from IC1) — but the one-time load can actually be a touch slower (there's unpacking work), and the on-the-fly quantize in bitsandbytes adds a moment at startup. You're optimizing the steady-state serving cost, not the cold start. And you still owe the model an eval: the warning after the lab is not boilerplate — INT4 can quietly cost you a few accuracy points on your task even when the demo sentence looks perfect. With that framing, the lab below is just the ordinary transformers load path with one extra config object bolted on.

quantize.pyimport torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # normal-float 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,  # compute in bf16, store in 4-bit
)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    quantization_config=quant, device_map="auto",
)
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

ids = tok("Explain quantization in one sentence.", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))

params = 7_000_000_000
bytes_per = {"fp16": 2, "int8": 1, "int4": 0.5}
for name, b in bytes_per.items():
    gb = params * b / 1024**3
    print(f"{name}: {gb:.1f} GB")
# fp16: 13.0 GB   int8: 6.5 GB   int4: 3.3 GB

def fits(params_b, vram_gb, overhead=1.20):
    out = {}
    for name, bpp in {"fp16":2, "int8":1, "int4":0.5}.items():
        need = params_b * bpp / 1024**3 * overhead
        out[name] = (round(need,1), need <= vram_gb)
    return out

for name,(need,ok) in fits(13_000_000_000, 24).items():
    print(f"{name}: needs {need} GB -> {'FITS' if ok else 'no'}")
# fp16: needs 29.1 GB -> no
# int8: needs 14.5 GB -> FITS
# int4: needs 7.3 GB  -> FITS

def quantize(weights, bits):
    lo, hi = min(weights), max(weights)
    levels = (1 << bits) - 1                 # e.g. 4 bits -> 15 levels
    scale = (hi - lo) / levels or 1.0
    q = [round((w - lo) / scale) * scale + lo for w in weights]
    mae = sum(abs(a-b) for a,b in zip(weights,q)) / len(weights)
    return round(mae, 4)

w = [0.11, -0.4, 0.9, -0.05, 0.33, -0.72, 0.5, 0.02]
for bits in (8, 4, 2):
    print(f"{bits}-bit MAE: {quantize(w, bits)}")
# 8-bit MAE: ~0.0016   4-bit: ~0.017   2-bit: ~0.12

def pick_format(target, need_accuracy_4bit=False, doing_qlora=False):
    if doing_qlora:
        return "bitsandbytes — on-the-fly 4-bit for QLoRA (FT3)"
    if target in ("laptop", "cpu", "apple-silicon"):
        return "GGUF — llama.cpp, runs CPU+GPU locally"
    if target == "gpu-server":
        return "AWQ — activation-aware, strong 4-bit accuracy" if need_accuracy_4bit \
               else "GPTQ — calibration-based, great on vLLM/TGI"
    return "GGUF — safe local default"

print(pick_format("laptop"))
print(pick_format("gpu-server", need_accuracy_4bit=True))
print(pick_format("gpu-server", doing_qlora=True))

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"   # a pre-quantized AWQ repo
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",          # place on available GPU(s)
)

msgs = [{"role": "user", "content": "Explain quantization in one line."}]
inputs = tok.apply_chat_template(msgs, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

candidates = [
    # name, accuracy, vram_gb
    ("fp16", 0.812, 13.0),
    ("int8", 0.809, 6.5),
    ("int4", 0.771, 3.3),
]
baseline = candidates[0][1]     # fp16 accuracy
TOL = 0.01                      # allow <=1 point absolute drop
VRAM = 8.0

ok = [c for c in candidates if baseline - c[1] <= TOL and c[2] <= VRAM]
pick = min(ok, key=lambda c: c[2]) if ok else None   # smallest that qualifies
print("qualify:", [c[0] for c in ok])   # ['int8']  (int4 drops 4.1 pts, fp16 too big)
print("PICK:", pick[0] if pick else "none -- relax VRAM or tolerance")

Knowledge check check yourself