LoRA & QLoRA with PEFT

LoRA: train a little, change a lot essential

Here's the wall people hit. You've decided (back in FT1) that fine-tuning is the right move, you've built a clean dataset (FT2), and you go to actually train a 7B model — and your GPU immediately runs out of memory. Not because the model is too big to hold, but because training it means storing a gradient for every weight and two optimizer numbers for every weight on top of the weights themselves. That's the moment most people either rent a cluster they can't afford or give up. LoRA is the technique that lets you walk right past that wall on the hardware you already have — and it's why "lora peft" is one of the first things anyone tuning open models learns.

The analogy that makes it click: imagine a huge published textbook you're not allowed to reprint. You can't rewrite the pages — but you can slip a thin stack of sticky notes between them with your corrections and additions. The book stays exactly as printed (that's the frozen base); your sticky notes are the only thing you write (those are the adapters); and anyone reading gets book-plus-notes as if it were one edited volume. Crucially, your notes are a few pages, not a few thousand — so they're cheap to write, cheap to mail to a colleague, and you can peel one set off and stick a different set on for a different purpose. That last part is the superpower: one base model, many swappable adapters.

Why "low-rank" is the clever part, in plain terms. A weight matrix is a big grid of numbers — say 4096×4096, which is ~16 million values. LoRA never touches that grid. Instead it learns the change you'd want to make to it, and it forces that change to be expressible as two skinny matrices multiplied together: one that's 4096×r and one that's r×4096, where r (the "rank") is small — often 8 or 16. Multiply those two skinny matrices and you get something the same shape as the original grid, but you only ever stored and trained the two skinny factors. At r=8 that's ~65k numbers instead of 16 million — under half a percent (you'll compute this exactly in Exercise 1). The bet LoRA makes is that the useful adaptation for a specific task lives in a low-rank space — that you don't need all 16 million degrees of freedom to teach the model a new tone or format, just a handful of directions. In practice that bet pays off remarkably often.

The common mistake to head off now: LoRA does not shrink or change the base model, and it does not make inference cheaper by itself. The base still has all its billions of weights and still does all its usual math. What LoRA changes is the cost of training — the gradients and optimizer state now cover under 1% of the parameters instead of 100%. Keep "LoRA saves training memory, not model size" straight and the whole chapter stays clear.

Full fine-tuning updates all billions of weights — huge memory, easy to overfit. LoRA (Low-Rank Adaptation) freezes the model and trains tiny adapter matrices injected next to the big weight matrices. You train <1% of the parameters, get most of the benefit, and can swap adapters like plugins.

A real QLoRA run essential

LoRA cut the training memory; QLoRA cuts the part LoRA left behind — the base itself. Even frozen, a 7B model in its normal 16-bit form is ~14 GB just to hold, before you've stored a single activation. On a 16 GB consumer card that alone is game over. QLoRA's move is to load that frozen base in 4-bit — roughly a quarter of the memory, ~3.5 GB — using the quantization you met in IC2. Now the base fits with room to spare, the tiny LoRA adapters train on top in full precision, and a model that "needed a cluster" tunes on a single card. That combination — 4-bit frozen base + LoRA adapters — is the entire idea, and the lab below is it, end to end, in about a dozen real lines.

The subtlety that trips everyone, so grab it before the code: "load in 4-bit" and "do the math in 4-bit" are two different things, and QLoRA deliberately splits them. The weights are stored at 4 bits to save memory, but each time they're actually used in a matrix multiply they're temporarily de-quantized to 16-bit (that's what bnb_4bit_compute_dtype=torch.bfloat16 sets). Think of it like a compressed photo: it sits on disk tiny, but your screen still renders it at full resolution — you compress for storage, decompress to use. If you skipped that and did the arithmetic in raw 4-bit, accuracy would fall off a cliff. So the mental model is "store cheap, compute careful," and it's exactly why the knowledge-check at the end asks about this pairing.

Don't try to memorize every argument below — that's the surest way to feel lost. Read it as four moving parts stacked in order: (1) the BitsAndBytesConfig that makes the base 4-bit, (2) the LoraConfig that says which layers get adapters and how big, (3) the dataset you prepared in FT2, and (4) the SFTTrainer that ties them together and runs the loop. Everything else is a dial you can leave at its default on your first run. The one line worth staring at is the last one: you save only the adapter — a few megabytes — never the multi-gigabyte base, because the base didn't change.

qlora_train.pyimport torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

base = "mistralai/Mistral-7B-Instruct-v0.3"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto")
tok = AutoTokenizer.from_pretrained(base); tok.pad_token = tok.eos_token

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],  # attention projections
)
ds = load_dataset("json", data_files="train.jsonl", split="train")  # has a "text" field (FT2)

trainer = SFTTrainer(
    model=model, tokenizer=tok, train_dataset=ds, peft_config=lora,
    args=SFTConfig(output_dir="out", num_train_epochs=1, per_device_train_batch_size=2,
                   gradient_accumulation_steps=4, learning_rate=2e-4, bf16=True,
                   logging_steps=10, save_strategy="epoch"),
)
trainer.train()
trainer.save_model("out/adapter")     # just the small adapter, not the whole model

Choosing the knobs intermediate

This is where good intentions go to die. You've got a run working, so the temptation is to "tune it properly" by bumping rank, adding target modules, and nudging the learning rate — all at once, in one heroic edit. Then the eval moves and you have no idea which change did it, so you can't repeat the win or undo the loss. The knobs below are powerful, but the real skill in this section isn't knowing what each dial means — it's the discipline to move one at a time. Read the table as a set of sensible starting values, not a set of things to immediately change.

The two knobs people misunderstand most are r and lora_alpha, so let's build intuition for them before the table. Rank r is the capacity of the adapter — how much the model is allowed to learn. Think of it like the number of dials on a mixing board: more dials means finer control, but also more that can be misconfigured, and more memory to hold them. Small tasks (a fixed output format, a consistent tone) need very few; genuinely hard behavioral shifts need more. lora_alpha is a scaling factor on how strongly the adapter's learned change is applied — LoRA multiplies its update by alpha/r. The near-universal heuristic is alpha ≈ 2×r, and the reason is quietly elegant: as you raise r, keeping alpha at twice it holds that alpha/r ratio roughly constant, so bumping capacity doesn't accidentally also crank up the update's strength and destabilize training. That's why you'll almost never see alpha tuned independently on a first pass — you set it as a function of r and forget it.

Where to point the adapters is the third real decision. target_modules lists which of the model's weight matrices get an adapter bolted on. The standard, cheapest choice is the attention projections (q_proj, k_proj, v_proj, o_proj) — the parts of the model that decide what to pay attention to, which is usually where task-specific behavior lives. If evals later say the model is underfitting — trying but not quite getting there — you widen the net to include the MLP layers (gate_proj, up_proj, down_proj) for more capacity, at the cost of more memory and more parameters to train. The honest default, and the one the tip below repeats, is to start narrow and only widen when the evidence tells you to. Guessing at "more is better" up front just burns GPU hours and muddies your results.

def lora_params(d_in, d_out, r):
    return d_in * r + r * d_out          # A (d_in x r) + B (r x d_out)

d = 4096
full = d * d                              # full weight matrix
adapter = lora_params(d, d, r=8)
print(f"full:    {full:,}")               # 16,777,216
print(f"adapter: {adapter:,}")            # 65,536
print(f"trained: {adapter/full*100:.2f}% of the matrix")  # 0.39%

params = 7e9
def gb(bytes_total): return bytes_total / 1024**3

# Full FT (mixed precision, rough): weights(2) + grad(2) + Adam m,v(4+4) ~ 12 B/param
full_ft = gb(params * 12)

# QLoRA: base frozen @ 4-bit (0.5 B/param), adapters ~0.4% trained @ 16 B/param
qlora = gb(params * 0.5 + params * 0.004 * 16)
print(f"full FT : {full_ft:.0f} GB  -> multi-GPU")   # ~78 GB
print(f"QLoRA   : {qlora:.1f} GB  -> one GPU")        # ~3.7 GB (weights) + activations

def lora_knobs(task_difficulty):
    r = {"easy": 8, "medium": 16, "hard": 32}[task_difficulty]
    alpha = 2 * r                         # a common, stable default
    targets = ["q_proj", "v_proj"]        # attention proj is the usual minimum
    if task_difficulty == "hard":
        targets += ["k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
    scaling = alpha / r                   # the factor LoRA multiplies its update by
    return {"r": r, "alpha": alpha, "scaling": scaling, "target_modules": targets}

for t in ("easy", "hard"):
    print(t, "->", lora_knobs(t))

def adapter_mb(params_b, r, frac_targeted=0.004, bytes_per=2):
    return params_b * frac_targeted * bytes_per / 1024**2   # only the adapter weights

def full_model_gb(params_b, bytes_per=2):
    return params_b * bytes_per / 1024**3

N = 20
a = adapter_mb(7e9, r=16) * N
f = full_model_gb(7e9) * N
print(f"{N} adapters : {a:.0f} MB total")   # ~1067 MB
print(f"{N} full models: {f:.0f} GB total")  # ~260 GB
print(f"adapters are ~{f*1024/a:.0f}x smaller on disk")

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model_id = "mistralai/Mistral-7B-Instruct-v0.3"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype="bfloat16")
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb,
                                             device_map="auto")
tok = AutoTokenizer.from_pretrained(model_id)

peft_cfg = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"],
                      lora_dropout=0.05, task_type="CAUSAL_LM")
ds = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(model=model, train_dataset=ds, peft_config=peft_cfg,
                     args=SFTConfig(output_dir="out", num_train_epochs=1,
                                    per_device_train_batch_size=2))
trainer.train()
trainer.save_model("out/adapter")   # tiny adapter, not a full model

def choose(vram_gb, params_b, need_many_variants, huge_clean_dataset):
    full_need = params_b * 12 / 1024**3      # weights+grads+optimizer, rough
    if huge_clean_dataset and vram_gb >= full_need:
        return f"FULL FT — you can afford {full_need:.0f} GB and have the data"
    if vram_gb < 24:
        return "QLoRA — 4-bit base fits a single small GPU"
    if need_many_variants:
        return "LoRA/QLoRA — swappable adapters for a multi-task fleet"
    return "LoRA — cheap, fast, adapter artifact"

print(choose(16, 7e9, True, False))    # QLoRA
print(choose(320, 7e9, False, True))   # FULL FT

Knowledge check check yourself