LoRA & QLoRA with PEFT
Parameter-efficient tuning: LoRA trains tiny adapters instead of all weights, and QLoRA adds a 4-bit frozen base so a 7B model tunes on one consumer GPU. A full, runnable training job.
- a GPU (QLoRA fits a 7B model on one consumer card) +
pip install transformers peft trl bitsandbytes - a GPU +
pip install transformers peft trl datasets
Learning objectives
- Explain how LoRA tunes a model by training small adapter matrices, not all weights.
- Explain how QLoRA adds 4-bit quantization to fit big models on one GPU.
- Run a real QLoRA training job with PEFT + TRL.
- Choose rank, alpha, and target modules sensibly.
code/ft3-lora-qlora/ in the course, with a README. Run the scripts or copy the configs directly.LoRA: train a little, change a lot essential
Full fine-tuning updates all billions of weights — huge memory, easy to overfit. LoRA (Low-Rank Adaptation) freezes the model and trains tiny adapter matrices injected next to the big weight matrices. You train <1% of the parameters, get most of the benefit, and can swap adapters like plugins.
This picture explains the trick behind LoRA in three boxes. The big idea: don't retrain the giant model — freeze it and bolt on tiny trainable pieces.
- Frozen base weights (left) is the original multi-billion-parameter model, left unchanged. "Frozen" means training never touches it.
- + small LoRA adapters (trained) (middle) are the little matrices added alongside the big ones. They are the only thing that learns — the
<1% of paramslabel means you train less than one percent of the model. - Merged at inference (right) means when you actually use the model, the tiny adapters can be folded back into the base to act as one model — or kept separate and swapped like plugins.
In short: LoRA = freeze the big model, train a small add-on. Because the add-on is tiny, it's cheap to train, quick to store, and easy to swap in and out.
A real QLoRA run essential
qlora_train.pyimport torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
base = "mistralai/Mistral-7B-Instruct-v0.3"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto")
tok = AutoTokenizer.from_pretrained(base); tok.pad_token = tok.eos_token
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], # attention projections
)
ds = load_dataset("json", data_files="train.jsonl", split="train") # has a "text" field (FT2)
trainer = SFTTrainer(
model=model, tokenizer=tok, train_dataset=ds, peft_config=lora,
args=SFTConfig(output_dir="out", num_train_epochs=1, per_device_train_batch_size=2,
gradient_accumulation_steps=4, learning_rate=2e-4, bf16=True,
logging_steps=10, save_strategy="epoch"),
)
trainer.train()
trainer.save_model("out/adapter") # just the small adapter, not the whole model
This is a complete, real QLoRA training job. It loads a 7B model compressed to 4-bit so it fits on one GPU, attaches trainable LoRA adapters, and runs the trainer. Don't worry about every argument — focus on the four moving parts.
BitsAndBytesConfig(load_in_4bit=True, ...)is the Q in QLoRA: it tells the loader to squash the huge base model down to 4-bit numbers so it barely uses memory.AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, ...)loads it that way.LoraConfig(r=16, lora_alpha=32, ...)defines the adapters:ris their size/capacity,lora_alphascales them, andtarget_modules=["q_proj", "k_proj", "v_proj", "o_proj"]picks which layers (the attention projections) get an adapter.load_dataset(...)reads yourtrain.jsonl— the file already has a"text"field formatted in FT2, so the trainer knows what to learn from.SFTTrainer(...)wires the model, tokenizer, dataset and LoRA config together; theSFTConfig(...)holds training settings (1 epoch, batch size, learning rate).trainer.train()runs the training loop.trainer.save_model("out/adapter")saves only the adapter — a few megabytes, not the whole multi-gigabyte model.
What the output means: During training you'll see loss numbers printed every 10 steps (they should trend down). At the end, out/adapter holds your trained adapter — the small artifact you serve or merge later.
Try this: Change num_train_epochs=1 to 2 for a longer run, or raise r=16 to 32 for more adapter capacity. Change one knob at a time so you can tell what actually helped when you evaluate in FT6.
Choosing the knobs intermediate
| Knob | What it does | Sensible start |
|---|---|---|
| r (rank) | adapter capacity | 8–32 (higher = more capacity, more memory) |
| lora_alpha | adapter scaling | ≈ 2× r |
| target_modules | which layers get adapters | attention projections; add MLP for more capacity |
| learning_rate | step size | 1e-4 to 3e-4 for LoRA |
Exercise FT3.1 — Tune an adapter
Context: Running QLoRA once end-to-end — even before you judge quality — is how you confirm the pipeline actually trains and moves the model's behaviour.
Your task: Run QLoRA on your FT2 dataset for one epoch, save the adapter, then generate on a few held-out prompts and eyeball whether the behaviour shifted toward your target.
Requirements:
- Train for a single epoch on the FT2 dataset
- Save the adapter artifact
- Generate on several held-out prompts
- Confirm the outputs changed (behaviour shift), not that they're good yet
- Defer quality judgement to FT6
💡 Hint: You're checking that training ran and changed outputs — the QLoRA recipe from the Professional rung is the thing to launch.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: LoRA trains under 1% of a model by adding two small low-rank matrices to a weight instead of updating it: for a (d_in × d_out) weight, a rank-r adapter is a (d_in × r) and an (r × d_out) factor.
Your task: Compute the trainable parameter count of a rank-r LoRA adapter against the full matrix and show the fraction, for d=4096, r=8.
Requirements:
- Adapter params =
d_in×r + r×d_out - Full matrix params =
d_in×d_out - Print both counts and the trained fraction (≈0.4%)
- Make clear the adapter is a tiny swappable plug-in
💡 Hint: The two factors A and B are the whole parameter budget — sum their sizes and divide by the full matrix.
Show solution
The adapter parameter count is the whole cost-saving story:
def lora_params(d_in, d_out, r):
return d_in * r + r * d_out # A (d_in x r) + B (r x d_out)
d = 4096
full = d * d # full weight matrix
adapter = lora_params(d, d, r=8)
print(f"full: {full:,}") # 16,777,216
print(f"adapter: {adapter:,}") # 65,536
print(f"trained: {adapter/full*100:.2f}% of the matrix") # 0.39%A rank-8 adapter trains under half a percent of one matrix, so LoRA fits in a fraction of the memory and produces a tiny artifact you can swap like a plugin.
Context: QLoRA loads the frozen base in 4-bit and trains only the adapters in higher precision, which is what fits a 7B model on one consumer GPU.
Your task: Estimate QLoRA's memory (4-bit base + tiny adapter grads) and contrast it with full fine-tuning's weights + gradients + optimizer state for a 7B model.
Requirements:
- Full FT ≈ 12 bytes/param (weight 2 + grad 2 + Adam 4+4) → tens of GB, multi-GPU
- QLoRA base at 4-bit ≈ 0.5 bytes/param
- Add only the <1% trained adapters at 16 bytes/param
- Show the QLoRA total fits one GPU (single-digit GB)
- Attribute the saving to quantizing + freezing the base
💡 Hint: Full FT's cost is dominated by gradients and Adam's two moments, not the weights — freezing the base deletes almost all of it.
Show solution
Compare the two memory footprints directly:
params = 7e9
def gb(bytes_total): return bytes_total / 1024**3
# Full FT (mixed precision, rough): weights(2) + grad(2) + Adam m,v(4+4) ~ 12 B/param
full_ft = gb(params * 12)
# QLoRA: base frozen @ 4-bit (0.5 B/param), adapters ~0.4% trained @ 16 B/param
qlora = gb(params * 0.5 + params * 0.004 * 16)
print(f"full FT : {full_ft:.0f} GB -> multi-GPU") # ~78 GB
print(f"QLoRA : {qlora:.1f} GB -> one GPU") # ~3.7 GB (weights) + activationsFreezing the base and quantizing it to 4-bit removes almost all the memory; only the tiny adapters pay full-precision grad/optimizer cost. That is why a 7B tunes on a single consumer card.
Context: LoRA's knobs are rank r (capacity), alpha (scaling, often 2×r), and target modules (which layers get adapters); the update is scaled by alpha/r.
Your task: Write a helper that suggests sensible LoRA defaults from task difficulty and prints the effective scaling.
Requirements:
- Map easy/medium/hard → r = 8/16/32
- Set
alpha = 2×r(the stability heuristic) - Default targets to
q_proj, v_proj; add the rest of attention + MLP for hard tasks - Compute and print
scaling = alpha/r - Explain scaling is the factor LoRA multiplies its update by
💡 Hint: The alpha≈2r rule keeps scaling near a constant as you raise rank, which is why it's the stable default.
Show solution
Encode the common LoRA defaults:
def lora_knobs(task_difficulty):
r = {"easy": 8, "medium": 16, "hard": 32}[task_difficulty]
alpha = 2 * r # a common, stable default
targets = ["q_proj", "v_proj"] # attention proj is the usual minimum
if task_difficulty == "hard":
targets += ["k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
scaling = alpha / r # the factor LoRA multiplies its update by
return {"r": r, "alpha": alpha, "scaling": scaling, "target_modules": targets}
for t in ("easy", "hard"):
print(t, "->", lora_knobs(t))Rank sets adapter capacity; alpha≈2r keeps the update scaling stable; targeting more modules (all projections) adds capacity for harder tasks at more memory. Start small and only raise rank/targets if the task underfits.
Context: One frozen base can serve many task adapters, which is what makes a multi-task fleet cheap — adapters are MB-scale, full models are GB-scale.
Your task: Estimate total storage for N task adapters vs N full fine-tuned models and show why adapters win on disk.
Requirements:
- Size an adapter from the targeted fraction of params (MB-scale)
- Size a full model from all params at 2 bytes each (GB-scale)
- Compare totals for a fleet of ~20 adapters vs 20 full models
- Print the disk ratio (orders of magnitude)
- Note the base is shared and adapters hot-swap on top of it
💡 Hint: Multiply each per-artifact size by N and divide — the ratio is roughly the full-model size over the adapter size.
Show solution
The storage argument for adapters over full models:
def adapter_mb(params_b, r, frac_targeted=0.004, bytes_per=2):
return params_b * frac_targeted * bytes_per / 1024**2 # only the adapter weights
def full_model_gb(params_b, bytes_per=2):
return params_b * bytes_per / 1024**3
N = 20
a = adapter_mb(7e9, r=16) * N
f = full_model_gb(7e9) * N
print(f"{N} adapters : {a:.0f} MB total") # ~1067 MB
print(f"{N} full models: {f:.0f} GB total") # ~260 GB
print(f"adapters are ~{f*1024/a:.0f}x smaller on disk")Adapters are megabytes, full models are gigabytes, so a fleet of 20 task-specific adapters costs ~1 GB instead of ~260 GB — and one base can hot-swap between them (FT4).
Context: The whole QLoRA recipe is a real, runnable job: a 4-bit base under a LoRA config, trained by TRL's SFTTrainer, saving only the tiny adapter.
Your task: Show the lesson's real QLoRA training with PEFT + TRL so a teammate can launch it, using only documented APIs and labelling it as needing a GPU + libraries.
Requirements:
- Load the base in 4-bit via
BitsAndBytesConfig(nf4, bf16 compute) - Build a
LoraConfig(r, alpha, target_modules, dropout, CAUSAL_LM) - Train with TRL
SFTTrainerover a JSONL dataset - Save only the adapter with
save_model - Note the libs:
transformers peft trl bitsandbytes datasets
💡 Hint: It's the three earlier rungs made concrete — 4-bit base (rung 2), a LoraConfig (rung 3), and the tiny saved adapter (rung 1).
Show solution
The real training job — needs a GPU + pip install transformers peft trl bitsandbytes datasets:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
model_id = "mistralai/Mistral-7B-Instruct-v0.3"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16")
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb,
device_map="auto")
tok = AutoTokenizer.from_pretrained(model_id)
peft_cfg = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"],
lora_dropout=0.05, task_type="CAUSAL_LM")
ds = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(model=model, train_dataset=ds, peft_config=peft_cfg,
args=SFTConfig(output_dir="out", num_train_epochs=1,
per_device_train_batch_size=2))
trainer.train()
trainer.save_model("out/adapter") # tiny adapter, not a full modelThe 4-bit base (bitsandbytes) plus a small LoRA config (PEFT) fed to TRL's SFTTrainer is the whole QLoRA recipe. The saved artifact is just the adapter.
Context: The tuning-method choice is really a hardware-constraint decision, grounded in the ~12-bytes-per-param memory math, with QLoRA as the single-GPU default.
Your task: Given available VRAM, dataset size, and whether you must serve many task variants, write the decision that lands on full FT, LoRA, or QLoRA and justify it against the memory math.
Requirements:
- Compute full-FT VRAM need as
params × 12 bytes - Choose FULL FT only with a huge clean dataset AND enough VRAM
- Choose QLoRA when VRAM is tight (< ~24 GB)
- Prefer LoRA/QLoRA when many variants must be served
- Default to LoRA otherwise
💡 Hint: Start from the full-FT VRAM number: if the card can't hold it, quantize the base and the decision is essentially made for you.
Show solution
Route to a method from the real constraints:
def choose(vram_gb, params_b, need_many_variants, huge_clean_dataset):
full_need = params_b * 12 / 1024**3 # weights+grads+optimizer, rough
if huge_clean_dataset and vram_gb >= full_need:
return f"FULL FT — you can afford {full_need:.0f} GB and have the data"
if vram_gb < 24:
return "QLoRA — 4-bit base fits a single small GPU"
if need_many_variants:
return "LoRA/QLoRA — swappable adapters for a multi-task fleet"
return "LoRA — cheap, fast, adapter artifact"
print(choose(16, 7e9, True, False)) # QLoRA
print(choose(320, 7e9, False, True)) # FULL FTQLoRA is the default that makes single-GPU tuning possible; full fine-tuning is only justified when you have both the hardware to hold ~12 bytes/param and a large clean dataset to warrant the larger behavior shift.
✓ Checkpoint — you can move on when you can…
- Explain how LoRA trains adapters instead of all weights.
- Explain how QLoRA fits a big model on one GPU.
- Run a QLoRA job and save an adapter.
- Choose rank, alpha, and target modules sensibly.
Knowledge check check yourself
In the QLoRA config, why is the frozen base loaded in 4-bit (nf4) while bnb_4bit_compute_dtype is set to bfloat16?
Show answer
LoRA sets target_modules=["q_proj","k_proj","v_proj","o_proj"] and trains <1% of parameters. What does this achieve, and what does the r (rank) knob control?