Fine-tune a task adapter
The capstone: prepare data, QLoRA-tune (and optionally DPO-align) an open model for a narrow task, prove it beats the base with evals, and serve the merged model.
- a GPU +
pip install vllm(serving engine)
Learning objectives
- Prepare a clean, leakage-free dataset for a narrow task.
- QLoRA-tune a model and (optionally) align with DPO.
- Prove improvement with an eval-gated comparison to the base.
- Serve the tuned model and point course code at it.
code/proj-ft-adapter/ in the course, with a README. Run the scripts or copy the configs directly.The method advanced
Fine-tune a task adapter, end to end
- Pick a narrow task that hits one of FT1's four cases (style/format/latency/niche).
- FT2: build 300–1000 clean examples, chat-formatted, deduped, split 90/10.
- FT3: QLoRA-tune for 1–3 epochs; save the adapter.
- FT5 (optional): if it's a preference quality, add a DPO pass on ~100 pairs.
- FT6: eval tuned vs base on the task metric AND a general set — promote only if it wins clean.
- FT4 + IC6: merge the adapter and serve on vLLM; repoint an earlier lab's
base_url.
pipeline.sh# 1. prep.py -> train.jsonl, val.jsonl (FT2)
# 2. qlora_train.py -> out/adapter (FT3)
# 3. dpo_train.py -> dpo-out/ (optional) (FT5)
# 4. eval_serve.py -> target up? general intact? (FT6)
# 5. merge.py -> out/merged-model (FT4)
# 6. serve:
python -m vllm.entrypoints.openai.api_server --model out/merged-model
# 7. point any earlier lab at http://localhost:8000/v1 and confirm the
# tuned behavior shows up in a real workflow.
This shell script is the whole fine-tuning track on one page — the exact order to run the scripts from FT2 through FT6, ending with a served model. The numbered # lines are the recipe; the last real command actually starts the server.
- Steps 1–5 (all comments) chain the earlier labs:
prep.pymakes the data (FT2),qlora_train.pytrains the adapter (FT3),dpo_train.pyoptionally aligns it (FT5),eval_serve.pychecks it improved (FT6), andmerge.pybakes the adapter into a standalone model (FT4). The arrows show what each step produces. python -m vllm.entrypoints.openai.api_server --model out/merged-modelis the one line that actually runs: it starts a local server hosting your merged model, speaking the OpenAI API format so existing code can talk to it.- Step 7 (comment) is the payoff: point an earlier lab's
base_urlat your local server and confirm the tuned behavior shows up in a real workflow.
What the output means: Running the vllm line starts a server, typically at http://localhost:8000/v1. Any code that can call the OpenAI API can now use your fine-tuned model by pointing at that address.
Try this: Once the server is up, take a lab from earlier in the course and change its base_url to http://localhost:8000/v1. Seeing your tuned behavior appear inside a real workflow is the proof the whole pipeline worked.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Fine-tuning is data first. Dedup, role validation, and a deterministic split are what make a run reproducible and stop train/val leakage from inflating your eval.
Your task: Write prep(examples) that dedupes, validates roles, and does a deterministic 90/10 split into train.jsonl/val.jsonl.
Requirements:
- Validate each example has system, user, and assistant roles
- Drop exact duplicates
- Order deterministically before splitting (reproducible runs)
- Split 90/10 into train and val
- Demonstrate a dupe being dropped and the split sizes
💡 Hint: Hash each example for both dedup and a stable sort order so the same input always yields the same split.
Show solution
Runnable stdlib data prep — the highest-leverage, least-glamorous step:
import json, hashlib
def valid(ex):
roles = [m["role"] for m in ex["messages"]]
return roles[:1] == ["system"] and "user" in roles and "assistant" in roles
def prep(examples, train_path="train.jsonl", val_path="val.jsonl"):
seen, clean = set(), []
for ex in examples:
if not valid(ex):
continue
key = hashlib.md5(json.dumps(ex, sort_keys=True).encode()).hexdigest()
if key in seen:
continue
seen.add(key); clean.append(ex)
clean.sort(key=lambda e: hashlib.md5(json.dumps(e, sort_keys=True)
.encode()).hexdigest()) # deterministic order
cut = int(len(clean) * 0.9)
return clean[:cut], clean[cut:]
ex = lambda a: {"messages":[{"role":"system","content":"s"},
{"role":"user","content":"u"},{"role":"assistant","content":a}]}
tr, va = prep([ex("x"), ex("x"), ex("y"), ex("z")])
print(len(tr), len(va)) # 2 1 (dupe dropped, 90/10)Dedup + role validation + a deterministic split is what makes a run reproducible and stops train/val leakage from inflating your eval.
Context: The adapter config is small, and validating it before you burn GPU hours is free. A 4-bit base plus a low-rank adapter is what lets a 7B model tune on one consumer GPU.
Your task: Write the training config and launch (labelled needs-GPU), keeping config validation offline-testable.
Requirements:
- A config with rank, alpha, learning rate, and epochs
- A validator with sane ranges (e.g. alpha ≥ rank, 1–3 epochs, QLoRA-range lr)
- The validator runs offline and catches bad configs
- The training launch (peft/transformers, 4-bit base) is labelled needs-GPU
- The adapter is saved to an output dir
💡 Hint: The offline validator catches the config mistakes (alpha below rank, too many epochs) that quietly wreck a run before any GPU time is spent.
Show solution
The adapter config is small; validating it before you burn GPU hours is free. GPU launch is labeled:
from dataclasses import dataclass
@dataclass
class LoraConfig:
r: int = 16
alpha: int = 32
lr: float = 2e-4
epochs: int = 2
def validate(self):
assert 4 <= self.r <= 128, "rank out of range"
assert self.alpha >= self.r, "alpha should be >= rank"
assert 1 <= self.epochs <= 3, "keep epochs small to avoid overfit"
assert 1e-5 <= self.lr <= 5e-4, "lr out of typical QLoRA range"
return self
LoraConfig().validate() # offline: passes
print("config OK")
# --- needs GPU + libs: peft + transformers QLoRA training ---
# from peft import LoraConfig as PeftLora, get_peft_model
# from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
# model = AutoModelForCausalLM.from_pretrained(BASE, load_in_4bit=True)
# model = get_peft_model(model, PeftLora(r=16, lora_alpha=32,
# target_modules=["q_proj","v_proj"], task_type="CAUSAL_LM"))
# Trainer(model, args=TrainingArguments(output_dir="out/adapter",
# num_train_epochs=2, learning_rate=2e-4), ...).train()
# model.save_pretrained("out/adapter")The 4-bit base + low-rank adapter is what lets a 7B model tune on one consumer GPU. The offline validator catches the config mistakes (alpha < rank, too many epochs) that quietly wreck a run.
Context: A tuned model that wins the task but regresses on general prompts is a net loss. The two-sided gate is the single most important decision in the pipeline — it makes the tradeoff explicit and measurable.
Your task: Build promote(tuned, base) that requires a task-metric win AND a general-set score no worse than base, and prove it blocks a regression.
Requirements:
- Promotion requires the tuned model to beat base on the task metric
- It also requires general-set score no worse than base (within a tolerance)
- Return the decision and a reason
- A model that wins the task but drops general is blocked
- Demonstrate both a promote and a blocked regression
💡 Hint: Require both conditions to hold; the general-hold check is what stops you shipping a model that's great at one thing and worse at everything else.
Show solution
Two-sided gate — the single most important decision in the pipeline:
def promote(tuned, base, general_tol=0.0):
task_win = tuned["task"] > base["task"]
general_ok = tuned["general"] >= base["general"] - general_tol
decision = task_win and general_ok
return {"promote": decision,
"reason": ("wins task, holds general" if decision else
"regressed" if not general_ok else "no task gain")}
base = {"task": 0.62, "general": 0.80}
good = {"task": 0.81, "general": 0.80} # wins task, holds general
bad = {"task": 0.85, "general": 0.71} # wins task but forgot general
print(promote(good, base)) # promote True
print(promote(bad, base)) # promote False -> regressedFine-tuning trades general capability for task skill; without the general-hold check you ship a model that's great at one thing and worse at everything else. The gate makes that tradeoff explicit and measurable.
Context: When the task needs a specific style or refusal behavior, DPO on preference pairs directly optimizes 'prefer this answer over that one' — with far fewer, higher-quality pairs than SFT.
Your task: Model the DPO preference objective offline so the mechanics are clear, and label the real training as needs-GPU.
Requirements:
- A DPO loss over (prompt, chosen, rejected) using policy and reference log-probs
- A good pair (chosen more likely than reference) yields low loss
- A bad pair (model prefers the rejected) yields high loss
- The objective raises the log-prob margin of chosen over rejected
- The real trl training is labelled needs-GPU; the objective runs offline
💡 Hint: The loss is -log σ(β·margin) where margin is the policy-minus-reference gap for chosen vs rejected.
Show solution
The DPO objective in miniature — it raises the log-prob margin of chosen over rejected:
import math
def sigmoid(x): return 1 / (1 + math.exp(-x))
def dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta=0.1):
# margin of (policy - reference) for chosen vs rejected
margin = beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected))
return -math.log(sigmoid(margin))
# a good pair (chosen more likely than ref, rejected less) -> low loss
print(round(dpo_loss(-1.0, -3.0, -1.5, -2.0), 3)) # ~0.28
# a bad pair (model prefers the rejected answer) -> high loss
print(round(dpo_loss(-3.0, -1.0, -2.0, -1.5), 3)) # ~1.28
# --- needs GPU + libs: real DPO with trl ---
# from trl import DPOTrainer, DPOConfig
# DPOTrainer(model, ref_model, args=DPOConfig(output_dir="dpo-out", beta=0.1),
# train_dataset=pairs).train() # pairs: prompt/chosen/rejectedDPO needs far fewer, higher-quality pairs than SFT and directly optimizes "prefer this answer over that one" — ideal for tone, format, or refusal behavior that plain SFT data can't easily express.
Context: Ship a standalone artifact, not a base+adapter pair. Merging removes the adapter runtime dependency, and vLLM's OpenAI-compatible server means every earlier lab points at it with zero client changes.
Your task: Merge the adapter into a standalone model and serve it behind vLLM's OpenAI-compatible API, then confirm a client call returns tuned behavior.
Requirements:
- Merge the adapter into the base to a standalone model dir (labelled needs-GPU)
- Serve it with vLLM's OpenAI-compatible server (labelled)
- A provider-agnostic client call runs once the server is up
- Repointing an earlier lab's
base_urlis the whole integration - Confirm the call returns tuned behavior
💡 Hint: Because the server speaks the OpenAI protocol, only the endpoint moves — no client code changes to prove the tuned model in situ.
Show solution
Merging removes the adapter runtime dependency; vLLM exposes the standard /v1 API so every earlier lab points at it unchanged:
# --- needs GPU + libs: merge adapter into a standalone model ---
# from peft import PeftModel
# from transformers import AutoModelForCausalLM
# base = AutoModelForCausalLM.from_pretrained(BASE)
# merged = PeftModel.from_pretrained(base, "out/adapter").merge_and_unload()
# merged.save_pretrained("out/merged-model")
# --- needs GPU: serve the merged model, OpenAI-compatible ---
# python -m vllm.entrypoints.openai.api_server \
# --model out/merged-model --max-num-seqs 128 --port 8000
# client call is provider-agnostic (OpenAI-compatible) -- runnable once served:
from openai import OpenAI # needs: pip install openai + running server
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(model="out/merged-model",
messages=[{"role": "user", "content": "classify: refund my order"}])
print(resp.choices[0].message.content)Because the server speaks the OpenAI protocol, repointing an earlier lab's base_url to localhost:8000/v1 is the whole integration — no client changes to prove the tuned model in situ.
Context: A stakeholder must sign off before a tuned model replaces the base in production. The deliverable is a defensible comparison plus a safe path to roll it out and back — the fine-tune isn't 'done' until this loop exists.
Your task: Produce a before/after table (task + general + latency) and a canary+rollback plan keyed on a live quality metric.
Requirements:
- A table comparing base vs tuned on task, general, and p95 latency with deltas
- A canary that watches a live quality metric
- It rolls back automatically when live quality drops below a ratio of base
- The table proves the win offline; the canary guards production
- Demonstrate the table and a rollback trigger
💡 Hint: Keep the tuned model only while it holds, say, ≥98% of base's live quality; below that, auto-rollback.
Show solution
The deliverable that gets a fine-tune into production is a defensible comparison plus a safe path to roll it out and back:
def report(base, tuned):
rows = [("task acc", base["task"], tuned["task"]),
("general", base["general"], tuned["general"]),
("p95 ms", base["p95"], tuned["p95"])]
print(f"{'metric':10} {'base':>8} {'tuned':>8} {'delta':>8}")
for name, b, t in rows:
print(f"{name:10} {b:>8.2f} {t:>8.2f} {t-b:>+8.2f}")
def canary(live_quality, base_quality, min_ratio=0.98):
# keep tuned only if it holds >=98% of base's live quality
if live_quality < base_quality * min_ratio:
return "ROLLBACK"
return "KEEP"
report({"task":0.62,"general":0.80,"p95":90},
{"task":0.81,"general":0.80,"p95":95})
print(canary(live_quality=0.60, base_quality=0.80)) # ROLLBACKThe table proves the win offline; the canary watches the live metric and rolls back automatically if the tuned model underperforms in production — the fine-tune isn't "done" until this loop exists.
✓ Checkpoint — you can move on when you can…
- Prepare a clean dataset for a narrow task.
- QLoRA-tune and optionally DPO-align a model.
- Prove improvement with an eval-gated comparison.
- Merge, serve, and use the tuned model in a real workflow.
| Dimension | Meets the bar | Above the bar (staff) |
|---|---|---|
| Dataset quality | 300–1000 clean, chat-formatted examples that fit a narrow task; deduped, with a held-out 90/10 split. | Leakage between train and val is checked and ruled out; examples are audited for label noise and coverage of the task distribution. |
| Training done right | QLoRA-tuned for 1–3 epochs with the adapter saved; loss curves are sane (no obvious overfit/underfit). | Hyperparameters (rank, LR, epochs) were chosen deliberately and a small sweep justifies them; runs are reproducible from config. |
| Eval win over baseline | Tuned vs base is compared on the task metric and the tuned model wins by a margin you report. | The win is measured on a held-out set with enough examples to be meaningful; the eval is task-appropriate, not just loss. |
| No capability regression | The tuned model is also checked on a general set and hasn't obviously degraded on off-task ability. | Regression is quantified (general-set score before/after) and promotion is gated on 'wins on task AND holds on general'. |
| Alignment (if used) | If a preference quality mattered, a DPO pass on ~100 pairs was added and its effect measured separately. | The DPO effect is isolated from the SFT effect, and you can state what it bought and whether it was worth the added complexity. |
| Serving & integration | The adapter is merged and served on vLLM behind an OpenAI-compatible API; an earlier lab runs against it unchanged. | The deliverable ships as adapter + before/after eval table, and the tuned behavior is demonstrated inside a real workflow end-to-end. |
Score each dimension 0 (missing), 1 (meets bar), or 2 (above bar). 0–4: keep building. 5–8: a solid, defensible submission. 9–12: staff-level — you could hand this to a reviewer and defend every call. Any dimension at 0 blocks shipping regardless of the total.
Knowledge check check yourself
Why is the deliverable of a fine-tune the adapter plus a before/after eval table, not just the adapter?
Show answer
Before promoting a tuned model, why check it on a general set as well as the task metric?