DPO, RLHF & alignment
Some qualities are easier to express as preferences than gold answers. RLHF does this with a reward model + RL; DPO does it directly on chosen/rejected pairs — the applied default.
- a GPU +
pip install trl datasets
Learning objectives
- Explain the difference between SFT and preference tuning (DPO/RLHF).
- Understand RLHF conceptually and why DPO simplified it.
- Build a preference dataset (chosen vs rejected).
- Run a DPO training step on top of an SFT model.
code/ft5-dpo-rlhf/ in the course, with a README. Run the scripts or copy the configs directly.Beyond imitation: preferences intermediate
SFT (what FT3 did) teaches a model to imitate good examples. But some qualities — helpfulness, harmlessness, 'which of these two answers is better' — are easier to express as preferences than as gold answers. Preference tuning optimizes the model toward preferred over rejected responses.
This flow shows preference tuning: taking a model that already imitates good answers and nudging it to prefer better answers over worse ones. Read the four boxes left to right.
- SFT model (left) is where you start — a model that has learned to imitate good examples (that's what FT3 produced).
- Preference data (chosen/rejected) is the new ingredient: pairs of answers to the same prompt where one is marked better (chosen) and one worse (rejected).
- DPO training optimizes the model directly on those pairs — the
no reward modellabel means it skips the complicated extra model that older RLHF needed. - Aligned model (right) is the result: it now prefers good responses over the kind you marked as worse.
In short: Preference tuning teaches a model "this answer is better than that one", which is often easier than writing one perfect gold answer. DPO does it in one step, with no separate reward model.
RLHF, then DPO intermediate
RLHF (Reinforcement Learning from Human Feedback) trains a separate reward model on human preferences, then uses RL (PPO) to optimize the LLM against it — powerful but complex and unstable. DPO (Direct Preference Optimization) gets similar results by optimizing the preference objective directly on the pairs — no reward model, no RL loop. For applied work, start with DPO.
| RLHF (PPO) | DPO | |
|---|---|---|
| Reward model | yes, separate | no |
| RL loop | yes (PPO) | no — direct loss |
| Stability/complexity | harder | much simpler |
| Applied default | rarely | start here |
Build preference data & run DPO advanced
dpo_train.pyfrom datasets import load_dataset
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
# Preference data: each row has prompt, chosen, rejected.
ds = load_dataset("json", data_files="prefs.jsonl", split="train")
# e.g. {"prompt": "...", "chosen": "helpful answer", "rejected": "worse answer"}
base = "out/merged-model" # your SFT model from FT4
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
tok = AutoTokenizer.from_pretrained(base); tok.pad_token = tok.eos_token
trainer = DPOTrainer(
model=model, args=DPOConfig(output_dir="dpo-out", beta=0.1,
per_device_train_batch_size=2, learning_rate=5e-6, num_train_epochs=1),
train_dataset=ds, tokenizer=tok,
)
trainer.train()
This runs DPO (Direct Preference Optimization) on top of your already-trained model. Instead of one correct answer per example, DPO learns from pairs: a better answer and a worse one.
load_dataset("json", data_files="prefs.jsonl", ...)loads the preference data. Each row has three fields — aprompt, achosen(better) answer, and arejected(worse) answer, as the comment shows.base = "out/merged-model"points at your SFT model from FT4, and the next two lines load that model and its tokenizer — DPO starts from the model you already tuned.DPOTrainer(...)withDPOConfig(..., beta=0.1, ...)sets up the run.betacontrols how strongly it pushes toward chosen over rejected; the smalllearning_rate=5e-6keeps the nudge gentle so it doesn't wreck existing ability.trainer.train()runs the preference-tuning loop.
What the output means: Training prints loss numbers as it runs. When it finishes, the model in dpo-out has shifted toward the style of your chosen answers — no separate reward model was ever needed.
Try this: The rejected answer must be genuinely worse in the way you care about. Try building 10 pairs by hand where "chosen" is concise and "rejected" is rambling — sloppy pairs push the model in confused directions.
Exercise FT5.1 — Align a behavior
Context: Preference tuning is easiest to feel by aligning one concrete quality end-to-end and checking the behaviour appeared without breaking correctness.
Your task: Build ~100 preference pairs for a quality you want (e.g. concise over verbose), run DPO on your SFT model, and compare before/after on held-out prompts.
Requirements:
- Assemble ~100 chosen/rejected pairs for one target quality
- Run DPO on the SFT model
- Compare before vs after on held-out prompts
- Confirm the aligned behaviour appears
- Confirm correctness didn't regress
💡 Hint: Reuse the pair-validation and DPOTrainer steps from the ladder; the finding is whether the behaviour moved without a correctness regression.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: SFT imitates one gold answer; preference tuning learns from (chosen, rejected) pairs. A single correct output points to SFT; a “better than” comparison points to DPO.
Your task: Classify a set of training goals into which paradigm fits best — SFT or preference tuning (DPO).
Requirements:
- Map exact-output goals (copy a macro, emit an exact schema) → SFT
- Map comparative-quality goals (prefer polite over blunt, be more helpful) → DPO
- State the rule: one right answer → SFT; “better than” → preference tuning
- Cover at least one goal of each kind
💡 Hint: Ask whether the goal has a single gold answer or only a comparison between two — that fork is the classifier.
Show solution
Sort goals by whether a single gold answer exists:
goals = [
("copy this exact support macro", "SFT"),
("prefer polite over blunt phrasing", "preference (DPO)"),
("be more helpful, less evasive", "preference (DPO)"),
("emit this exact JSON schema", "SFT"),
]
for goal, method in goals:
print(f"{goal:38} -> {method}")When there is one correct answer, SFT imitation works. When 'better' is a comparison between two answers (helpful vs evasive, polite vs blunt), preferences capture it more naturally than any single gold label.
Context: RLHF trains a separate reward model then optimizes with PPO; DPO optimizes the preference objective directly, achieving RLHF-like alignment without the reward model or the unstable RL loop.
Your task: Encode the RLHF-vs-DPO comparison and print the applied-work default.
Requirements:
- Contrast reward model (yes/no), RL loop (PPO / none), and complexity
- Print the comparison as a table
- Declare DPO the default for applied work
- Explain DPO optimizes directly on pairs, skipping reward model + PPO
💡 Hint: The row that matters is “how many moving parts” — DPO drops both the reward model and the RL loop.
Show solution
The comparison table as a decision:
compare = {
"reward model": ("yes, separate", "no"),
"RL loop": ("yes (PPO)", "no — direct loss"),
"stability/complexity": ("harder", "much simpler"),
"applied default": ("rarely start", "start here"),
}
print(f"{'axis':22}{'RLHF (PPO)':18}DPO")
for k,(rlhf,dpo) in compare.items():
print(f"{k:22}{rlhf:18}{dpo}")
print("\nDEFAULT for applied work: DPO — similar results, no reward model, no RL loop")DPO gets RLHF-like alignment by optimizing directly on the pairs, skipping the separate reward model and unstable RL loop. For applied work you start with DPO and only reach for PPO if you must.
Context: A DPO example is {prompt, chosen, rejected}, and the model learns from the gap between chosen and rejected — so identical or empty sides carry no signal.
Your task: Write a builder that validates preference pairs (chosen ≠ rejected, both non-empty, same prompt) and drops invalid rows with a reason.
Requirements:
- Keep valid
{prompt, chosen, rejected}rows - Drop rows with an empty chosen or rejected side
- Drop rows where chosen equals rejected after stripping
- Return both the good rows and the dropped ones with reasons
- Explain that no gap means no learning signal
💡 Hint: Each drop condition maps to a reason string; a pair where the two sides match teaches the model nothing, so it must go.
Show solution
Validate the (chosen, rejected) structure DPO requires:
def build_prefs(rows):
good, dropped = [], []
for r in rows:
if not r.get("chosen") or not r.get("rejected"):
dropped.append((r, "empty side")); continue
if r["chosen"].strip() == r["rejected"].strip():
dropped.append((r, "chosen == rejected")); continue
good.append({"prompt": r["prompt"], "chosen": r["chosen"],
"rejected": r["rejected"]})
return good, dropped
rows = [
{"prompt":"Refund?","chosen":"Sure, here's how...","rejected":"No."},
{"prompt":"Refund?","chosen":"Same","rejected":"Same"}, # invalid
{"prompt":"Hi","chosen":"Hello!","rejected":""}, # invalid
]
good, dropped = build_prefs(rows)
print(f"kept {len(good)}, dropped {len(dropped)}") # kept 1, dropped 2
for r, why in dropped: print(" drop:", why)DPO learns from the gap between chosen and rejected, so identical or empty sides carry no signal and must be dropped. Clean pairs are as important here as clean examples are for SFT.
Context: DPO's learning signal is the margin between chosen and rejected: a clear chosen>rejected gap gives a strong gradient, while a near-tie teaches almost nothing.
Your task: Model the DPO intuition without training — score answers by a proxy “quality”, compute each pair's margin, and rank pairs by learning signal.
Requirements:
- Define a toy quality proxy (reward length/structure, penalize refusals)
- Compute
margin = quality(chosen) − quality(rejected)per pair - Label large margins strong signal and near-ties weak signal
- Rank pairs by margin
- Conclude you should curate real quality gaps
💡 Hint: It's the same lesson as the deduper, one level up: pairs that barely differ are the ones to prune, this time measured by the quality margin.
Show solution
A proxy showing margin = learning signal:
def quality(ans): # toy proxy: helpful, non-refusing, specific
score = 0
score += 2 if len(ans.split()) >= 8 else 0
score += 2 if "here" in ans.lower() or "steps" in ans.lower() else 0
score -= 3 if ans.strip().lower() in ("no.", "can't help") else 0
return score
pairs = [
("Here are the steps to get a refund: ...", "No."),
("Sure, contact support.", "Please contact support."),
]
for chosen, rejected in pairs:
margin = quality(chosen) - quality(rejected)
print(f"margin={margin:+d} strong signal" if abs(margin)>=3
else f"margin={margin:+d} weak signal (near tie)")A pair where the chosen answer clearly beats the rejected one gives DPO a strong gradient; near-ties teach almost nothing. Curate pairs with real quality gaps, not arbitrary A/B rows.
Context: DPO is a real, runnable step on top of an SFT model: TRL's DPOTrainer optimizes the preference loss directly, with beta controlling how strongly it prefers chosen over rejected.
Your task: Show the real TRL DPOTrainer on top of an SFT model, using only documented APIs and labelling it as needing a GPU + libraries.
Requirements:
- Start from an SFT checkpoint (load model + tokenizer)
- Use a JSONL dataset with exactly
prompt, chosen, rejectedcolumns - Run
DPOTrainerwith aDPOConfig(beta, one epoch) - Train, then save the model — no reward model, no PPO
- Note
betatunes preference strength; libs:trl peft transformers datasets
💡 Hint: DPO starts where SFT left off — point it at the SFT checkpoint and a three-column pairs dataset, and beta is the one dial to know.
Show solution
The real DPO run — needs a GPU + pip install trl peft transformers datasets:
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset
model_id = "out/sft-model" # start from your SFT checkpoint (FT3)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
tok = AutoTokenizer.from_pretrained(model_id)
# dataset columns must be exactly: prompt, chosen, rejected
ds = load_dataset("json", data_files="prefs.jsonl", split="train")
trainer = DPOTrainer(
model=model, args=DPOConfig(output_dir="out/dpo", beta=0.1,
num_train_epochs=1),
train_dataset=ds, processing_class=tok,
)
trainer.train()
trainer.save_model("out/dpo")DPO starts from the SFT model and optimizes the preference loss directly — no reward model, no PPO loop. beta controls how strongly it prefers chosen over rejected.
Context: Alignment is layered by the data and risk you have: SFT is the foundation, DPO the refinement, and RLHF is reserved for when DPO stops improving.
Your task: For a support assistant, given whether you have gold answers, preference pairs, and a safety-critical bar, decide the alignment recipe and justify it.
Requirements:
- No gold answers → collect gold and do SFT first
- Gold but no pairs → SFT only, add DPO later
- Safety-critical with pairs → SFT + DPO (RLHF only if DPO plateaus)
- Otherwise SFT + DPO as the applied default
- Justify by available data and risk
💡 Hint: Walk the data you actually have: gold unlocks SFT, pairs unlock DPO, and RLHF is the last resort, not the first.
Show solution
Route the alignment recipe from what data and risk you have:
def recipe(have_gold, have_prefs, safety_critical):
if not have_gold:
return "COLLECT gold first — SFT is the foundation for any alignment"
if not have_prefs:
return "SFT only for now — add DPO once you can collect preference pairs"
if safety_critical:
return "SFT + DPO — DPO nudges toward safe/helpful; consider RLHF only if DPO plateaus"
return "SFT + DPO — the applied default"
print(recipe(True, True, True)) # SFT + DPO ...
print(recipe(True, False, False)) # SFT only for now ...SFT builds the base behavior, DPO refines it toward preferred responses cheaply, and full RLHF is reserved for cases where DPO stops improving. Start simple and layer alignment as your data allows.
✓ Checkpoint — you can move on when you can…
- Distinguish SFT from preference tuning.
- Explain RLHF and why DPO simplified it.
- Build a chosen/rejected preference dataset.
- Run a DPO step on an SFT model.
Knowledge check check yourself
How does DPO differ from RLHF (PPO) in what it requires, and why is DPO the applied default?
Show answer
A DPO dataset uses (prompt, chosen, rejected) triples. Why does the quality of the 'rejected' response matter as much as 'chosen'?