DPO, RLHF & alignment

Beyond imitation: preferences intermediate

Try to write down the single "correct" answer to "was that reply helpful?" and you'll feel the whole problem in your gut. There isn't one. There are a hundred good replies and a thousand mediocre ones, and the difference between them is a matter of taste — tone, restraint, whether it hedged in the right places. That's the wall every team hits after FT3: supervised fine-tuning got the model fluent and on-topic, but the last mile — sounding trustworthy, refusing gracefully, not rambling — resists being pinned to gold examples. You can spend a month writing "perfect" answers and still watch the model produce something technically-imitating-your-examples yet somehow off.

Here's the reframe that unlocks it. Instead of demanding the right answer, ask a much easier question: given these two replies, which is better? Humans are dramatically better at judging than at authoring. Handed a blank page, "write the ideal refund response" is paralyzing; handed two drafts, "the first one is warmer and clearer" is instant and reliable. Preference tuning is built entirely on that asymmetry — it collects comparisons rather than gold answers, then teaches the model to move toward the winners and away from the losers.

The analogy that sticks: SFT is like teaching someone to cook by handing them recipes to copy exactly. Preference tuning is like being the friend who tastes two versions of the same dish and says "this one — more of that." You never had to write the recipe; you just had to prefer. Over many tastings, the cook internalizes your palate. That's what DPO does with a few thousand (chosen, rejected) pairs: it doesn't hand the model new recipes, it shifts its palate.

The common mistake is treating this as a replacement for SFT rather than a layer on top of it. Preference tuning has almost nothing to teach a model that can't yet form a coherent answer — the comparisons only sharpen behaviour that's already roughly in place. That's why the pipeline is always SFT first (learn the shape of good answers), then preferences (refine which good answers you like best). Skip the SFT foundation and you're asking someone to refine a palate for a cuisine they've never cooked. Keep that ordering in mind as you read the flow below — it starts, deliberately, from an already-tuned SFT model.

SFT (what FT3 did) teaches a model to imitate good examples. But some qualities — helpfulness, harmlessness, 'which of these two answers is better' — are easier to express as preferences than as gold answers. Preference tuning optimizes the model toward preferred over rejected responses.

RLHF, then DPO intermediate

RLHF is the technique that made ChatGPT possible, and understanding it is worth your time even though you'll rarely run it yourself. It's the original answer to "how do we optimize a model against fuzzy human preferences," and it works — it's what turned raw, occasionally-unhinged base models into assistants people trusted. But it does the job in two stages, and each stage is a place things can go wrong. Knowing the shape of it is what lets you appreciate why DPO felt like such a relief when it arrived.

Here's the RLHF pipeline in plain terms. Stage one: build a judge. You collect human comparisons ("answer A beats answer B"), then train a separate neural network — the reward model — whose only job is to look at a response and output a number: how much a human would like it. You've essentially distilled the crowd's taste into a scorer. Stage two: coach the model against that judge. Now you use reinforcement learning (specifically PPO) to nudge the LLM to generate responses that score high with the reward model, while a leash (a KL penalty back to the original model) stops it from drifting into gibberish that games the score. The picture to hold: a student (the LLM) writing answers, a grader (the reward model) scoring each one, and a coach (PPO) adjusting the student to chase higher grades.

Why that's painful in practice. You're now training and babysitting three models at once (the policy, the reward model, and a frozen reference), the RL loop is famously twitchy — learning rates, batch sizes, and the KL coefficient all have to be just right or training diverges — and the model loves to reward-hack: it finds responses the imperfect reward model scores highly but humans actually hate (endless flattery, hedge-everything non-answers). Getting RLHF stable is a real research skill, not a config change.

DPO's insight is almost cheeky: you can skip the middleman. The researchers showed, with a bit of algebra, that the reward model and the RL loop can be collapsed into a single loss function you compute directly on the (chosen, rejected) pairs. There's no separate judge to train and no RL coach to tune — you just run ordinary gradient descent, the same machinery as SFT, on a loss that says "raise the probability of the chosen answer relative to the rejected one, but don't stray too far from where you started." Same goal as RLHF, one model, one stable training loop. Read the table below as a checklist of moving parts removed: every "yes" in the RLHF column that becomes "no" under DPO is a thing that can no longer break. The mistake to avoid is reaching for PPO by default because it sounds more powerful — for the overwhelming majority of applied alignment, DPO gets you there with a fraction of the risk, and you only escalate to full RLHF if DPO visibly plateaus.

RLHF (Reinforcement Learning from Human Feedback) trains a separate reward model on human preferences, then uses RL (PPO) to optimize the LLM against it — powerful but complex and unstable. DPO (Direct Preference Optimization) gets similar results by optimizing the preference objective directly on the pairs — no reward model, no RL loop. For applied work, start with DPO.

Build preference data & run DPO advanced

Here's the part that surprises people who braced for the hard math: the DPO run itself is almost boringly short. The whole intellectual weight of preference tuning is in the data — the (prompt, chosen, rejected) triples — and the training call is a dozen lines of library code that look a lot like the SFT you already did in FT3. If you were dreading a reinforcement-learning tangle, exhale: DPO deliberately makes the code the easy bit so your effort goes where it belongs, into curating pairs that actually encode the behaviour you want.

Picture the data format first, because it's the thing you'll spend real time on. Every example is a single prompt paired with two completions to it: a chosen one you'd rather the model produce, and a rejected one you'd rather it didn't. Crucially, both must be plausible answers to the same prompt — DPO learns from the contrast, so the two sides need to differ on exactly the axis you care about and be otherwise comparable. If your "chosen" is a great answer and your "rejected" is off-topic gibberish, the model learns "avoid gibberish," which it already knew; you've spent a training pass teaching nothing. The signal lives in the gap, and a well-chosen gap ("warm and concise" vs "warm but rambling") is worth ten sloppy ones.

The one dial worth understanding before you run anything is beta. Think of it as how hard the model is allowed to lean toward your preferences versus how tightly it's tethered to where it started (the SFT model). A small beta (like the 0.1 in the lab) keeps the model close to its original behaviour and nudges gently — safe, and the usual starting point. A large beta pushes aggressively toward the chosen answers but risks the model over-committing and losing fluency or general ability it had before. The common mistake is cranking beta or the learning rate to "make it learn faster" and instead watching the model degrade — preference tuning is a scalpel, not a hammer, and gentle-and-check beats aggressive-and-pray every time.

One reassurance as you read the lab: nothing here trains a reward model or runs a reinforcement-learning loop. You load your SFT checkpoint, hand DPOTrainer the pairs and a beta, and call train() — the direct-loss machinery from the previous section does the rest. The loss quietly compares, for each pair, how much more likely the model makes the chosen answer versus the rejected one (relative to the frozen starting model), and steps to widen that margin. That's the whole mechanism, wrapped in an API that looks like every other trainer you've used.

dpo_train.pyfrom datasets import load_dataset
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

# Preference data: each row has prompt, chosen, rejected.
ds = load_dataset("json", data_files="prefs.jsonl", split="train")
# e.g. {"prompt": "...", "chosen": "helpful answer", "rejected": "worse answer"}

base = "out/merged-model"           # your SFT model from FT4
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
tok = AutoTokenizer.from_pretrained(base); tok.pad_token = tok.eos_token

trainer = DPOTrainer(
    model=model, args=DPOConfig(output_dir="dpo-out", beta=0.1,
        per_device_train_batch_size=2, learning_rate=5e-6, num_train_epochs=1),
    train_dataset=ds, tokenizer=tok,
)
trainer.train()

goals = [
    ("copy this exact support macro",         "SFT"),
    ("prefer polite over blunt phrasing",     "preference (DPO)"),
    ("be more helpful, less evasive",         "preference (DPO)"),
    ("emit this exact JSON schema",           "SFT"),
]
for goal, method in goals:
    print(f"{goal:38} -> {method}")

compare = {
    "reward model":         ("yes, separate", "no"),
    "RL loop":              ("yes (PPO)",     "no — direct loss"),
    "stability/complexity": ("harder",        "much simpler"),
    "applied default":      ("rarely start",  "start here"),
}
print(f"{'axis':22}{'RLHF (PPO)':18}DPO")
for k,(rlhf,dpo) in compare.items():
    print(f"{k:22}{rlhf:18}{dpo}")
print("\nDEFAULT for applied work: DPO — similar results, no reward model, no RL loop")

def build_prefs(rows):
    good, dropped = [], []
    for r in rows:
        if not r.get("chosen") or not r.get("rejected"):
            dropped.append((r, "empty side")); continue
        if r["chosen"].strip() == r["rejected"].strip():
            dropped.append((r, "chosen == rejected")); continue
        good.append({"prompt": r["prompt"], "chosen": r["chosen"],
                     "rejected": r["rejected"]})
    return good, dropped

rows = [
    {"prompt":"Refund?","chosen":"Sure, here's how...","rejected":"No."},
    {"prompt":"Refund?","chosen":"Same","rejected":"Same"},   # invalid
    {"prompt":"Hi","chosen":"Hello!","rejected":""},          # invalid
]
good, dropped = build_prefs(rows)
print(f"kept {len(good)}, dropped {len(dropped)}")   # kept 1, dropped 2
for r, why in dropped: print("  drop:", why)

def quality(ans):                      # toy proxy: helpful, non-refusing, specific
    score = 0
    score += 2 if len(ans.split()) >= 8 else 0
    score += 2 if "here" in ans.lower() or "steps" in ans.lower() else 0
    score -= 3 if ans.strip().lower() in ("no.", "can't help") else 0
    return score

pairs = [
    ("Here are the steps to get a refund: ...", "No."),
    ("Sure, contact support.",                  "Please contact support."),
]
for chosen, rejected in pairs:
    margin = quality(chosen) - quality(rejected)
    print(f"margin={margin:+d}  strong signal" if abs(margin)>=3
          else f"margin={margin:+d}  weak signal (near tie)")

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

model_id = "out/sft-model"            # start from your SFT checkpoint (FT3)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
tok = AutoTokenizer.from_pretrained(model_id)

# dataset columns must be exactly: prompt, chosen, rejected
ds = load_dataset("json", data_files="prefs.jsonl", split="train")
trainer = DPOTrainer(
    model=model, args=DPOConfig(output_dir="out/dpo", beta=0.1,
                                num_train_epochs=1),
    train_dataset=ds, processing_class=tok,
)
trainer.train()
trainer.save_model("out/dpo")

def recipe(have_gold, have_prefs, safety_critical):
    if not have_gold:
        return "COLLECT gold first — SFT is the foundation for any alignment"
    if not have_prefs:
        return "SFT only for now — add DPO once you can collect preference pairs"
    if safety_critical:
        return "SFT + DPO — DPO nudges toward safe/helpful; consider RLHF only if DPO plateaus"
    return "SFT + DPO — the applied default"

print(recipe(True, True, True))    # SFT + DPO ...
print(recipe(True, False, False))  # SFT only for now ...

Knowledge check check yourself