Specialized Topics · Part T4

Fine-tuning vs RAG vs Prompting

Q: The lesson's decision framework climbs prompting → RAG → fine-tuning. Why is fine-tuning usually the wrong answer when you need to add knowledge?

Fine-tuning changes the model's behavior (style, format, a narrow skill), not what it knows — it's a poor way to add facts. Missing knowledge (your docs, product details, fresh data) is a RAG problem: you retrieve the right text into the prompt. Prompting and RAG are cheaper and solve most problems, so fine-tuning is the last-resort rung you reach only when you need consistent behavior.

"Should we fine-tune?" is the most common — and most misunderstood — question in applied LLMs. The answer is usually no, not yet. This part gives you the decision framework: what each technique actually changes, when each wins, how LoRA makes fine-tuning cheap, and how to prepare data — so you pick deliberately instead of reaching for the most expensive option first.

⏱️ ~1.5 hours🧪 3 labs🎯 Beginner→Tech-lead

Learning objectives

State the fine-tune vs RAG vs prompt decision clearly.
Recognize the cases where each wins.
Estimate the cost/effort of fine-tuning honestly.
Make and defend the build decision as a lead.

▶ Runnable companionEvery code block in this lesson is also a standalone file under code/xt4-finetuning/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

1 · The most misunderstood question essential

"Should we fine-tune?" is usually answered no, not yet. Prompting and RAG solve most problems faster and cheaper. Fine-tuning changes behavior (style, format, a narrow skill) — it's a poor way to add knowledge (that's RAG). Match the tool to the problem.

🗺️ How to read this diagram

This is the whole lesson in one row of boxes: when your LLM output isn't good enough, you climb these rungs left to right, cheapest first, and you stop as soon as the output is good. Most real problems never reach the last box.

Need better output (leftmost) is the starting point — you have a task and the model's answers aren't right yet. The label cheapest is a reminder: always begin at the cheap end.
Prompt engineering is the first thing to try: rewrite the instructions, add examples, set the format. It's free, instant to change, and fixes a surprising amount. The arrow means "if that wasn't enough, move right."
+ RAG (knowledge) is next: when the model is missing facts (your docs, product details, fresh data), you fetch the right text and paste it into the prompt. RAG adds knowledge, not new behavior.
Fine-tune (behavior) is the last, most expensive rung — labeled last resort. You only reach it when you need a consistent style, a strict format, or a narrow skill that prompting and RAG couldn't deliver. Note it changes behavior, not knowledge.

In short: Read the boxes as a staircase you climb only when forced. The single most common mistake in applied LLMs is jumping straight to the far-right box (fine-tuning) before trying the two cheap ones on its left.

2 · What each technique changes essential

Technique	Changes	Best for
Prompting	instructions in-context	most tasks; instant iteration
RAG	knowledge in-context	facts, docs, freshness, citations
Fine-tuning	the model's behavior	consistent style/format, narrow skill, latency

3 · Intermediate — the four cases fine-tuning wins intermediate

Tune when: a consistent style/voice is needed, a strict output format must be reliable, a small tuned model can replace a big prompted one (latency/cost), or a narrow task where a specialized model beats a general one. If none apply, don't tune.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Python · should you fine-tune? (runs)

decision.pydef should_finetune(needs_facts, needs_style, needs_strict_format,
                    latency_critical, prompt_rag_tried):
    if not prompt_rag_tried:
        return "NO — try prompting + RAG first"
    if needs_facts and not (needs_style or needs_strict_format):
        return "NO — that's a RAG problem, not fine-tuning"
    if needs_style or needs_strict_format or latency_critical:
        return "MAYBE — a real fine-tuning case; measure vs prompt+RAG first"
    return "NO — no clear behavior change needed"

print(should_finetune(True, False, False, False, True))     # facts -> RAG
print(should_finetune(False, True, True, True, True))        # style+format+latency -> maybe
print(should_finetune(False, False, False, False, False))    # haven't tried basics

NO — that's a RAG problem, not fine-tuning
MAYBE — a real fine-tuning case; measure vs prompt+RAG first
NO — try prompting + RAG first

▶ How this works

This little function turns the whole "should we fine-tune?" debate into a set of plain if-checks. You feed it yes/no answers about your task, and it tells you NO, MAYBE, or points you at the cheaper tool instead. Read the checks top to bottom — the first one that matches wins and returns.

The function takes five true/false inputs describing your task: does it need facts, a consistent style, a strict format, low latency, and — crucially — whether you've already tried prompting + RAG.
The first check is the gatekeeper: if not prompt_rag_tried means "you haven't even tried the cheap options yet" — so it immediately returns NO, try prompting + RAG first. This alone stops most premature fine-tuning.
The second check catches the classic mistake: if you only need facts (and not style or format), that's a knowledge problem — the comment # facts -> RAG flags it. Fine-tuning is the wrong tool; use RAG.
The third check is the one real green light: if you genuinely need style, a strict format, or speed, it returns MAYBE — a real fine-tuning case, and even then tells you to measure against prompt+RAG first. If nothing matched, the final line returns NO — no clear behavior change needed.

What the output means: The three print calls run three example tasks. You see NO — that's a RAG problem… (facts only), then MAYBE — a real fine-tuning case… (style + format + latency), then NO — try prompting + RAG first (basics not tried). Notice the order of outputs matches the order the checks fire, not the order of the calls.

Try this: Change the first call's last argument from True to False (pretend you skipped prompt+RAG) and predict the answer before running — the gatekeeper check should flip it to NO, try prompting + RAG first, no matter what the other flags say.

4 · Advanced — the true cost of fine-tuning advanced

Fine-tuning isn't just a training run. The real cost: dataset creation (the dominant effort), training compute, evaluation, and ongoing maintenance (re-tune when the base model or requirements change). Estimate it before committing.

Python · estimate fine-tuning effort (runs)

cost.pydef finetune_effort(examples_needed, hours_per_100_examples, gpu_hours, eval_hours):
    data_hours = examples_needed / 100 * hours_per_100_examples
    total = data_hours + gpu_hours + eval_hours
    return {"data_prep_hours": round(data_hours), "total_hours": round(total),
            "data_is_pct": round(100*data_hours/total)}

print(finetune_effort(examples_needed=1000, hours_per_100_examples=3,
                      gpu_hours=4, eval_hours=8))

{'data_prep_hours': 30, 'total_hours': 42, 'data_is_pct': 71}

▶ How this works

Fine-tuning feels like "just run a training command," but the real work is building the dataset. This function adds up where your hours actually go so you can see that truth in numbers before you commit.

It takes four honest estimates: how many training examples you need, how many hours it takes to hand-build 100 examples, the GPU hours for the training run, and the eval hours to check the result.
data_hours is the big one: examples_needed / 100 * hours_per_100_examples scales your per-100 labeling rate up to the full dataset. This is usually the dominant cost.
total sums data + GPU + eval. Then round(100*data_hours/total) computes what percent of the whole effort is just data prep — the number that makes the point.
It returns a small dictionary of three figures so the caller can print them clearly, rather than one buried number.

What the output means: For 1000 examples at 3 hours per 100, the result is {'data_prep_hours': 30, 'total_hours': 42, 'data_is_pct': 71}. The headline: 71% of the effort is building the dataset — the training run (4 GPU hours) is the small part. That's why the warning below says data prep is most of the work.

Try this: Bump examples_needed to 3000 and re-run. Watch data_is_pct climb even higher — more scale means data prep dominates even more, which is exactly why teams underestimate fine-tuning.

Data prep is most of the workThe training command is minutes; building a clean, labeled dataset is days. Teams underestimate this and abandon half-done tunes. If you can't commit to quality data, you can't fine-tune well — do prompt+RAG instead. (The hands-on how is the Fine-tuning track.)

5 · Professional — measure tuned vs baseline professional

A fine-tune is only justified if it beats prompt+RAG on your evals — target metric up, general capability not regressed. Never ship a tune on faith; the eval-gated comparison (Ch 5) is the decision-maker.

6 · Tech-lead — the build decision tech-lead

A lead owns the fine-tune vs RAG vs prompt call: run the decision framework, estimate cost honestly, require a measured win over the cheaper alternatives, and account for maintenance. Most of the time the disciplined answer is "prompt + RAG" — and saying no to a premature fine-tune is a senior move.

The hands-on fine-tuning track goes deeperThis chapter is the decision; the Fine-tuning track (FT1–FT6) is the hands-on how — LoRA/QLoRA, datasets, DPO, serving. Decide here, build there.

Exercise XT4.1 — Make the call

Context: The whole lesson is a decision, not a training run. Applying the decision and effort helpers to a real task you're tempted to tune is where it becomes concrete.

Your task: Take a real task you're tempted to fine-tune, run it through the decision helper, estimate the effort, and state what you'd try with prompt engineering plus RAG first.

Requirements:

Describe a genuine task and which of the four cases (facts/style/format/volume) it hits
Run the decision helper and record its recommendation
Estimate the effort with the cost helper for your example count
Write down a prompt + RAG plan you'd try before tuning
If none of the four cases apply, state plainly why you would not tune it — that's the lesson

💡 Hint: "No, not yet" is a valid and common outcome — the exercise is as much about ruling tuning out as choosing it.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · The decision: should you fine-tune? (offline)Beginner

Context: Fine-tuning is usually the wrong first move. It teaches behavior — style, format, lower per-call cost at scale — but it does not add fresh facts; that's what RAG is for.

Your task: Reproduce the lesson's decision helper should_finetune(...) that returns a recommendation based on whether you need facts, style, strict format, or high volume.

Requirements:

If the need is fresh facts, recommend RAG, not fine-tuning
If the need is style, strict format, or high volume, recommend trying prompt engineering first and tuning only if it plateaus
Otherwise recommend prompt engineering as likely sufficient
Stdlib only; take the four needs as boolean parameters
Show a facts case returning "use RAG" and a style/format/volume case returning "maybe"

💡 Hint: Order the checks so the facts case short-circuits first — it's the one hard "no" regardless of the other flags.

Show solution

Runnable, stdlib only:

def should_finetune(needs_facts, needs_style, needs_strict_format, high_volume):
    if needs_facts:
        return "No -- use RAG. Fine-tuning teaches behavior, not fresh facts."
    if needs_style or needs_strict_format or high_volume:
        return "Maybe -- try prompt engineering first; fine-tune if it plateaus."
    return "No -- prompt engineering is likely enough."

print(should_finetune(needs_facts=True,  needs_style=False,
                      needs_strict_format=False, high_volume=False))
print(should_finetune(needs_facts=False, needs_style=True,
                      needs_strict_format=True,  high_volume=True))

The usual answer is "no, not yet": prompt engineering is free and instant; reach for tuning only when it plateaus on style/format/cost — never to inject knowledge.

Exercise 2 · Estimate the true effort — data prep dominates (offline)Intermediate

Context: Teams underestimate tuning because they picture the GPU run, not the labeling. In reality data prep dominates the effort — the training run is the small part.

Your task: Reproduce finetune_effort(...) and report the total hours and the percentage that is data prep for 1000 examples at 3 hours per 100.

Requirements:

Compute data-prep hours as examples/100 × hours-per-100
Total = data-prep + GPU hours + eval hours
Return data-prep hours, total hours, and data-prep as a percentage of total
Stdlib only; round the reported numbers
Show that for the given inputs data prep is the large majority of the effort

💡 Hint: The interesting output is the percentage — it makes the point that labeling, not the GPU run, is where the time goes.

Show solution

Runnable, stdlib only:

def finetune_effort(examples_needed, hours_per_100_examples, gpu_hours, eval_hours):
    data_hours = examples_needed / 100 * hours_per_100_examples
    total = data_hours + gpu_hours + eval_hours
    return {"data_prep_hours": round(data_hours),
            "total_hours": round(total),
            "data_is_pct": round(data_hours / total * 100)}

print(finetune_effort(examples_needed=1000, hours_per_100_examples=3,
                      gpu_hours=4, eval_hours=8))
# {'data_prep_hours': 30, 'total_hours': 42, 'data_is_pct': 71}

71% of the effort is building the dataset; the GPU run is the small part. Teams underestimate tuning because they picture the training run, not the labeling.

Exercise 3 · Why LoRA is cheap: count trainable parameters (offline)Advanced

Context: LoRA is cheap because it freezes the base weights and trains small low-rank adapters instead of the full matrix. Counting the trainable parameters shows just how small that is.

Your task: Write lora_params(d_out, d_in, r) comparing full fine-tuning (d_out×d_in) against LoRA (r×(d_out+d_in)) and compute the ratio for a 4096×4096 layer at r=8.

Requirements:

Full fine-tuning parameter count is d_out × d_in
LoRA parameter count is r × (d_out + d_in) (the two adapter matrices)
Return full count, LoRA count, and LoRA as a percentage of full
Stdlib only; format the counts readably
Show LoRA is well under 1% of the layer's parameters at r=8
Connect the tiny count to fitting on modest GPUs and producing swappable adapter files

💡 Hint: The ratio collapses to roughly r×(d_out+d_in) over d_out×d_in — a fraction of a percent for a square layer at small rank.

Show solution

Runnable, stdlib only:

def lora_params(d_out, d_in, r):
    full = d_out * d_in
    lora = r * (d_out + d_in)          # A: (r x d_in) + B: (d_out x r)
    return full, lora, round(lora / full * 100, 3)

full, lora, pct = lora_params(4096, 4096, r=8)
print(f"full={full:,}  lora={lora:,}  lora is {pct}% of full")
# full=16,777,216  lora=65,536  lora is 0.391% of full

LoRA trains <0.5% of the parameters of that layer, which is why it fits on modest GPUs and produces tiny, swappable adapter files instead of a full model copy.

Exercise 4 · Measure tuned vs baseline before you trust it (offline)Expert

Context: Never ship a tuned model on vibes. A tuned model must beat the prompted baseline by a real margin to justify its ongoing retraining, maintenance, and serving cost.

Your task: Given a labeled eval set and predictions from a baseline and a tuned model, compute each model's accuracy and only recommend the tuned model if it clears the baseline by a set margin. Runnable offline.

Requirements:

Write an accuracy(preds, gold) over paired predictions and labels
Compute accuracy for both the baseline and the tuned predictions
Compare against a required margin (e.g. 5 points) before recommending the tuned model
Stdlib only; print both accuracies and the decision
Recommend keeping the cheaper baseline when the gain doesn't clear the margin

💡 Hint: Gate the recommendation on tuned - baseline >= MARGIN, not on the tuned model merely being higher.

Show solution

Runnable, stdlib only:

def accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

gold     = ["refund", "refund", "cancel", "cancel", "other", "refund"]
baseline = ["refund", "cancel", "cancel", "cancel", "refund", "refund"]
tuned    = ["refund", "refund", "cancel", "cancel", "other", "refund"]

a_base = accuracy(baseline, gold)     # 0.666...
a_tuned = accuracy(tuned, gold)       # 1.0
MARGIN = 0.05
print(round(a_base, 3), round(a_tuned, 3))
if a_tuned - a_base >= MARGIN:
    print("ship tuned: +{:.0%} over baseline".format(a_tuned - a_base))
else:
    print("keep baseline: gain not worth the maintenance cost")

A tuned model must clear a real margin (here 5 points) to justify its ongoing maintenance, retraining, and serving cost — otherwise the cheaper prompted baseline wins.

Exercise 5 · Format a supervised fine-tuning dataset + validate it (offline)Professional

Context: Most of fine-tuning is the data. Trainers consume clean chat-formatted JSONL, and duplicate or empty examples quietly leak into eval and inflate accuracy.

Your task: Write a validator to_sft_records(pairs) that turns raw (prompt, completion) pairs into chat-style JSONL records, rejecting empty examples and dropping duplicates. Runnable offline with stdlib json.

Requirements:

Raise on any pair with an empty prompt or completion
Deduplicate on the (stripped prompt, stripped completion) key
Emit each record as a chat message list with a user and an assistant turn
Strip whitespace on the stored content
Show a duplicate being dropped (three pairs in, two records out) and print one record as a JSONL line
Explain why dedupe and non-empty checks matter for honest eval metrics

💡 Hint: Track a seen set of the stripped pair so the second copy is skipped before it becomes a record.

Show solution

Runnable, stdlib only:

import json

def to_sft_records(pairs):
    records, seen = [], set()
    for prompt, completion in pairs:
        if not prompt.strip() or not completion.strip():
            raise ValueError("empty prompt or completion")
        key = (prompt.strip(), completion.strip())
        if key in seen:                      # dedupe: leaked dupes inflate metrics
            continue
        seen.add(key)
        records.append({"messages": [
            {"role": "user", "content": prompt.strip()},
            {"role": "assistant", "content": completion.strip()},
        ]})
    return records

pairs = [("Classify: love it", "positive"),
         ("Classify: love it", "positive"),   # duplicate -> dropped
         ("Classify: hate it", "negative")]
recs = to_sft_records(pairs)
print(len(recs))                              # 2
print(json.dumps(recs[0]))                    # one JSONL line

Deduping and non-empty checks matter: duplicate examples leak into eval and inflate accuracy; clean chat-formatted JSONL is exactly what SFT/LoRA trainers consume.

Exercise 6 · Run a real LoRA fine-tune (needs GPU + libs)Industry scenario

Context: Once you've decided tuning is worth it and prepared the data, the actual LoRA run is a standard PEFT skeleton. It needs a GPU and the libraries; the decision, data, and eval rungs above are the runnable offline part.

Your task: Write the training setup with transformers + peft to LoRA-tune a base model. Needs a GPU and the libraries (transformers, peft, datasets).

Requirements:

Load a base causal-LM model and tokenizer
Build a LoraConfig (rank, alpha, dropout, target modules, causal task type) and wrap the model with get_peft_model
Print trainable parameters to confirm well under 1% — matching the LoRA math
Load the prepared JSONL dataset and train with an SFT trainer and standard TrainingArguments
Save only the adapter (megabytes), not a full model copy
Frame it as: decide with the offline rungs first, then spend the GPU hours

💡 Hint: The saved artifact is just the small adapter, which is exactly why LoRA is cheap to store, swap, and serve.

Show solution

Needs a GPU and libraries (transformers, peft, trl/datasets). Standard PEFT-LoRA skeleton — no invented APIs:

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
from datasets import load_dataset

base = "meta-llama/Llama-3.2-1B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")

lora = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
                  target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(model, lora)
model.print_trainable_parameters()      # <1% trainable -- matches the LoRA math above

ds = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(
    model=model, tokenizer=tok, train_dataset=ds,
    args=TrainingArguments(output_dir="out", num_train_epochs=1,
                           per_device_train_batch_size=4, learning_rate=2e-4),
)
trainer.train()
model.save_pretrained("adapter")        # tiny adapter, not a full model copy

The saved artifact is just the LoRA adapter (megabytes), which is why it's cheap to store, swap, and serve. Decide with the offline rungs above; only then spend the GPU hours here.

✓ Checkpoint — you can move on when you can…

State the fine-tune vs RAG vs prompt decision.
Recognize the four cases tuning wins.
Estimate fine-tuning's true cost (data dominates).
Require a measured win; own the build decision.

Knowledge check check yourself

✓ Knowledge check

The lesson's decision framework climbs prompting → RAG → fine-tuning. Why is fine-tuning usually the wrong answer when you need to add knowledge?

Show answer

Fine-tuning changes the model's behavior (style, format, a narrow skill), not what it knows — it's a poor way to add facts. Missing knowledge (your docs, product details, fresh data) is a RAG problem: you retrieve the right text into the prompt. Prompting and RAG are cheaper and solve most problems, so fine-tuning is the last-resort rung you reach only when you need consistent behavior.

✓ Knowledge check

The cost lab shows data prep as ~71% of fine-tuning effort. Why does the lesson stress this, and what does it imply about when you can fine-tune well?

Show answer

Because fine-tuning feels like "just run a training command," but the training run is minutes while building a clean, labeled dataset is days — data prep is the dominant, easily-underestimated cost, and it climbs further as example count grows. The implication: if you can't commit to producing quality data, you can't fine-tune well, so do prompt+RAG instead.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · The most misunderstood question essential

2 · What each technique changes essential

3 · Intermediate — the four cases fine-tuning wins intermediate

4 · Advanced — the true cost of fine-tuning advanced

5 · Professional — measure tuned vs baseline professional

6 · Tech-lead — the build decision tech-lead

Exercise XT4.1 — Make the call

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Specialized topics