Dataset preparation
The part that decides success: sourcing, formatting to the chat template, cleaning, dedup, and leakage-free splits. A few hundred clean examples beat tens of thousands of noisy ones.
Learning objectives
- Explain why data quality decides fine-tuning success more than any hyperparameter.
- Format examples with the model's chat template.
- Clean, deduplicate, and split a dataset correctly.
- Estimate how much data you actually need.
code/ft2-data/ in the course, with a README. Run the scripts or copy the configs directly.Data is the whole game essential
The single biggest predictor of a good fine-tune is data quality, not learning rate or rank. A few hundred clean, consistent, on-distribution examples beat tens of thousands of noisy ones. Most failed fine-tunes are data problems wearing a hyperparameter costume.
This is the assembly line every training dataset goes through, left to right. Each box is a stage, and the arrows show the order — you can't skip ahead.
- Raw examples (left) is your source: whatever question→answer pairs you collected, in whatever messy shape they arrived.
- Format (chat template) turns each example into the exact role turns (user says X, assistant says Y) that the model expects — Lab FT2.1 does this.
- Clean + dedup is the quality stage: throw out broken examples and remove duplicates so the same thing isn't counted twice.
- Train / val split (right) divides the data into a part to learn from and a held-back part to test on — the no leakage label means those two parts must never share examples.
In short: Data flows one way: raw → format → clean → split. Getting this pipeline right matters more than any training setting — clean data is the whole game.
Format with the chat template essential
Instruction models expect a specific structure of role turns. Use the tokenizer's apply_chat_template so your data matches exactly what the model saw in training — a common silent bug is hand-formatting prompts that don't match the template.
format.pyfrom transformers import AutoTokenizer
from datasets import load_dataset
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
def to_text(ex):
msgs = [
{"role": "user", "content": ex["question"]},
{"role": "assistant", "content": ex["answer"]},
]
# exactly the format the model was trained on:
return {"text": tok.apply_chat_template(msgs, tokenize=False)}
ds = load_dataset("json", data_files="raw.jsonl", split="train").map(to_text)
print(ds[0]["text"][:300])
A model was trained to expect messages in one very specific layout. This script takes your plain question/answer pairs and rewrites them into that exact layout, so the model recognizes them.
AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")downloads the tokenizer for a specific model — it knows the precise chat format that model was trained with.to_text(ex)is a small function run on each example. It builds amsgslist of two turns: auserturn holdingex["question"]and anassistantturn holdingex["answer"].tok.apply_chat_template(msgs, tokenize=False)is the key line: it stamps those turns into the model's official format and returns it as text (tokenize=Falsemeans "give me readable text, not number IDs").load_dataset(...).map(to_text)loads every row fromraw.jsonland runsto_texton each one, adding the new formatted"text"field.
What the output means: print(ds[0]["text"][:300]) shows the first 300 characters of one formatted example — you'll see the special role markers the model expects wrapped around your Q and A.
Try this: Change [:300] to [:800] to see a whole formatted example. Compare it to your raw question/answer — those special tokens around the text are exactly what hand-formatting usually gets wrong.
Clean, dedup, split intermediate
A dataset-prep checklist
- Clean: remove truncated, empty, or off-task examples; fix inconsistent formatting.
- Dedup: near-duplicate examples inflate size without adding signal (and leak into val).
- Balance: if it's classification, check class balance; over-represented classes bias the model.
- Split: hold out a validation set BEFORE training — and make sure no example leaks across.
prep.pyfrom datasets import load_dataset
import hashlib
ds = load_dataset("json", data_files="raw.jsonl", split="train")
seen, keep = set(), []
for ex in ds:
h = hashlib.md5(ex["question"].strip().lower().encode()).hexdigest()
if h not in seen:
seen.add(h); keep.append(ex)
ds = ds.select([i for i, ex in enumerate(ds)
if hashlib.md5(ex["question"].strip().lower().encode()).hexdigest() in seen])
split = ds.train_test_split(test_size=0.1, seed=42) # val holdout, fixed seed
print("train:", len(split["train"]), "val:", len(split["test"]))
This script does two cleanup jobs that quietly decide whether your evaluation later tells the truth: it removes duplicate examples, then splits the data into a training set and a held-back validation set.
seenis a set of fingerprints andkeepcollects the unique rows. For each example,hashlib.md5(ex["question"].strip().lower().encode()).hexdigest()makes a short fingerprint of the question (lower-cased and trimmed) so near-identical questions get the same fingerprint.if h not in seen:keeps a row only the first time its fingerprint appears — that's how duplicates get dropped.ds.train_test_split(test_size=0.1, seed=42)splits off 10% as the validation ("test") set.seed=42fixes the randomness so you get the same split every run — that's what makes results reproducible.
What the output means: print("train:", ..., "val:", ...) prints the two sizes, e.g. train: 450 val: 50 — most rows for learning, a small slice held back to test on.
Try this: Change test_size=0.1 to 0.2 and re-run — the val count grows and the train count shrinks. Deduping before this split is what stops the same example landing in both sides and faking a good score.
How much data? advanced
For style/format tuning, hundreds of high-quality examples often suffice. For a narrow skill, low thousands. More matters less than consistent and on-distribution. Start small, measure (FT6), and add data only if evals say you need it.
Exercise FT2.1 — Build a clean dataset
Context: Nothing teaches data quality like running the whole pipeline on your own examples and eyeballing the output — if a formatted example looks wrong to you, it looks wrong to the model.
Your task: Take 200–500 examples for a task you care about, format them with the chat template, dedup, and split 90/10, then inspect ten random formatted examples by eye.
Requirements:
- Use 200–500 real examples
- Format with the model's chat template
- Dedup, then split 90/10 without leakage
- Manually inspect ten random formatted examples
- Fix anything that reads wrong before you'd train on it
💡 Hint: The eyeball pass is the point — run the full format → dedup → split pipeline from the earlier rungs, then read the output like a reviewer.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Instruction models expect role-tagged turns, and getting that message shape wrong is the single most common silent fine-tuning bug.
Your task: Without any libraries, transform raw question/answer pairs into the [{role, content}] message shape a tokenizer's chat template consumes.
Requirements:
- Map each
{question, answer}to a user turn then an assistant turn - Produce a two-element list of
{role, content}dicts - Keep it pure Python — no libraries
- Frame this as the pre-template step
apply_chat_templateconsumes
💡 Hint: This is deliberately upstream of the tokenizer — you're producing the message list, not the final templated string.
Show solution
The pre-template step, in plain Python:
raw = [
{"question": "Reset my password?", "answer": "Go to Settings > Security."},
{"question": "Enable 2FA?", "answer": "Same page, toggle 2FA."},
]
def to_messages(ex):
return [{"role": "user", "content": ex["question"]},
{"role": "assistant", "content": ex["answer"]}]
for ex in raw:
print(to_messages(ex))
# [{'role': 'user', 'content': 'Reset my password?'}, {'role': 'assistant', ...}]This message list is exactly what tokenizer.apply_chat_template turns into the model's training format. Matching that format is the single most common silent bug to avoid.
Context: Duplicates make the model overweight repeated examples, and most duplicates in scraped data differ only in casing or whitespace.
Your task: Write a deduper that drops exact duplicates and simple near-duplicates (same normalized text) and reports how many were removed.
Requirements:
- Normalize by lowercasing and collapsing runs of whitespace
- Key a
seenset on the normalized (question, answer) pair - Keep the first occurrence of each
- Report kept-of-total counts
- Target casing/whitespace variants, the common scraped-data case
💡 Hint: Normalize first, then dedup on the normalized key — the raw text still gets kept, you just compare on a canonical form.
Show solution
Normalize, then key on the normalized form:
import re
def norm(s):
return re.sub(r"\s+", " ", s.strip().lower())
def dedup(examples):
seen, kept = set(), []
for ex in examples:
key = (norm(ex["question"]), norm(ex["answer"]))
if key not in seen:
seen.add(key); kept.append(ex)
return kept
data = [
{"question":"Reset password?","answer":"Settings > Security."},
{"question":"reset password?","answer":"settings > security."}, # near-dup
{"question":"Enable 2FA?","answer":"Toggle 2FA."},
]
kept = dedup(data)
print(f"kept {len(kept)} of {len(data)} ({len(data)-len(kept)} removed)") # kept 2 of 3Casing/whitespace variants are the most common duplicates in scraped data. Removing them stops the model from silently over-training on the same content.
Context: A train/val split must never share examples, or your validation score measures memorization instead of generalization.
Your task: Write a deterministic split that also guards against leakage: an example whose question already appears in train cannot land in val.
Requirements:
- Assign each example deterministically by hashing its text into a 0–99 bucket
- Compare the bucket to
val_fracso runs are reproducible - Track seen questions and demote a would-be-val example to train if its question already appeared
- Verify the final train/val overlap is empty
💡 Hint: Hashing gives reproducibility; the extra seen-question guard is what turns a random split into a leakage-free one.
Show solution
Deterministic hashing split plus a leakage guard:
import hashlib
def bucket(text, val_frac=0.2):
h = int(hashlib.md5(text.encode()).hexdigest(), 16)
return "val" if (h % 100) < val_frac*100 else "train"
def split(examples, val_frac=0.2):
train, val, seen_q = [], [], set()
for ex in examples:
q = ex["question"].strip().lower()
where = bucket(q, val_frac)
if where == "val" and q in seen_q: # would leak -> keep in train
where = "train"
(val if where == "val" else train).append(ex)
seen_q.add(q)
return train, val
data = [{"question":f"q{i}","answer":f"a{i}"} for i in range(10)]
tr, va = split(data)
print(f"train={len(tr)} val={len(va)} overlap={set(e['question'] for e in tr) & set(e['question'] for e in va)}")
# overlap=set() -- no leakageHashing makes the split reproducible across runs; the guard ensures a val example never duplicates a train question, so your val score reflects generalization, not memorization.
Context: A few hundred clean examples often beat tens of thousands of noisy ones, so “we need more data” is frequently really “we need cleaner data”.
Your task: Write an estimator that recommends a dataset size from task narrowness and desired quality, and flags when the real fix is cleaning rather than collecting.
Requirements:
- Pick a base target by narrowness (very narrow < narrow < broad)
- Compute usable count as raw × (1 − dup rate) × (1 − noise rate)
- Return ENOUGH-but-clean vs collect-to-target
- Separate usable examples from raw counts to push cleaning first
💡 Hint: Discount the raw count by the dup and noise rates before comparing to the target — often the gap closes once you count only usable rows.
Show solution
A rule-of-thumb estimator that favors quality:
def recommend_size(task_narrowness, current_n, current_dup_rate, current_noise_rate):
base = {"very narrow": 300, "narrow": 800, "broad": 3000}[task_narrowness]
clean_n = current_n * (1 - current_dup_rate) * (1 - current_noise_rate)
if clean_n >= base:
return f"ENOUGH clean examples (~{int(clean_n)} of {current_n}); CLEAN, don't collect"
return f"collect to ~{base} clean; you have ~{int(clean_n)} usable now"
print(recommend_size("narrow", current_n=5000, current_dup_rate=0.3, current_noise_rate=0.4))
# ENOUGH clean examples (~2100 of 5000); CLEAN, don't collect
print(recommend_size("narrow", current_n=400, current_dup_rate=0.05, current_noise_rate=0.05))
# collect to ~800 clean; you have ~361 usable nowOften you already have enough raw rows — the fix is dedup + noise removal, not more collection. The estimator separates "usable" from "raw" so you clean before you scrape.
Context: The format step must match exactly what the model saw in pretraining, which is why you never hand-write special tokens — you call the model's own chat template.
Your task: Show the lesson's real format step using transformers/datasets so it matches the model's template exactly, labelling it as needing the libraries.
Requirements:
- Load the tokenizer with
AutoTokenizer.from_pretrained - Build user/assistant messages and format via
apply_chat_template(msgs, tokenize=False) - Load JSONL with
load_datasetand.mapthe formatter - Never hand-format special tokens
- Note the code needs
pip install transformers datasets
💡 Hint: The tokenizer already carries the model's template — let apply_chat_template emit the tokens so you can't get them wrong.
Show solution
The real formatting pass — needs pip install transformers datasets:
from transformers import AutoTokenizer
from datasets import load_dataset
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
def to_text(ex):
msgs = [{"role": "user", "content": ex["question"]},
{"role": "assistant", "content": ex["answer"]}]
# apply_chat_template emits EXACTLY the format the model was trained on:
return {"text": tok.apply_chat_template(msgs, tokenize=False)}
ds = load_dataset("json", data_files="raw.jsonl", split="train").map(to_text)
print(ds[0]["text"][:300])Never hand-format the special tokens — apply_chat_template uses the model's own template, which is the difference between a working tune and a silently mistrained one.
Context: Most failed fine-tunes are data problems in disguise, so a cheap pre-flight audit pays for itself before you commit any GPU hours.
Your task: Before an expensive run, write a data audit that reports count, exact-dup rate, empty/short answers, and label imbalance, then gates the run on quality thresholds and prints PASS/FAIL with reasons.
Requirements:
- Compute example count and dup rate from a normalized (question, answer) set
- Count short answers (e.g. fewer than two words)
- Assemble a report dict of the rates
- Gate: FAIL on high dup rate, high short-answer rate, or too few examples
- Print PASS/FAIL joined with the specific failing reasons
💡 Hint: Each threshold breach becomes a reason string; PASS is simply the case where the reason list came back empty.
Show solution
A pre-flight audit that gates the training run:
import re
def audit(examples):
n = len(examples)
norm = lambda s: re.sub(r"\s+"," ",s.strip().lower())
dups = n - len({(norm(e['question']), norm(e['answer'])) for e in examples})
short = sum(1 for e in examples if len(e['answer'].split()) < 2)
report = {"n": n, "dup_rate": round(dups/n,2), "short_rate": round(short/n,2)}
problems = []
if report["dup_rate"] > 0.10: problems.append("too many duplicates")
if report["short_rate"] > 0.05: problems.append("too many empty/short answers")
if n < 200: problems.append("dataset likely too small")
report["gate"] = "PASS" if not problems else "FAIL: " + "; ".join(problems)
return report
data = [{"question":f"q{i%40}","answer":"ok"} for i in range(300)]
print(audit(data)) # high dup_rate + short answers -> FAILMost failed fine-tunes are data problems in disguise. A cheap audit that blocks the run on dup/short/size thresholds saves the GPU hours a bad dataset would waste.
✓ Checkpoint — you can move on when you can…
- Explain why data quality outweighs hyperparameters.
- Format examples with
apply_chat_template. - Clean, dedup, and split without leakage.
- Estimate the data volume a task needs.
Knowledge check check yourself
Why must you format training examples with the tokenizer's apply_chat_template rather than hand-writing the prompt structure?
Show answer
Why must deduplication happen before the train/validation split, and what goes wrong if it doesn't?