Dataset preparation

Data is the whole game essential

Here's the scene that plays out over and over. A team spends a week arguing about learning rate, rank, and epochs, kicks off a costly GPU run, and the tuned model comes back worse than the base one — more confident, more fluent, and more confidently wrong. They tweak the hyperparameters and run again. Same result. What they never opened was the dataset, where a quarter of the "answers" were truncated, a third were near-duplicates, and a handful taught the model the exact opposite of what they wanted. The knobs were never the problem. The data was.

The oldest law in machine learning is garbage in, garbage out, and fine-tuning obeys it more strictly than almost anything else you'll do. Think of it like teaching an apprentice by example: the apprentice doesn't argue with your demonstrations, doesn't sanity-check them, doesn't know which ones were sloppy. They copy exactly what you show them, warts and all. A fine-tune is that apprentice with a perfect memory and no judgement — every bad example you feed it is a lesson it dutifully learns. Show it ten contradictory answers to the same question and you don't get an average; you get a model that has learned to be inconsistent.

The single biggest predictor of a good fine-tune is data quality, not learning rate or rank. A few hundred clean, consistent, on-distribution examples beat tens of thousands of noisy ones. Most failed fine-tunes are data problems wearing a hyperparameter costume.

The most common mistake is treating data prep as the boring chore you rush through to get to the "real" work of training. It's backwards: the training run is the cheap, automatable part — a script you launch and walk away from — while the dataset is the one thing only you can get right, because only you know what "good" looks like for your task. Every hour you spend reading your own examples and fixing what's wrong buys you more than any hyperparameter sweep. That's why this whole chapter is a pipeline: source it, format it to the model's template, clean and dedup it, then split it so your evaluation stays honest. Get that order right and the model almost tunes itself; get it wrong and no learning rate will save you. The diagram below is the assembly line every serious dataset walks through, in the one order that works.

Format with the chat template essential

This is the section where the most fine-tunes quietly die — and the failure leaves no error message. Your data is clean, your training loss drops beautifully, the run completes, and the model comes back subtly broken: rambling, ignoring instructions, or refusing to stop talking. Nine times out of ten the cause is here, in formatting, because you hand-built the prompt structure and got a character wrong that you never even saw.

To understand why the exact format matters so much, picture how an instruction model actually learned to behave. It wasn't shown plain text — it was shown millions of conversations wrapped in a rigid chat template: invisible special tokens that mark "here begins a user turn," "here begins the assistant's reply," "the turn ends here." Those markers are the model's grammar of conversation. It learned that right after the assistant-start token, it's its turn to speak; it learned to stop at the end token. The template is the punctuation of a language the model speaks fluently — and if you feed it sentences with the wrong punctuation, it gets confused in ways that look like the model being dumb but are really you speaking broken grammar.

The common mistake — and it's seductive. You look at what the template produces, see something like [INST] question [/INST] answer, and think "I can just write that myself with an f-string." Then you get a space wrong, or use the wrong bracket style, or forget the leading beginning-of-sequence token, or use a different model's format entirely. Every one of those is a silent mismatch: the tokens you emit don't line up with the tokens the model was trained on, so it's now learning from examples written in a dialect it half-recognizes. Nothing crashes. The loss still goes down. But you've taught it noise. The analogy: it's like training a translator on sentences where the quotation marks are randomly wrong — they'll learn something, but not the clean signal you meant.

The fix is refreshingly simple: never write the format by hand. Every tokenizer ships with the exact template its model was trained on, and one function call, apply_chat_template, stamps your messages into it perfectly — right tokens, right spacing, right order, every time. You hand it a plain list of role/content turns and it hands back the model's native format. Below, the lab does exactly that; the rule to burn in is that the tokenizer already knows the format, so your job is only to give it clean turns and let it do the stamping.

Instruction models expect a specific structure of role turns. Use the tokenizer's apply_chat_template so your data matches exactly what the model saw in training — a common silent bug is hand-formatting prompts that don't match the template.

format.pyfrom transformers import AutoTokenizer
from datasets import load_dataset

tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

def to_text(ex):
    msgs = [
        {"role": "user", "content": ex["question"]},
        {"role": "assistant", "content": ex["answer"]},
    ]
    # exactly the format the model was trained on:
    return {"text": tok.apply_chat_template(msgs, tokenize=False)}

ds = load_dataset("json", data_files="raw.jsonl", split="train").map(to_text)
print(ds[0]["text"][:300])

Clean, dedup, split intermediate

Formatting made your data the right shape; this step decides whether it's telling the truth. A dataset can be perfectly formatted and still poison a run — full of half-finished answers, the same example copied forty times, or a validation set that secretly overlaps with training. These aren't exotic edge cases; they're what real collected data looks like before anyone cleans it. This section is the quality gate, and skipping it is the difference between an eval score you can trust and one that lies to your face.

Take each job in turn, because each fixes a distinct failure. Cleaning is throwing out the examples that teach the wrong thing — a truncated answer teaches the model to stop mid-sentence; an off-task row teaches it to wander. Remember the apprentice: it copies whatever you show it, so a broken demonstration is a broken lesson, not a harmless one it ignores. Dedup matters because duplicates aren't free — they're a thumb on the scale. If one question appears fifty times, the model treats that pattern as fifty times more important than a question it saw once, quietly skewing what it learns toward whatever your scraper happened to over-collect. Most duplicates in real data aren't identical either; they differ only in casing or stray whitespace, which is why you compare a normalized form, not the raw string.

The most dangerous mistake lives in the split, and it has a name: leakage. You hold back a slice of data to measure how well the model generalizes to things it hasn't seen — that's the entire point of a validation set. But if a training example (or a near-duplicate of one) sneaks into that held-back slice, you're no longer measuring generalization; you're measuring memorization. The model saw the answer during training, so of course it aces the "test." Your eval lights up green, you ship with confidence, and production falls flat because the real world doesn't hand you questions you've already memorized. That's why the order is non-negotiable: dedup first, then split — and split on a fixed seed so the exact same partition happens every run and your results are reproducible. Do it the other way around and a duplicate can land on both sides of the wall, and the leak is invisible until it costs you. The checklist below is that order made concrete; treat it as a sequence, not a menu.

A dataset-prep checklist

1. Clean: remove truncated, empty, or off-task examples; fix inconsistent formatting.
2. Dedup: near-duplicate examples inflate size without adding signal (and leak into val).
3. Balance: if it's classification, check class balance; over-represented classes bias the model.
4. Split: hold out a validation set BEFORE training — and make sure no example leaks across.

prep.pyfrom datasets import load_dataset
import hashlib

ds = load_dataset("json", data_files="raw.jsonl", split="train")

seen, keep = set(), []
for ex in ds:
    h = hashlib.md5(ex["question"].strip().lower().encode()).hexdigest()
    if h not in seen:
        seen.add(h); keep.append(ex)
ds = ds.select([i for i, ex in enumerate(ds)
                if hashlib.md5(ex["question"].strip().lower().encode()).hexdigest() in seen])

split = ds.train_test_split(test_size=0.1, seed=42)   # val holdout, fixed seed
print("train:", len(split["train"]), "val:", len(split["test"]))

How much data? advanced

"How much data do I need?" is the first question everyone asks, and it's almost always the wrong one. The instinct — inherited from the old days of training models from scratch, where more data was nearly always better — is to assume you need tens of thousands of examples and to feel stuck until you have them. For fine-tuning an already-capable model, that instinct will send you collecting for weeks when you could have shipped in an afternoon. The base model already knows the language, the facts, the reasoning; you are not teaching it to think. You're teaching it a style, a format, or a narrow skill — and nudging a model that already knows almost everything takes far less than you'd guess.

Here's the mental model that reframes it. Fine-tuning isn't filling an empty bucket; it's showing a fluent expert how you'd like them to behave in one specific situation. To get a brilliant colleague to always answer support tickets in your house tone, you don't hand them a library — you show them a few dozen good examples and they get it. The same holds here: for style or format tuning, hundreds of clean examples often move the needle decisively; for a genuinely narrow new skill, low thousands. What matters far more than raw count is that the examples are consistent (they don't contradict each other) and on-distribution (they look like the inputs you'll actually see in production). Ten thousand inconsistent examples teach the model to be inconsistent; three hundred coherent ones teach it one clear behavior.

The mistake this section exists to prevent is "we need more data" when the honest diagnosis is "we need cleaner data." When a first run underperforms, adding rows is the tempting reflex — but if a third of your set is duplicates and another third is noise, doubling the pile just doubles the mess. Count only your usable examples (raw count discounted by your duplicate and noise rates) and the gap to "enough" often closes without collecting a single new row. So the discipline is the opposite of the reflex: start small, tune, then measure against a real held-out eval (that's FT6), and let the evals — not your anxiety — tell you whether the bottleneck is quantity or quality. Add data only when the measurement demands it. Below is the rule of thumb, but keep the reframe in front of it: consistency beats volume, and cleaning beats collecting.

For style/format tuning, hundreds of high-quality examples often suffice. For a narrow skill, low thousands. More matters less than consistent and on-distribution. Start small, measure (FT6), and add data only if evals say you need it.

raw = [
    {"question": "Reset my password?", "answer": "Go to Settings > Security."},
    {"question": "Enable 2FA?",         "answer": "Same page, toggle 2FA."},
]
def to_messages(ex):
    return [{"role": "user",      "content": ex["question"]},
            {"role": "assistant", "content": ex["answer"]}]

for ex in raw:
    print(to_messages(ex))
# [{'role': 'user', 'content': 'Reset my password?'}, {'role': 'assistant', ...}]

import re
def norm(s):
    return re.sub(r"\s+", " ", s.strip().lower())

def dedup(examples):
    seen, kept = set(), []
    for ex in examples:
        key = (norm(ex["question"]), norm(ex["answer"]))
        if key not in seen:
            seen.add(key); kept.append(ex)
    return kept

data = [
    {"question":"Reset password?","answer":"Settings > Security."},
    {"question":"reset password?","answer":"settings > security."},  # near-dup
    {"question":"Enable 2FA?","answer":"Toggle 2FA."},
]
kept = dedup(data)
print(f"kept {len(kept)} of {len(data)} ({len(data)-len(kept)} removed)")  # kept 2 of 3

import hashlib
def bucket(text, val_frac=0.2):
    h = int(hashlib.md5(text.encode()).hexdigest(), 16)
    return "val" if (h % 100) < val_frac*100 else "train"

def split(examples, val_frac=0.2):
    train, val, seen_q = [], [], set()
    for ex in examples:
        q = ex["question"].strip().lower()
        where = bucket(q, val_frac)
        if where == "val" and q in seen_q:   # would leak -> keep in train
            where = "train"
        (val if where == "val" else train).append(ex)
        seen_q.add(q)
    return train, val

data = [{"question":f"q{i}","answer":f"a{i}"} for i in range(10)]
tr, va = split(data)
print(f"train={len(tr)} val={len(va)}  overlap={set(e['question'] for e in tr) & set(e['question'] for e in va)}")
# overlap=set()  -- no leakage

def recommend_size(task_narrowness, current_n, current_dup_rate, current_noise_rate):
    base = {"very narrow": 300, "narrow": 800, "broad": 3000}[task_narrowness]
    clean_n = current_n * (1 - current_dup_rate) * (1 - current_noise_rate)
    if clean_n >= base:
        return f"ENOUGH clean examples (~{int(clean_n)} of {current_n}); CLEAN, don't collect"
    return f"collect to ~{base} clean; you have ~{int(clean_n)} usable now"

print(recommend_size("narrow", current_n=5000, current_dup_rate=0.3, current_noise_rate=0.4))
# ENOUGH clean examples (~2100 of 5000); CLEAN, don't collect
print(recommend_size("narrow", current_n=400, current_dup_rate=0.05, current_noise_rate=0.05))
# collect to ~800 clean; you have ~361 usable now

from transformers import AutoTokenizer
from datasets import load_dataset

tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

def to_text(ex):
    msgs = [{"role": "user",      "content": ex["question"]},
            {"role": "assistant", "content": ex["answer"]}]
    # apply_chat_template emits EXACTLY the format the model was trained on:
    return {"text": tok.apply_chat_template(msgs, tokenize=False)}

ds = load_dataset("json", data_files="raw.jsonl", split="train").map(to_text)
print(ds[0]["text"][:300])

import re
def audit(examples):
    n = len(examples)
    norm = lambda s: re.sub(r"\s+"," ",s.strip().lower())
    dups = n - len({(norm(e['question']), norm(e['answer'])) for e in examples})
    short = sum(1 for e in examples if len(e['answer'].split()) < 2)
    report = {"n": n, "dup_rate": round(dups/n,2), "short_rate": round(short/n,2)}
    problems = []
    if report["dup_rate"] > 0.10: problems.append("too many duplicates")
    if report["short_rate"] > 0.05: problems.append("too many empty/short answers")
    if n < 200: problems.append("dataset likely too small")
    report["gate"] = "PASS" if not problems else "FAIL: " + "; ".join(problems)
    return report

data = [{"question":f"q{i%40}","answer":"ok"} for i in range(300)]
print(audit(data))  # high dup_rate + short answers -> FAIL

Knowledge check check yourself