Evaluate & serve the model
A fine-tune is done when you've proven it helped — target up, general capability intact. Then serving it is just serving a model: the whole inference track applies.
- a GPU +
pip install vllm(serving engine)
Learning objectives
- Evaluate a tuned model: did it improve the target without regressing?
- Guard against catastrophic forgetting and overfitting.
- Serve the tuned model (merged or adapter) via the inference stack.
- Close the loop: eval-gated promotion, like the rest of the course.
code/ft6-eval-serve/ in the course, with a README. Run the scripts or copy the configs directly.Did it actually help? advanced
A fine-tune isn't done when training finishes — it's done when you've proven it helped. Run the evals from Ch 5 on both the base and tuned model: the target metric should rise and general capability shouldn't drop. Tuning that improves one thing while quietly breaking others is the most common failure.
This flow is the quality gate a fine-tune must pass before you ship it. The rule: prove the tuned model is better and not secretly worse elsewhere. Read left to right.
- Base model (left) is your baseline — the original, untuned model you're trying to beat.
- Tuned model is the candidate — the one you just trained and hope is an improvement.
- Eval both on same set is the fair test: run both models on the identical questions and measure two things — the
targetskill you tuned for and anyregression(general ability slipping). - Promote only if better (right) is the gate: you ship the tuned model only if it wins on the target
no regression— better at the job without getting dumber overall.
In short: Never ship a fine-tune just because training finished. Ship it only when the same-set comparison proves it beats the base and didn't break general ability.
Serve it expert
Serving a tuned model is just serving a model — the whole Inference track applies. Merge the adapter (FT4) and serve on vLLM, or keep adapters separate and hot-swap. Your app code doesn't change; only the model does.
eval_serve.py# 1) Evaluate both on the SAME set (reuse your Ch 5 harness)
def load_task_evals(): ... # your task eval cases
def load_general_evals(): ... # a general-capability eval set
def run_evals(model_path, eval_cases): ... # generate + score, return a float
cases, general_cases = load_task_evals(), load_general_evals()
def score(model_path, eval_cases):
# generate answers, compute your metric (accuracy / format-valid % / judge), average:
return run_evals(model_path, eval_cases)
base_score = score("mistralai/Mistral-7B-Instruct-v0.3", cases)
tuned_score = score("out/merged-model", cases)
base_general = score("mistralai/Mistral-7B-Instruct-v0.3", general_cases)
tuned_general = score("out/merged-model", general_cases)
gen_regress = tuned_general < base_general # did general ability drop?
print(f"target: {base_score:.2f} -> {tuned_score:.2f} | regressed: {gen_regress}")
# 2) Only if better AND no regression, serve it (IC6):
# python -m vllm.entrypoints.openai.api_server --model out/merged-model
This script is the fair before/after test. It scores the base model and the tuned model on the same questions — twice: once on your task, once on general ability — then decides whether tuning actually helped. The ... bodies are placeholders you fill in with your Chapter 5 eval code.
- The three
deflines at the top are stubs:load_task_evalsgives your task-specific test cases,load_general_evalsgives a broad set, andrun_evalsgenerates answers and returns a single score. score(model_path, eval_cases)is a small helper that runs a model over a set of cases and returns one average number — higher is better.- The four
score(...)calls produce the key numbers:base_scorevstuned_scoreon the task, andbase_generalvstuned_generalon general ability — always comparing the two models on the identical cases. gen_regress = tuned_general < base_generalis the safety check: it'sTrueif the tuned model got worse at general tasks (catastrophic forgetting).- The commented
vllmline at the bottom is how you'd serve the model — but only after the numbers say it's better with no regression.
What the output means: The print shows something like target: 0.71 -> 0.86 | regressed: False: the task score rose from 0.71 to 0.86 and general ability did not drop — a clean win worth shipping.
Try this: Imagine the line printed regressed: True. That means your tune improved the task but broke general ability — the fix is fewer epochs, a lower learning rate, or more varied data (FT2), not shipping it.
Exercise FT6.1 — Prove your tune
Context: The whole track lands here: a tune earns production only if it wins on the task without regressing generally — and if it regressed, the fix is to diagnose and re-tune, not to ship.
Your task: Evaluate your FT3/FT5 model against the base on a task metric AND a general set, and only if it wins on the task without regressing generally, merge and serve it on vLLM.
Requirements:
- Score base vs tuned on both a task set and a general set
- Promote only on a task win with no general regression
- On a pass, merge and serve on vLLM
- On a regression, diagnose (too many epochs? too narrow data?) and re-tune
- Treat the two-metric gate as the promotion contract
💡 Hint: This is the FT6 gates applied to your own model — a regression sends you back to the FT2/FT3 knobs, not forward to serving.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: A tune is only done once you've proven it helped, and a tuned score means nothing without the base model's score on the exact same questions.
Your task: Given per-question scores for base and tuned on the same questions, compute each model's accuracy and the improvement.
Requirements:
- Accuracy = mean of the 0/1 correctness list
- Score base and tuned on identical questions
- Print both accuracies and the delta as percentages
- Reinforce that a raw tuned score is meaningless without the same-set base comparison
💡 Hint: Both lists cover the same questions, so a plain mean of each and their difference is the entire answer.
Show solution
The fair same-set comparison, in plain Python:
base = [0,1,1,0,1,0,1,1,0,1] # 1 = correct on that question
tuned = [1,1,1,0,1,1,1,1,0,1]
def acc(scores): return sum(scores)/len(scores)
print(f"base : {acc(base):.0%}") # 60%
print(f"tuned: {acc(tuned):.0%}") # 80%
print(f"delta: {acc(tuned)-acc(base):+.0%}") # +20%Always evaluate both models on the identical questions — a raw tuned score means nothing without the base to beat.
Context: Tuning too hard can lift the target while regressing general ability (catastrophic forgetting), so promotion needs a two-metric gate, not a single number.
Your task: Given target and general scores for base and tuned, gate promotion: promote only if the target rises AND general ability doesn't drop.
Requirements:
- Compute target-improved and general-regressed booleans
- Allow a small tolerance band on the general drop
- Return PROMOTE, REJECT–forgetting, or REJECT–no-gain
- Catch forgetting rather than trusting the target metric alone
💡 Hint: Two conditions must both hold; the general-drop check needs a tolerance so noise doesn't look like forgetting.
Show solution
Check both metrics, not just the one you tuned for:
def promote(base_target, tuned_target, base_general, tuned_general, tol=0.02):
improved = tuned_target > base_target
regressed = (base_general - tuned_general) > tol
if improved and not regressed:
return "PROMOTE — target up, general held"
if improved and regressed:
return "REJECT — target up but GENERAL REGRESSED (catastrophic forgetting)"
return "REJECT — target did not improve"
print(promote(0.60, 0.82, 0.75, 0.74)) # PROMOTE (general within tol)
print(promote(0.60, 0.85, 0.75, 0.68)) # REJECT — forgettingA tune that wins the target but drops general ability is the most common silent failure. Keeping a general eval set alongside the task set is what catches it.
Context: Improvement on the training distribution can just be memorization; a big train-vs-held-out gap is the overfitting signal.
Your task: Compare tuned accuracy on train-like vs held-out questions and flag overfitting when the gap is large.
Requirements:
- Compute
gap = train_acc − heldout_acc - Flag OVERFIT when the gap exceeds a tolerance, else OK
- Format the gap as a percentage
- Point to FT2 remedies (more diverse data, fewer epochs, lower LR)
💡 Hint: It's a single subtraction against a threshold — a wide gap means the model learned the train set, not the task.
Show solution
Quantify the train-vs-held-out gap:
def overfit_check(train_acc, heldout_acc, gap_tol=0.10):
gap = train_acc - heldout_acc
if gap > gap_tol:
return f"OVERFIT risk — gap {gap:.0%} (memorizing; add data / fewer epochs)"
return f"OK — gap {gap:.0%} within tolerance (generalizing)"
print(overfit_check(0.95, 0.72)) # OVERFIT risk — gap 23%
print(overfit_check(0.83, 0.80)) # OK — gap 3%A model that aces training-like questions but stumbles on held-out ones has memorized, not learned. The fix is the FT2 levers: more diverse data, fewer epochs, lower learning rate.
Context: The course closes every loop with a scorecard; here you combine target improvement, the general-regression guard, and the overfit gap into one pass/fail verdict.
Your task: Combine the three prior gates into a single promotion scorecard with reasons.
Requirements:
- Check target improved, no general regression (within tolerance), not overfit (gap within tolerance)
- Collect the failing checks
- Return PROMOTE only if none failed, else HOLD with reasons
- Include the per-check PASS/FAIL map
💡 Hint: Reuse the three earlier rung checks verbatim; PROMOTE is just the case where the failed-check list is empty.
Show solution
One gate that combines every check:
def scorecard(m):
checks = {
"target improved": m["tuned_target"] > m["base_target"],
"no gen regression": (m["base_general"] - m["tuned_general"]) <= 0.02,
"not overfit": (m["train_acc"] - m["heldout_acc"]) <= 0.10,
}
failed = [k for k,ok in checks.items() if not ok]
return ("PROMOTE" if not failed else "HOLD: " + ", ".join(failed)), checks
metrics = dict(base_target=0.60, tuned_target=0.81, base_general=0.75,
tuned_general=0.74, train_acc=0.88, heldout_acc=0.80)
verdict, checks = scorecard(metrics)
print(verdict) # PROMOTE
for k,v in checks.items(): print(f" {k}: {'PASS' if v else 'FAIL'}")Eval-gated promotion turns "training finished" into "proven better and not secretly worse". Only a candidate that passes all three checks ships.
Context: Serving a tuned model is identical to serving any model — only the path changes — so the one genuinely new step is the eval gate before you serve.
Your task: Show the real close-the-loop step: eval base vs tuned on the same harness, then serve the merged model on vLLM, using only real APIs and labelling it as needing a GPU + libraries.
Requirements:
- Define an
evaluate(model_path, questions)that returns an accuracy float - Eval the base and the merged (FT4) model on the same question set
- Gate with an assertion that tuned beats base
- Serve via a
vllm serveline only after the gate passes - Note the libs:
transformers vllm
💡 Hint: Serving is unchanged from any model — the merged path goes straight to vllm serve; the assertion is the only new gate.
Show solution
The real eval-then-serve loop — needs a GPU + pip install transformers vllm:
# 1) Evaluate BOTH on the SAME set (reuse your Ch 5 harness)
def evaluate(model_path, questions):
# ... load model, run generate(), score against references ...
return accuracy # float
base_acc = evaluate("mistralai/Mistral-7B-Instruct-v0.3", QS)
tuned_acc = evaluate("out/merged-model", QS) # merged adapter (FT4)
assert tuned_acc > base_acc, "do NOT promote a non-improvement"
# 2) Serve the tuned model — identical to serving any model (IC6)
# vllm serve out/merged-model --port 8000
# app code is unchanged; only the model path differsServing a tuned model is just serving a model — the whole Inference track applies. The only new step is the eval gate that must pass before promotion.
Context: As a lead you encode the promotion policy once as thresholds, and CI decides whether each nightly tune auto-promotes to staging — policy as code.
Your task: Define the CI gate that decides whether a nightly fine-tune auto-promotes: encode thresholds, a regression guard, and a rollback trigger, then print the decision for a candidate run.
Requirements:
- Encode thresholds (min target gain, max general drop, max overfit gap)
- Compute each metric for the candidate run
- Append a human-readable reason per threshold breach
- Return AUTO-PROMOTE when clean, else BLOCK + alert with reasons
- Make the thresholds a single policy object
💡 Hint: It's the scorecard from rung 4 with the tolerances pulled out into a policy dict — CI reads the reasons list to decide promote vs block.
Show solution
The promotion policy a CI job would enforce:
POLICY = {"min_target_gain": 0.03, "max_gen_drop": 0.02, "max_overfit_gap": 0.10}
def ci_gate(run, policy=POLICY):
gain = run["tuned_target"] - run["base_target"]
gen_drop = run["base_general"] - run["tuned_general"]
gap = run["train_acc"] - run["heldout_acc"]
reasons = []
if gain < policy["min_target_gain"]: reasons.append(f"gain {gain:+.0%} too small")
if gen_drop > policy["max_gen_drop"]: reasons.append(f"general dropped {gen_drop:.0%}")
if gap > policy["max_overfit_gap"]: reasons.append(f"overfit gap {gap:.0%}")
return ("AUTO-PROMOTE to staging" if not reasons
else "BLOCK + alert: " + "; ".join(reasons))
run = dict(base_target=0.60, tuned_target=0.66, base_general=0.75,
tuned_general=0.745, train_acc=0.84, heldout_acc=0.78)
print(ci_gate(run)) # AUTO-PROMOTE to stagingA lead owns the thresholds, not the individual runs: encode the minimum gain, the regression guard, and the overfit ceiling once, and let CI promote or block every nightly tune automatically.
✓ Checkpoint — you can move on when you can…
- Evaluate tuned vs base on the same set.
- Detect and prevent catastrophic forgetting.
- Serve the tuned model via the inference stack.
- Apply eval-gated promotion to a fine-tune.
Knowledge check check yourself
Why must you evaluate a fine-tuned model on both a task set and a general-capability set before shipping it?
Show answer
If your tuned model shows regressed: True (general ability dropped), what are the fixes — and what should you NOT do?