Prompt & Context Engineering · Chapter E3

Prompt Optimization, Evaluation & DSPy

Hand-tuning prompts by intuition doesn't scale and doesn't survive model upgrades. This chapter is the engineering discipline around prompts: measure before you tune, optimize against data, and — with DSPy — stop writing prompt strings by hand and let a framework compile them from examples and a metric.

⏱️ ~1.5 hours🧪 3 labs🎯 Beginner→Tech-lead

Learning objectives

Explain why hand-tuning prompts doesn't scale.
Understand DSPy: programming, not prompting.
Define signatures and let an optimizer tune the prompt.
Adopt automatic optimization responsibly on a team.

▶ Runnable companionEvery code block in this lesson is also a standalone file under code/pe3-prompt-optimization-dspy/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

1 · The hand-tuning ceiling essential

Tweaking prompt wording by hand is slow, brittle, and doesn't transfer across models. As systems grow (multiple prompts, model upgrades), manual prompt maintenance becomes a bottleneck. Automatic prompt optimization treats the prompt as something you compile, not craft.

2 · DSPy — program, don't prompt essential

DSPy lets you declare what you want (a signature: inputs → outputs) and an optimizer figures out the prompt (instructions + few-shot examples) that maximizes a metric on your data. You program the pipeline; DSPy tunes the strings.

🗺️ How to read this diagram

This picture is the whole idea of DSPy in one line. Normally you sit and write the prompt string by hand. DSPy flips that: you describe the job and hand it examples, and a program writes and tunes the prompt for you. Read the four boxes left to right — they are the assembly line.

Box 1 — Signature (in→out): you declare only what goes in and what should come out (e.g. a ticket goes in, an urgency comes out). The caption says "what, not how" — you do not write any prompt wording here.
Box 2 — Your data + metric: you supply a few labeled examples (inputs with their correct answers) and a metric — a way to score how good an answer is. This is the fuel; without it there is nothing to aim at.
Box 3 — Optimizer: the engine that searches. It tries many candidate prompts (different instructions, different example picks), scores each with your metric, and hunts for the winner.
Box 4 — Tuned prompt: the output — a finished, compiled prompt that scored best. The arrows (→) mean "feeds into": signature feeds data+metric, which feeds the optimizer, which produces the tuned prompt.

In short: You write the two left boxes (task + data/metric); the machine produces the two right boxes (the search and the final prompt). That is what "program, don't prompt" means.

3 · A signature intermediate

A signature is a typed declaration of a step. This is illustrative DSPy shape (runs with pip install dspy).

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Python · a DSPy signature

signature.pyimport dspy

class Classify(dspy.Signature):
    """Classify a support ticket's urgency."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="one of: low, medium, high")

classify = dspy.Predict(Classify)
# result = classify(ticket="Prod is down!")  ->  result.urgency == "high"
# You never wrote the prompt — DSPy generates + optimizes it.

▶ How this works

This is a DSPy signature — a typed declaration of one step of work. It uses the external dspy library (install with pip install dspy), so it's illustrative here, but the shape is the real point: you spell out inputs and outputs, and you write no prompt text at all. Compare it to the diagram's first box — this is that box, in code.

import dspy pulls in the framework. class Classify(dspy.Signature): declares one task by subclassing dspy.Signature. The name Classify and the docstring ("""Classify a support ticket's urgency.""") tell DSPy the goal in plain words.
ticket: str = dspy.InputField() says "this step takes one input called ticket, and it's text." urgency: str = dspy.OutputField(desc="one of: low, medium, high") says "it returns one output called urgency," and the desc= hint constrains it to three allowed values.
classify = dspy.Predict(Classify) turns that declaration into something callable. dspy.Predict is what actually builds and runs a prompt behind the scenes from your signature.
The commented lines show usage: calling classify(ticket="Prod is down!") would give back an object whose result.urgency is "high". The last comment is the punchline — you never wrote the prompt; DSPy generated and optimized it from the signature.

What the output means: Nothing prints — this file only defines the task. The value is the shape: inputs, outputs, and zero hand-written prompt wording.

Try this: Change desc="one of: low, medium, high" to add a fourth level like critical, or add a second output field such as reason: str = dspy.OutputField(). Notice you're editing a declaration, never a prompt string.

4 · Advanced — the optimize loop, modeled advanced

An optimizer proposes prompt variants (instructions + example selections), scores each on your metric over a train set, and keeps the best. Model that search offline to see the idea.

Python · a tiny prompt optimizer (runs)

optimize.pydef evaluate(prompt_variant, examples):
    """Fake metric: variants that mention 'step by step' + few-shot score higher."""
    score = 0.5
    if "step by step" in prompt_variant["instructions"]: score += 0.2
    score += 0.05 * min(prompt_variant["few_shot"], 4)
    return min(score, 1.0)

candidates = [
    {"instructions": "Classify the ticket.", "few_shot": 0},
    {"instructions": "Classify the ticket step by step.", "few_shot": 2},
    {"instructions": "Classify the ticket step by step.", "few_shot": 4},
]
best = max(candidates, key=lambda c: evaluate(c, examples=[]))
print("best variant scores:", round(evaluate(best, []), 2))
print("chosen:", best)

best variant scores: 0.9
chosen: {'instructions': 'Classify the ticket step by step.', 'few_shot': 4}

▶ How this works

This is a runnable, stripped-down model of what an optimizer actually does — no external library needed. It stands in for the diagram's Optimizer box: score several candidate prompts and keep the best one. The scoring here is fake (invented rules) so you can watch the search logic without a real model or API.

def evaluate(prompt_variant, examples): is the metric — it hands each candidate prompt a score. Here it starts at 0.5, adds 0.2 if the instructions contain "step by step", and adds a little for each few-shot example (0.05 per example, capped at 4). Real metrics score against real data; this one just fakes the shape.
candidates is a list of three prompt variants — each a dictionary with instructions text and a few_shot count. This is the search space the optimizer looks through.
best = max(candidates, key=lambda c: evaluate(c, examples=[])) is the search: max(..., key=...) scores every candidate with evaluate and returns the one with the highest score. The lambda is just a tiny throwaway function saying "score this candidate."
The two print(...) lines report the winner's score and the winning variant itself, so you can see what won and why.

What the output means: It prints best variant scores: 0.9 and the chosen variant — the one with "step by step" instructions and 4 few-shot examples (0.5 + 0.2 + 0.05×4 = 0.9). That's the highest-scoring combination the metric could reward.

Try this: Add a fourth candidate with "few_shot": 10 and re-run — the score won't beat 0.9 because the few-shot bonus is capped at 4 examples (min(..., 4)). Then drop "step by step" from the best one and watch the winner change. Swapping the fake evaluate for a real metric over real data is exactly what DSPy does.

Optimization needs a metric + dataAutomatic optimization is only as good as your eval metric and examples (Ch 5). No metric = nothing to optimize toward. The prompt-optimization win comes from having a real, measurable objective — the same discipline as fine-tuning (xt4).

5 · Professional — when to reach for it professional

Hand-write prompts for one-offs and quick iterations. Reach for DSPy/optimization when you have many prompts, a measurable metric, and model upgrades that would otherwise mean re-tuning everything by hand. It shines in maintained, evaluated pipelines.

6 · Tech-lead — optimization as maintainable infra tech-lead

A lead treats prompts like code that's compiled against evals: when the model or data changes, re-optimize instead of hand-editing. This keeps a large prompt surface maintainable and makes model migrations a re-compile, not a rewrite.

Compile prompts, don't hand-maintain themOnce you have evals, letting an optimizer produce prompts means a model upgrade is a re-run, not weeks of manual re-tuning across dozens of prompts. That's the maintainability argument a lead makes for adopting it.

Exercise PE3.1 — Optimize toward a metric

Context: You cannot optimize a prompt without both a metric and labelled data — that pair is the whole prerequisite for treating prompts as something you compile.

Your task: Define a classification task, a small labelled set, and a metric; sketch a DSPy signature; then search a few prompt variants and pick the best by score — and explain why you couldn't optimize without the metric and data.

Requirements:

Define the task, a small labelled set, and a scoring metric
Sketch a DSPy signature for the task
Search a few prompt variants and select the best by score
Explain why the metric + data are prerequisites for optimization

💡 Hint: The optimizer is only as good as the metric it maximises — no labelled data means no metric, which means nothing to optimize toward.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Declare a DSPy signature (needs dspy)Beginner

Context: DSPy's idea is "program, don't prompt": you declare typed inputs and outputs and DSPy writes the prompt. The signature is a typed contract — you describe what, not the wording, which is what lets an optimizer rewrite the wording later.

Your task: Write a classification DSPy Signature with an input field and a described output field, and turn it into a callable — labelled as needing the library.

Requirements:

Declare the task in the signature docstring
Declare an InputField for the ticket
Declare an OutputField with a described value set
Wrap it in dspy.Predict to get a callable
Label the block as needing pip install dspy; write no prompt string by hand

💡 Hint: You never author a prompt string — the docstring plus the typed fields are the whole declaration, and DSPy generates the wording.

Show solution

A typed task declaration — needs pip install dspy (no prompt text written by hand):

import dspy

class Classify(dspy.Signature):
    """Classify a support ticket's urgency."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="one of: low, medium, high")

classify = dspy.Predict(Classify)
# result = classify(ticket="Prod is down!")   ->   result.urgency == "high"
# You never wrote the prompt string -- DSPy generates and later optimizes it.

The signature is a typed contract: the docstring states the task, InputField/OutputField declare the shape, and dspy.Predict turns it into a callable. You describe what, not the wording — which is what lets an optimizer rewrite the wording for you.

Exercise 2 · Why hand-tuning hits a ceilingIntermediate

Context: Hand-tuning prompts does not scale: with many prompts, every model change means re-editing all of them by hand. The ceiling is exactly where re-compiling against evals becomes cheaper.

Your task: Model the maintenance cost of hand-tuning versus re-compiling and show where compiling wins.

Requirements:

Model hand-tuning cost as prompts × model changes × minutes per edit
Model compile cost as one optimizer run per model change
Show hand-tuning cost growing with the prompt count
Show compile cost growing only with model changes
Demonstrate the crossover on a realistic prompt count

💡 Hint: Hand-tuning scales with prompts × changes; compiling scales with changes alone — past a handful of prompts the second curve wins.

Show solution

The maintenance-cost argument as arithmetic (pure stdlib):

def hand_tuning_cost(n_prompts, model_changes, minutes_per_edit=30):
    # every model change forces a manual re-edit of every prompt
    return n_prompts * model_changes * minutes_per_edit

def compile_cost(model_changes, minutes_per_recompile=10):
    # re-run the optimizer once per model change; prompts regenerate
    return model_changes * minutes_per_recompile

print("hand-tune 20 prompts x 4 model bumps:", hand_tuning_cost(20, 4), "min")
print("re-compile 4 times                  :", compile_cost(4), "min")

Hand-tuning cost grows with prompts x changes; compiling cost grows only with changes. Past a handful of prompts, editing by hand every time the model updates is untenable — the ceiling is exactly where re-compiling against evals becomes cheaper.

Exercise 3 · Implement the optimize loop offlineAdvanced

Context: An optimizer is just search: generate candidate prompts, score each on a metric over labelled data, keep the best. The metric is what it aims at, so a bad metric optimizes for the wrong thing.

Your task: Implement the lesson's evaluate() metric and pick the winning candidate from three prompt variants, fully offline.

Requirements:

Score a variant on a metric (e.g. rewards "step by step" and more few-shot examples)
Cap the metric so it can't run away
Evaluate three candidate variants
Select the highest-scoring candidate
Print the winner and its score

💡 Hint: This is the loop DSPy automates by hand — max() over the candidates keyed by your metric is the optimizer's core.

Show solution

The optimize loop, modeled offline exactly as the lesson does (pure stdlib):

def evaluate(variant, examples=None):
    # metric: 'step by step' + more few-shot examples score higher (capped at 4)
    score = 0.5
    if "step by step" in variant["instructions"]:
        score += 0.2
    score += 0.05 * min(variant["few_shot"], 4)
    return min(score, 1.0)

candidates = [
    {"instructions": "Classify the ticket.",              "few_shot": 0},
    {"instructions": "Classify the ticket step by step.", "few_shot": 2},
    {"instructions": "Classify the ticket step by step.", "few_shot": 4},
]
best = max(candidates, key=lambda c: evaluate(c))
print("winner:", best, "score:", evaluate(best))   # score 0.9

This is the loop DSPy automates: generate candidate prompts, score each against a metric on labeled data, keep the best. The optimizer is just search — the metric is what it aims at, so a bad metric optimizes for the wrong thing.

Exercise 4 · Compile against evals so model upgrades are a re-compileExpert

Context: A compiled prompt survives a model upgrade: instead of re-editing by hand you re-run the optimizer against the same eval set on the new model, and it may pick a different variant — no prompt text edited.

Your task: Model re-compiling on a model change: pick the best variant per model against the frozen eval set without editing any prompt text.

Requirements:

Score each variant per model
Let different models prefer different variants
Re-run the selection against the frozen eval set per model
Show the optimizer picking a different winner on the new model
Edit no prompt text by hand anywhere in the flow

💡 Hint: Treating prompts as code compiled against evals turns a model migration into a re-compile — the selection changes, the authoring doesn't.

Show solution

Re-compile, don't re-write, on a model change (pure stdlib):

def score_on_model(variant, model):
    base = 0.5 + (0.2 if "step by step" in variant["instructions"] else 0)
    base += 0.05 * min(variant["few_shot"], 4)
    # different models prefer slightly different variants -- the optimizer adapts
    if model == "new-model" and variant["few_shot"] >= 4:
        base -= 0.1     # new model needs fewer examples
    return min(base, 1.0)

candidates = [
    {"instructions": "Classify step by step.", "few_shot": 2},
    {"instructions": "Classify step by step.", "few_shot": 4},
]
for model in ["old-model", "new-model"]:
    best = max(candidates, key=lambda c: score_on_model(c, model))
    print(model, "->", best, round(score_on_model(best, model), 2))

When the model changes, you re-run the optimizer against the frozen eval set and it may pick a different variant — no prompt text edited by hand. Treating prompts as code compiled against evals turns a model migration into a re-compile, keeping a large prompt surface maintainable.

Exercise 5 · Decide whether DSPy is worth itProfessional

Context: DSPy pays off only with scale (many prompts) AND a real eval set to optimize against. The prerequisite is a metric — with no eval set there is nothing for the optimizer to aim at.

Your task: Encode the adoption decision: no eval set → not yet; few prompts → hand-tune; many prompts + evals + frequent model changes → adopt.

Requirements:

Return NO when there is no eval set, regardless of scale
Return NO when the surface is tiny and churn is low
Return YES at high prompt counts or frequent model churn (given evals)
Leave a MAYBE band for the middle
Demonstrate the YES, small-NO, and no-evals-NO cases

💡 Hint: Gate on the eval set first — it is a hard prerequisite; scale and churn only decide the answer once a metric exists.

Show solution

The adoption gate, straight from the lesson (pure logic):

def should_use_dspy(n_prompts, have_evalset, model_changes_per_qtr):
    if not have_evalset:
        return "NO -- no metric to optimize against; build an eval set first"
    if n_prompts <= 3 and model_changes_per_qtr <= 1:
        return "NO -- too small; hand-tuning is cheaper than the setup"
    if n_prompts >= 10 or model_changes_per_qtr >= 2:
        return "YES -- scale + churn justify compiling against evals"
    return "MAYBE -- measure hand-tune time vs optimizer setup"

print(should_use_dspy(20, have_evalset=True,  model_changes_per_qtr=3))  # YES
print(should_use_dspy(2,  have_evalset=True,  model_changes_per_qtr=1))  # NO small
print(should_use_dspy(20, have_evalset=False, model_changes_per_qtr=3))  # NO no evals

The prerequisite is a metric: with no eval set there is nothing for the optimizer to aim at. Given evals, DSPy earns its setup cost when you have many prompts or frequent model churn — otherwise hand-tuning a few prompts is simpler.

Exercise 6 · Run prompt optimization as maintainable infraIndustry scenario

Context: As lead you treat compiled prompts like any build artifact: the eval set lives in git, the optimizer runs in CI on every model bump, and a score gate blocks a re-compile that regressed — making model migrations routine and reviewable.

Your task: Model the CI gate that blocks a regressed re-compile: compile the best candidate and block if it is below the score gate or dropped versus the last artifact.

Requirements:

Select the best candidate via the metric
Block when the score is below the minimum gate
Block when the score dropped sharply versus the previous artifact
Otherwise ship the compiled prompt as the artifact
Demonstrate the gate on a candidate

💡 Hint: Two blocking conditions — an absolute gate and a regression-versus-last check — are what make a re-compile a reviewable build step, not a risky hand-edit.

Show solution

Optimization as a CI-gated pipeline (pure stdlib):

def compile_and_gate(candidates, evaluate, min_score, prev_score=None):
    best = max(candidates, key=evaluate)
    score = evaluate(best)
    if score < min_score:
        return {"status": "BLOCK", "reason": f"score {score:.2f} < gate {min_score}"}
    if prev_score is not None and prev_score - score > 0.05:
        return {"status": "BLOCK", "reason": "re-compile regressed vs last artifact"}
    return {"status": "SHIP", "artifact": best, "score": score}

evaluate = lambda c: min(0.5 + (0.2 if "step" in c["instructions"] else 0)
                         + 0.05*min(c["few_shot"],4), 1.0)
cands = [{"instructions": "Classify step by step.", "few_shot": 4}]
print(compile_and_gate(cands, evaluate, min_score=0.8, prev_score=0.9))

A lead treats compiled prompts like any build artifact: the eval set lives in git, the optimizer runs in CI on every model bump, and a score gate blocks a re-compile that regressed. That is what makes model migrations a routine, reviewable re-compile instead of a risky hand-edit.

✓ Checkpoint — you can move on when you can…

Explain why hand-tuning doesn't scale.
Describe DSPy signatures + optimizers.
Model the optimize loop against a metric.
Decide when to adopt optimization; treat prompts as compiled infra.

Knowledge check check yourself

✓ Knowledge check

DSPy's slogan is 'program, don't prompt.' What do you write yourself, and what does the framework produce?

Show answer

You declare a signature (typed inputs→outputs, the 'what, not how') and supply labeled data plus a metric; DSPy's optimizer searches candidate prompts (instructions + few-shot example selections), scores each on your metric, and produces the compiled/tuned prompt. You never hand-write the prompt string.

✓ Knowledge check

Why does the lesson insist automatic prompt optimization is 'only as good as your metric and examples', and what tech-lead maintainability argument follows?

Show answer

With no metric there's nothing to optimize toward — the win comes from a real, measurable objective over real data. It follows that treating prompts as compiled-against-evals infra makes a model upgrade a re-run of the optimizer rather than weeks of hand re-tuning across dozens of prompts.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · The hand-tuning ceiling essential

2 · DSPy — program, don't prompt essential

3 · A signature intermediate

4 · Advanced — the optimize loop, modeled advanced

5 · Professional — when to reach for it professional

6 · Tech-lead — optimization as maintainable infra tech-lead

Exercise PE3.1 — Optimize toward a metric

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Prompt engineering