Prompt Optimization, Evaluation & DSPy

1 · The hand-tuning ceiling essential

Here's the moment this chapter is written for. You spent an afternoon getting one prompt just right — you tried "be concise," then "answer in one sentence," then added an example, then reworded the system message, and finally it behaved. You shipped it. Six weeks later the provider releases a smarter model, you switch to it because it's cheaper and faster… and your carefully hand-tuned prompt now behaves worse than before. The wording you sweated over was quietly overfit to the old model's quirks, and none of that effort transferred. Now multiply that pain by the thirty prompts scattered across your codebase. That is the hand-tuning ceiling, and everyone hits it.

Why intuition-tuning feels productive but isn't. When you eyeball a prompt change and think "yeah, that's better," you're running a sample size of one, on inputs you happened to think of, scored by a judge (you) who already knows the answer. That's not measurement — it's a vibe. The change that looks better on your three favourite test cases can easily be worse on the long tail of real traffic you never tried. Hand-tuning has no memory, no regression test, and no way to prove a tweak actually helped rather than just felt like it did.

The analogy that unlocks the whole chapter: hand-tuning a prompt is like hand-tuning assembly code. In the early days people wrote assembly by hand and lovingly optimized each instruction. Then compilers arrived — you write what you want in a high-level language, and the compiler generates optimized machine code for the specific chip you're targeting. Change chips, recompile; you don't hand-rewrite the assembly. Automatic prompt optimization is that same leap for prompts: you stop crafting the prompt string and instead treat it as an artifact you compile — from a task description, some labeled examples, and a metric that says what "good" means. The prompt becomes an output of your pipeline, not a hand-written input to it.

The common mistake to head off right now: people hear "automatic optimization" and imagine it replaces thinking. It doesn't. It replaces fiddling. You still do the hard, human work — defining the task, collecting honest examples, and deciding how to score an answer. What you hand off is the tedious search over wordings and example selections that you were doing badly by hand anyway. Keep that division in mind for the rest of the chapter: you own the objective; the optimizer owns the string.

Tweaking prompt wording by hand is slow, brittle, and doesn't transfer across models. As systems grow (multiple prompts, model upgrades), manual prompt maintenance becomes a bottleneck. Automatic prompt optimization treats the prompt as something you compile, not craft.

2 · DSPy — program, don't prompt essential

Imagine you never had to write a prompt again — you just described the job. That's the promise DSPy is chasing, and its slogan says it out loud: program, don't prompt. Instead of hand-writing a wall of instructions and examples, you write a small, typed program that declares the shape of each step — what goes in, what comes out — and let the framework generate and tune the actual prompt text. The name is telling too: DSPy comes from Stanford and stands for "Declarative Self-improving Python." "Declarative" is the key word — you declare intent, not wording.

The cleanest way to feel the difference is a familiar analogy: DSPy is to prompts what an ORM is to SQL, or what PyTorch is to raw matrix math. With PyTorch you define the architecture of a network — the layers, their shapes, the loss — and an optimizer (gradient descent) tunes the millions of weights for you; you'd never dream of setting weights by hand. DSPy borrows that exact mental model for language programs. You define the modules and their signatures (the architecture), you supply a metric (the loss) and examples (the data), and a DSPy optimizer tunes the prompts (the weights). You're doing machine-learning-style engineering, but the thing being learned is prompt text rather than floating-point weights.

Three moving parts do all the work, and it's worth naming them cleanly because the rest of the chapter uses them:

• Signatures — a typed declaration of one step: named input fields and output fields, plus a short docstring stating the goal. This is the "what, not how." You'll write one in §3.
• Modules — the strategy for fulfilling a signature. dspy.Predict is the plain "just answer it" module; dspy.ChainOfThought is the same signature but asks the model to reason step by step first. Swapping the module changes the approach without touching the signature.
• Optimizers (DSPy calls them teleprompters) — the search engine that, given your metric and examples, proposes candidate prompts (different instructions, different few-shot example picks), scores each, and keeps the best. This is §4.

The mistake to avoid: don't picture DSPy as "a fancier way to write prompt templates." It is the opposite — it's a way to not write them. If you find yourself hand-crafting the instruction string inside a DSPy program, you've slipped back into the old world. The whole point is that the string is generated for you and re-generated whenever your data, metric, or model changes. Read the diagram below as an assembly line: you build the two boxes on the left (the task and the data+metric), and the machine produces the two on the right (the search and the finished, compiled prompt).

DSPy lets you declare what you want (a signature: inputs → outputs) and an optimizer figures out the prompt (instructions + few-shot examples) that maximizes a metric on your data. You program the pipeline; DSPy tunes the strings.

3 · A signature intermediate

A signature is the single idea that makes the rest of DSPy click, so it's worth slowing down here. The word is borrowed from programming on purpose. When you read a function signature like def classify(ticket: str) -> str, you learn everything you need to use it — what it takes, what it returns — without seeing a line of its body. A DSPy signature is the same contract for a language step: it names the inputs, names the outputs, gives their types, and states the goal in a docstring — and it deliberately says nothing about the wording used to get there. The body (the actual prompt) is left blank, to be filled in and tuned by the framework.

Think of it as ordering at a restaurant versus writing the recipe. Hand-prompting is writing the recipe: every step, every measurement, in order. A signature is just the order — "one urgency classification, please, from this ticket, and the answer must be low, medium, or high." You trust the kitchen (DSPy plus the optimizer) to produce the dish. And here's the payoff that the recipe approach can never give you: because you only specified the order, the kitchen is free to change the recipe when the ingredients change. Swap to a new model, and the same signature yields a freshly-tuned prompt with zero edits from you. That single indirection — declaring the interface, not the implementation — is what lets an optimizer rewrite the wording later.

The common misread is to treat the docstring and the desc= hints as "the prompt in disguise" and start cramming instructions into them. Resist that. The docstring is a one-line statement of intent ("Classify a support ticket's urgency"), and desc= is a lightweight constraint on a field ("one of: low, medium, high") — think of them as type annotations and doc comments, not as prompt copy. If you feel the urge to write "be careful to consider the customer's tone and any deadline mentioned…", stop: that reasoning is exactly what the optimizer is supposed to discover and encode for you, informed by your examples. Your job in a signature is to be precise about the interface and silent about the technique.

Notice what typing buys you, too. Declaring urgency: str with a described value set means DSPy — and any validation layer downstream — knows the output should be one of three strings, which is the seam where structured-output checking and parsing hook in. As your steps grow to several inputs and outputs, the signature stays a clean, readable contract you can review at a glance, while the messy, model-specific wording lives where it belongs: in the compiled artifact, out of your source.

A signature is a typed declaration of a step. This is illustrative DSPy shape (runs with pip install dspy).

signature.pyimport dspy

class Classify(dspy.Signature):
    """Classify a support ticket's urgency."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="one of: low, medium, high")

classify = dspy.Predict(Classify)
# result = classify(ticket="Prod is down!")  ->  result.urgency == "high"
# You never wrote the prompt — DSPy generates + optimizes it.

4 · Advanced — the optimize loop, modeled advanced

"Optimizer" sounds like heavy machinery, so let's puncture the mystery immediately: at its heart, a prompt optimizer is just a loop that tries things and keeps the best one. There's no magic and, for the basic optimizers, no gradients — it's the same "generate candidates, score them, pick the winner" pattern you'd use to tune any knob. What makes it feel advanced is only what it's searching over (prompt wordings and which examples to show) and the fact that scoring means calling a language model. Strip those away and the skeleton is three lines: propose, score, argmax.

The analogy that makes it concrete: it's A/B testing your prompts, automated and run to convergence. You already know the manual version — write two prompt variants, run both on some cases, see which wins. The optimizer does that, but with dozens or hundreds of variants and a scorer that never gets tired or biased. It builds variants by turning two dials: the instructions (the wording of the task) and the few-shot examples (which of your labeled cases to paste in as demonstrations, and how many). Different real DSPy optimizers explore this space differently — some bootstrap good examples by running your program and keeping the ones that scored well, more advanced ones even ask a model to propose better instructions — but every one of them bottoms out in "score a candidate on the metric, prefer the higher score."

The metric is the soul of the whole thing, and the most common way to get burned. The optimizer is a genie that grants exactly what you measure — not what you meant. If your metric rewards answers that merely contain the right word, it will happily select a verbose prompt that pads every answer with all three labels to guarantee a match. If your metric is exact-match on a label, it optimizes for that and nothing else — not tone, not brevity, not safety. Whatever you forget to score, the optimizer is free to sacrifice. So the discipline is: before you optimize anything, write a metric that genuinely captures "good" on real, held-out examples, and be suspicious of a suspiciously high score.

The lab below is deliberately fake — the evaluate function scores by invented rules (does the instruction say "step by step"? how many few-shot examples?) instead of calling a model — precisely so you can watch the search logic in isolation, with no API key and no randomness. Read it as the exposed skeleton of a real optimizer: max(candidates, key=evaluate) is the optimizer's core, and swapping that hand-written evaluate for a real metric over real data is, almost literally, the only difference between this toy and what DSPy runs for you.

An optimizer proposes prompt variants (instructions + example selections), scores each on your metric over a train set, and keeps the best. Model that search offline to see the idea.

optimize.pydef evaluate(prompt_variant, examples):
    """Fake metric: variants that mention 'step by step' + few-shot score higher."""
    score = 0.5
    if "step by step" in prompt_variant["instructions"]: score += 0.2
    score += 0.05 * min(prompt_variant["few_shot"], 4)
    return min(score, 1.0)

candidates = [
    {"instructions": "Classify the ticket.", "few_shot": 0},
    {"instructions": "Classify the ticket step by step.", "few_shot": 2},
    {"instructions": "Classify the ticket step by step.", "few_shot": 4},
]
best = max(candidates, key=lambda c: evaluate(c, examples=[]))
print("best variant scores:", round(evaluate(best, []), 2))
print("chosen:", best)

best variant scores: 0.9
chosen: {'instructions': 'Classify the ticket step by step.', 'few_shot': 4}

5 · Professional — when to reach for it professional

Every powerful tool has a wrong size of problem, and reaching for DSPy on a one-off is like standing up a CI/CD pipeline to run a script once. The setup — collecting labeled examples, writing an honest metric, wiring the optimizer — is real work, and on a single throwaway prompt it will cost you far more than the ten minutes of hand-tuning it replaces. So the professional skill here isn't "know how to use DSPy," it's "know when the investment pays back." Getting that judgment wrong in either direction is expensive: over-engineering a trivial prompt wastes days, while hand-maintaining a large prompt surface quietly bleeds hours on every model bump.

The honest rule of thumb has three ingredients, and you need all three before the tool earns its keep. Many prompts — a dozen or more, so the per-prompt setup amortizes and the "re-tune everything by hand" cost is genuinely painful. A measurable metric over real, labeled data — this is the hard prerequisite, not a nice-to-have, because with nothing to score there is literally nothing for an optimizer to optimize; you'd just be running search over a coin flip. And churn — model upgrades or shifting data that force you to re-tune periodically, which is what turns a one-time hand-tune into a recurring tax. Take away any one of these and hand-writing wins: few prompts means low total cost, no metric means no optimizer, no churn means you tune once and forget it.

A useful way to hold it: think of the crossover the way you think about writing a test versus testing by hand. For a script you'll run once, manually checking the output is faster than writing a test. For code that ships, changes, and must keep working across refactors, the automated test pays for itself many times over. Prompts are the same — a one-off gets eyeballed, but a prompt embedded in a maintained, evaluated pipeline that survives model migrations wants to be compiled against evals, not babysat. The exercises in this chapter make the arithmetic of that crossover explicit so you can defend the decision rather than guess at it.

The mistake at this level is treating it as all-or-nothing. You don't have to compile every prompt in the codebase. The pragmatic move is to hand-write during early exploration when the task itself is still shifting — you're not sure what "good" even means yet — and graduate the prompts that have stabilized, matter most, and now have a real eval set behind them. Optimization is a stage a prompt earns its way into, not a religion you adopt on day one.

Hand-write prompts for one-offs and quick iterations. Reach for DSPy/optimization when you have many prompts, a measurable metric, and model upgrades that would otherwise mean re-tuning everything by hand. It shines in maintained, evaluated pipelines.

6 · Tech-lead — optimization as maintainable infra tech-lead

Zoom out from the single prompt and picture the thing a tech lead actually loses sleep over: a codebase with forty prompts in it, and a provider email announcing the model you built on is being deprecated in ninety days. On a hand-tuned codebase that email is a small crisis — someone has to re-open forty prompt strings, re-run them by feel against whatever cases they remember, and pray nothing regressed silently. On a compiled codebase it's a chore: re-run the optimizer against the eval set on the new model, read the score report, ship. Same event, two completely different Mondays. The lead's job is to build the second Monday before the email arrives.

The reframing that gets you there is to stop thinking of prompts as content and start thinking of them as build artifacts. Source code isn't the binary you ship — you compile it, and the compiler output is regenerated whenever the source or the target changes. A lead treats prompts identically: the source is your signatures, your labeled eval set (checked into git, versioned, reviewed like any other code), and your metric; the compiled prompt is the output. Nobody hand-edits the binary; nobody hand-edits the compiled prompt. When the model or the data changes, you don't patch the artifact — you recompile from source. That single discipline is what keeps a large prompt surface from rotting into a pile of undocumented, model-specific incantations nobody dares touch.

And because it's now a build step, it slots into the machinery you already trust. The optimizer runs in CI on every model bump, and — critically — you put a score gate in front of it, exactly like a test suite: a recompile that scores below an absolute threshold, or that regressed against the last shipped artifact, blocks the merge instead of quietly shipping worse behavior. This is the deepest reason the whole approach matters: it makes prompt changes reviewable and reversible. A diff shows the score moved from 0.87 to 0.91; a regression is caught by a machine, not a user; a bad model migration rolls back to the previous artifact. You've turned the scariest, most hand-wavy part of an LLM system — "did changing the model break our prompts?" — into a green or red check on a pull request.

A lead treats prompts like code that's compiled against evals: when the model or data changes, re-optimize instead of hand-editing. This keeps a large prompt surface maintainable and makes model migrations a re-compile, not a rewrite.

import dspy

class Classify(dspy.Signature):
    """Classify a support ticket's urgency."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="one of: low, medium, high")

classify = dspy.Predict(Classify)
# result = classify(ticket="Prod is down!")   ->   result.urgency == "high"
# You never wrote the prompt string -- DSPy generates and later optimizes it.

def hand_tuning_cost(n_prompts, model_changes, minutes_per_edit=30):
    # every model change forces a manual re-edit of every prompt
    return n_prompts * model_changes * minutes_per_edit

def compile_cost(model_changes, minutes_per_recompile=10):
    # re-run the optimizer once per model change; prompts regenerate
    return model_changes * minutes_per_recompile

print("hand-tune 20 prompts x 4 model bumps:", hand_tuning_cost(20, 4), "min")
print("re-compile 4 times                  :", compile_cost(4), "min")

def evaluate(variant, examples=None):
    # metric: 'step by step' + more few-shot examples score higher (capped at 4)
    score = 0.5
    if "step by step" in variant["instructions"]:
        score += 0.2
    score += 0.05 * min(variant["few_shot"], 4)
    return min(score, 1.0)

candidates = [
    {"instructions": "Classify the ticket.",              "few_shot": 0},
    {"instructions": "Classify the ticket step by step.", "few_shot": 2},
    {"instructions": "Classify the ticket step by step.", "few_shot": 4},
]
best = max(candidates, key=lambda c: evaluate(c))
print("winner:", best, "score:", evaluate(best))   # score 0.9

def score_on_model(variant, model):
    base = 0.5 + (0.2 if "step by step" in variant["instructions"] else 0)
    base += 0.05 * min(variant["few_shot"], 4)
    # different models prefer slightly different variants -- the optimizer adapts
    if model == "new-model" and variant["few_shot"] >= 4:
        base -= 0.1     # new model needs fewer examples
    return min(base, 1.0)

candidates = [
    {"instructions": "Classify step by step.", "few_shot": 2},
    {"instructions": "Classify step by step.", "few_shot": 4},
]
for model in ["old-model", "new-model"]:
    best = max(candidates, key=lambda c: score_on_model(c, model))
    print(model, "->", best, round(score_on_model(best, model), 2))

def should_use_dspy(n_prompts, have_evalset, model_changes_per_qtr):
    if not have_evalset:
        return "NO -- no metric to optimize against; build an eval set first"
    if n_prompts <= 3 and model_changes_per_qtr <= 1:
        return "NO -- too small; hand-tuning is cheaper than the setup"
    if n_prompts >= 10 or model_changes_per_qtr >= 2:
        return "YES -- scale + churn justify compiling against evals"
    return "MAYBE -- measure hand-tune time vs optimizer setup"

print(should_use_dspy(20, have_evalset=True,  model_changes_per_qtr=3))  # YES
print(should_use_dspy(2,  have_evalset=True,  model_changes_per_qtr=1))  # NO small
print(should_use_dspy(20, have_evalset=False, model_changes_per_qtr=3))  # NO no evals

def compile_and_gate(candidates, evaluate, min_score, prev_score=None):
    best = max(candidates, key=evaluate)
    score = evaluate(best)
    if score < min_score:
        return {"status": "BLOCK", "reason": f"score {score:.2f} < gate {min_score}"}
    if prev_score is not None and prev_score - score > 0.05:
        return {"status": "BLOCK", "reason": "re-compile regressed vs last artifact"}
    return {"status": "SHIP", "artifact": best, "score": score}

evaluate = lambda c: min(0.5 + (0.2 if "step" in c["instructions"] else 0)
                         + 0.05*min(c["few_shot"],4), 1.0)
cands = [{"instructions": "Classify step by step.", "few_shot": 4}]
print(compile_and_gate(cands, evaluate, min_score=0.8, prev_score=0.9))

Knowledge check check yourself