Advanced eval & interpretability

1 · Capability vs alignment evals

Start with the launch that went sideways. A team ships a model that tops every reasoning leaderboard. Marketing celebrates the numbers. Two weeks later a user coaxes it into writing a convincing phishing email — and the post-mortem discovers the safety team was never in the loop, because "the evals passed." The evals that passed measured whether the model was smart. Nobody had measured whether it was safe. Those are two different questions, and the whole discipline of eval design begins with refusing to blur them.

Here's an analogy that keeps the two straight. Hiring a surgeon involves two utterly separate checks: can they operate? (skill — you look at their board scores and case logs) and will they operate responsibly? (judgment — will they cut corners, operate drunk, tell the truth about a mistake?). A dazzlingly skilled surgeon with terrible judgment is more dangerous than a mediocre one, not less, because their skill lets them do more damage before anyone notices. You would never let a high board score stand in for a character reference. Models are the same: raw ability and trustworthy behaviour are orthogonal axes, and being strong on one tells you almost nothing about the other.

That analogy maps directly onto the two eval families. A capability eval asks "can it do X?" and lives in a comfortable world of objective ground truth — a math answer is right or wrong, code passes the tests or it doesn't, so you can score it automatically and rank models cleanly. An alignment (or safety) eval asks "does it behave as intended?" — will it refuse a request to synthesise a weapon, stay honest when flattery would be easier, resist a jailbreak — and here the ground truth is value-laden and context-dependent, often with no single correct answer everyone agrees on. The uncomfortable twist that trips people up: making a model more capable can make it score lower on alignment, because a smarter model is better at producing a fluent, persuasive, harmful answer when it does misbehave. The common mistake is treating a strong capability score as evidence of safety; it is evidence of the opposite risk. You need both numbers, and you must never let one substitute for the other.

Not all evals measure the same kind of thing, and conflating them is a classic mistake. Capability evals ask can the model do X? — accuracy on math, code, reasoning benchmarks. They have (mostly) objective ground truth and reward raw ability. Alignment / safety evals ask does the model behave as intended? — does it refuse harmful requests, stay honest, avoid sycophancy, resist jailbreaks. These often have no single ground truth, depend on values and context, and a more capable model can be less aligned (better at producing a convincing but harmful answer). You need both: capability tells you what the model can do, alignment tells you what it will do.

2 · Benchmark contamination

You have almost certainly watched a version of this in your own life. A student aces the practice exam — the exact one posted online last year — and everyone assumes they've mastered the material. Then the real exam arrives with the same concepts but different numbers, and they fall apart. They hadn't learned to solve; they'd learned to recognise. The practice paper leaked into their "training data," so the high score measured memory, not understanding. That is contamination, and it is the single most common reason a benchmark number is a lie.

The mechanism is exactly the same for a model. Public benchmarks live on the open web — in papers, GitHub repos, Stack Overflow threads, blog posts quoting the questions. Web-scale training data sweeps all of that up. So by the time a model is evaluated on a famous public benchmark, there is a real chance the test items (or near-identical paraphrases) were in the training set. When that happens the model isn't reasoning to the answer; it's retrieving one it has effectively already seen. The score looks spectacular and means nothing about how the model will do on problems it has genuinely never encountered — which is, of course, the only kind that matter in production.

So how do you catch it? The tell is a gap that appears only on fresh material. Watch for a suspiciously high score on an old, well-known public benchmark that does not reproduce on a newer private one; a model that will complete a benchmark item verbatim when you feed it only the first half of the question (pure recall); or a large drop between the public set and a freshly-written set of equivalent problems. The defenses follow directly from the mechanism: keep held-out private test sets that never touch the web, generate fresh items the model can't have seen, embed canary strings (unique markers you can later search training data for), and — the cheapest probe of all — apply perturbation: rephrase the wording and renumber the values, then re-score. A genuine capability barely moves under perturbation because the underlying skill is unchanged; memorization collapses because the surface form it latched onto is gone. The lab below turns that "does the score survive perturbation?" test into three lines of code.

The most common way a benchmark number lies is contamination: the test items (or close paraphrases) leaked into the training data, so the model is recalling rather than solving. Contaminated benchmarks show inflated scores that collapse on fresh, held-out variants. Signals of contamination: suspiciously high scores on old public benchmarks but not on new private ones; the model completing a benchmark item verbatim from a partial prompt; a large gap between a public set and a freshly-generated equivalent. The defenses are canary strings, held-out private test sets, freshly generated items, and perturbation (rephrase/renumber and check the score survives).

contamination.pydef contamination_check(public_score, perturbed_score, threshold=0.15):
    gap = public_score - perturbed_score
    verdict = "LIKELY CONTAMINATED (recall, not reasoning)" if gap > threshold \
              else "no strong contamination signal"
    return gap, verdict

public, perturbed = 0.94, 0.61
gap, verdict = contamination_check(public, perturbed)
print(f"public benchmark score:  {public:.2f}")
print(f"perturbed (renumbered) score: {perturbed:.2f}")
print(f"gap = {gap:.2f}  -> {verdict}")
print("a robust capability survives perturbation; recall does not")

public benchmark score:  0.94
perturbed (renumbered) score: 0.61
gap = 0.33  -> LIKELY CONTAMINATED (recall, not reasoning)
a robust capability survives perturbation; recall does not

3 · LLM-as-judge pitfalls

Picture the leaderboard that lied. A team can't hand-grade ten thousand chatbot replies, so they do the sensible thing: they ask a strong model to judge which of two answers is better, run it across every pair, and publish the ranking. Model A wins convincingly. Champagne. Then someone re-runs the exact same comparisons with the two answers swapped in order — and now Model B wins just as convincingly. The judge wasn't measuring quality at all. It was measuring which answer it saw first. The whole leaderboard was an artifact of formatting.

The reason this happens is worth internalising, because it changes how you trust any automated judge. An LLM judge is not a neutral measuring instrument like a ruler; it's another language model with the same statistical habits as the ones it's grading. Those habits leak into its scores as systematic biases — errors that don't average out with more samples because they all lean the same way. Think of a bathroom scale that always reads three kilos heavy: weighing yourself a hundred times won't fix it, because the error is baked into the instrument, not the noise. A biased judge is that scale. The four you must know by name: position bias — it favours whichever answer is first (or sometimes last) in a pairwise prompt; verbosity bias — it rewards longer answers as if length were quality; self-preference bias — it scores outputs from its own model family or writing style higher; and sycophancy — it rewards confident, agreeable phrasing over correct-but-hedged answers.

Left uncontrolled, these biases quietly convert your evaluation into a style contest: the model that writes long, confident, first-positioned prose wins regardless of whether it's actually right. The fixes attack each bias at its root. For position bias, judge every pair both ways and average — the symmetric part of the score cancels the ordering artifact, which is exactly the trick the lab below demonstrates. For verbosity, length-match the pairs or normalise for length. For self-preference, never let a model judge its own family without a human cross-check. None of these make the judge perfect; they make its errors random instead of directional, which is the difference between a noisy measurement you can average away and a rigged one you can't.

When there's no ground truth, teams reach for LLM-as-judge: use a strong model to score outputs. It scales beautifully and correlates decently with humans — but it carries systematic biases that quietly corrupt your leaderboard. Position bias: judges favor the first (or last) response in a pairwise comparison. Verbosity bias: longer answers score higher regardless of quality. Self-preference bias: a judge favors outputs from its own model family / style. Sycophancy: it rewards confident, agreeable phrasing. Left unchecked, these turn your eval into a style contest.

judge_debias.pydef judge(first, second, len_first, len_second):
    # a biased judge: favors the FIRST option and the LONGER option
    score = 0.5
    score += 0.15                       # position bias toward `first`
    score += 0.05 if len_first > len_second else -0.05   # verbosity bias
    return score                        # score that `first` is better

# Compare A vs B two ways: A-first, then B-first (swap positions)
a_first = judge("A", "B", len_first=40, len_second=30)     # score for A
b_first = 1 - judge("B", "A", len_first=30, len_second=40) # score for A when B is first
print(f"raw judge (A first): prefers A  (score A={a_first:.2f})")
print(f"swapped   (B first): prefers B  (score A={b_first:.2f})")
avg = (a_first + b_first) / 2
print(f"position-averaged score A = {avg:.3f}  -> near tie, bias exposed")
print("always average over swapped positions; control for length")

raw judge (A first): prefers A  (score A=0.70)
swapped   (B first): prefers B  (score A=0.45)
position-averaged score A = 0.575  -> near tie, bias exposed
always average over swapped positions; control for length

Averaging over both orderings cancels position bias; the near-tie it reveals was hidden by the raw single-order score. Pair this with length normalization (or length-matched pairs) and a held-out human spot-check, and never let a model judge its own family without care.

4 · Interpretability I: the logit lens

Every eval so far treated the model as a black box — feed it inputs, grade its outputs, never look inside. That's fine until the day a model does something baffling: gets a fact confidently wrong, or right for reasons you can't explain, and no amount of output-grading tells you why. At that point you need to open the box. Interpretability flips the question from "how well does it perform?" to "what is it actually computing in there?" — and the friendliest first tool for peering inside is the logit lens.

To see the idea, you need one picture of how a transformer thinks. The model doesn't compute its answer in a single leap; it refines a running "notepad" — the residual stream — layer by layer, each layer nudging that vector a little closer to the final prediction. At the very end, one fixed matrix (the unembedding, or output head) translates that final notepad into a probability over vocabulary tokens: the model's answer. The logit lens asks a delightfully cheeky question: what if we grabbed the notepad partway through — at layer 5, or 10 — and ran it through that same output head early? We'd get the model's best guess "as if it had to answer right now." It's like interrupting someone mid-thought and asking "if you had to blurt an answer this instant, what would it be?" — and doing that at every stage of their reasoning to watch the answer take shape.

When you do this across all the layers, you typically see something vivid: the top token starts vague and wrong in the early layers, then sharpens into the correct answer with rising confidence as you climb. That progression is the payoff — it shows you where in the network the answer forms, which is often surprisingly late. But be honest about what this does and doesn't buy you, because it's the classic misread: the logit lens is observational, not causal. It tells you when and where a prediction emerges, the way a series of X-rays shows a bone healing over weeks — but it never tells you what caused the healing. It's a correlation between layer depth and the forming answer, not a controlled experiment. (For the "what causes it" question you need the intervention in §5.) Reading the lens as proof of causation is the single most common beginner error; keep it filed under "reveals structure," not "proves mechanism." The lab runs the whole thing on toy vectors so you can watch the sharpening happen.

Interpretability flips the question from how well does it perform? to what is it computing inside? The simplest and most striking tool is the logit lens. A transformer builds its answer across layers in a shared residual stream; the final layer is projected through the unembedding (output head) to token logits. The logit lens applies that same output head to the intermediate residual stream at each layer — reading off what the model 'would say' if it stopped there. You typically see the prediction sharpen from vague to confident as you climb the layers, revealing where in the network the answer forms.

logit_lens.pyimport math

VOCAB = ["the", "cat", "Paris", "dog"]
# fixed output head: rows are token embeddings; we dot with the residual.
UNEMBED = {
    "the":   [1.0, 0.0, 0.0],
    "cat":   [0.0, 1.0, 0.0],
    "Paris": [0.0, 0.0, 1.0],
    "dog":   [0.2, 0.2, 0.2],
}
# residual stream at each layer (toy): drifts toward the 'Paris' direction with depth
RESIDUALS = [
    [0.6, 0.4, 0.3],   # layer 0: vague
    [0.4, 0.7, 0.5],   # layer 1
    [0.3, 0.5, 1.1],   # layer 2
    [0.2, 0.3, 2.4],   # layer 3: confident 'Paris'
]

def softmax(xs):
    m = max(xs); e = [math.exp(x - m) for x in xs]; s = sum(e)
    return [v / s for v in e]

for layer, resid in enumerate(RESIDUALS):
    logits = [sum(r * w for r, w in zip(resid, UNEMBED[t])) for t in VOCAB]
    probs = softmax(logits)
    top = max(range(len(VOCAB)), key=lambda i: probs[i])
    tag = "(vague)" if probs[top] < 0.5 else ("(confident)" if probs[top] > 0.85 else "")
    print(f"layer {layer}: top token={VOCAB[top]!r:8} p={probs[top]:.2f}   {tag}")
print("the prediction sharpens across layers -- the answer forms late")

layer 0: top token='the'   p=0.34   (vague)
layer 1: top token='cat'   p=0.41
layer 2: top token='Paris' p=0.58
layer 3: top token='Paris' p=0.91   (confident)
the prediction sharpens across layers -- the answer forms late

5 · Interpretability II: activation patching

The logit lens left us with a nagging gap. It showed us the answer appearing around layer 20 — but "appearing at layer 20" doesn't mean layer 20 caused it, any more than the moment you notice a headache is the moment it started. Correlation, not cause. To make a causal claim you can't just watch the machine; you have to poke it and see what breaks. That's the leap activation patching (also called causal tracing) makes: it turns interpretability from observation into experiment.

The method borrows the oldest trick in experimental science: change one thing and hold everything else fixed. Think of debugging a machine with a hundred wires by swapping one known-good wire in from a working unit and seeing if the fault clears — the wire that fixes it is the one that mattered. Concretely, you run the model twice. Once on a clean input that produces the right answer ("The Eiffel Tower is in Paris"), caching every internal activation. Once on a corrupted input that breaks the answer (swap "Eiffel" for "Colosseum," so it should now say Rome). Then comes the intervention: you take a single component's activation from the clean run and patch it into the corrupted run, leaving everything else corrupted, and measure how much the correct clean answer comes back. If patching that one component restores the answer, that component was causally responsible for carrying the fact; if nothing changes, it wasn't.

Why this is such a big deal: it's the difference between "the light comes on when I flip switches" and "this switch controls that light." Sweep the patch across every attention head and MLP layer and you get a causal map of where in the network a specific fact or behaviour actually lives — the foundation of real mechanistic interpretability, and the basis for later work on editing or steering what a model knows. The honest caveat to keep: a restored answer localises where the computation flows, not the human-readable "concept" a component encodes — that finer question is what sparse autoencoders in §6 take on. The lab models the whole clean-vs-corrupted-then-patch loop on toy per-layer contributions so the causal logic is visible without a GPU.

The logit lens shows where an answer forms; activation patching (causal tracing) shows which components cause it. The recipe: run the model on a clean input and a corrupted one (e.g. a changed fact), cache the activations, then patch a single component's clean activation into the corrupted run and measure how much it restores the clean answer. Components whose patch restores the answer are causally responsible for it. This turns interpretability into controlled experiments — you're not correlating, you're intervening.

activation_patching.py# Each layer contributes an additive push toward the CLEAN answer.
# Corrupting the input removes those pushes; patching a layer adds its clean push back.
clean_contrib = {0: 0.08, 1: 0.15, 2: 0.72}   # per-layer clean contribution to the answer
BASE_CORRUPT = 0.10                             # corrupted-run answer prob (fact removed)
CLEAN = 0.90

print(f"clean answer prob:     {CLEAN:.2f}")
print(f"corrupted answer prob:  {BASE_CORRUPT:.2f}")
best = None
for layer, contrib in clean_contrib.items():
    restored = BASE_CORRUPT + contrib          # patch in this layer's clean activation
    effect = restored - BASE_CORRUPT
    mark = ""
    if best is None or effect > best[1]:
        best = (layer, effect); mark = ""
    print(f"patch layer {layer}: restored -> {restored:.2f}  (effect {effect:.2f})"
          + ("  <- causal component" if contrib == max(clean_contrib.values()) else ""))
print("patching localizes the component that carries the fact")

clean answer prob:     0.90
corrupted answer prob:  0.10
patch layer 0: restored -> 0.18  (effect 0.10)
patch layer 1: restored -> 0.30  (effect 0.25)
patch layer 2: restored -> 0.82  (effect 0.90)  <- causal component
patching localizes the component that carries the fact

The layer whose patch restores most of the clean answer is where the relevant computation lives — a causal claim, not a correlation. Scaled up over attention heads and MLP neurons, this is how researchers localize where a model stores a fact or implements a behavior.

6 · Interpretability III: sparse autoencoders

Patching told us which layer carries a fact — but not what the neurons in it actually mean. The obvious next move is to open up a promising layer, look at its individual neurons, and read off what each one detects. Researchers tried exactly that, and hit a wall: a single neuron fires for "Python code" and "the color blue" and "legal jargon" and a dozen other unrelated things. No neuron cleanly means one thing. The dream of "here's the Golden-Gate-Bridge neuron" dies on contact with reality. The reason has a name — superposition — and getting past it is the whole point of this section.

Superposition is the model's clever compression trick, and an analogy makes it click. Imagine you have 10 audio channels but need to broadcast 1,000 different signals. You can't give each signal its own channel — so you mix many signals onto each channel, betting that they rarely all blare at once, and rely on the fact that any given moment only a handful are actually active. That's exactly what a neural network does: it has far more features it wants to represent (concepts, patterns, facts) than it has neurons, so it packs many features into overlapping combinations of neurons. Efficient for the model, unreadable for us — because now reading one neuron is like listening to one overloaded channel and trying to name every signal tangled together on it.

A sparse autoencoder (SAE) is the un-mixer. The insight it exploits is the "few active at once" bet above: if only a handful of true features are live in any given activation, we should be able to re-express that activation in a much wider space where each unit stands for one clean feature and almost all units are off. So the SAE learns a small neural network that encodes a layer's activation into an overcomplete hidden space (many more units than the original had) under a strong sparsity penalty (an L1 term) that forces all but a few of those units to zero, then decodes back and is trained to reconstruct the original activation. The magic empirical result: when you inspect which units light up, they turn out to be monosemantic — one unit for "Golden Gate Bridge," another for "DNA sequences," each meaning one nameable thing. The mistake to avoid is thinking sparsity is free: you deliberately accept a little reconstruction error in exchange for features you can actually read, name, and even steer. The lab strips this to its essence — an L1-style top-k mask on toy activations — so you can watch a dense, uninterpretable code collapse into a sparse, legible one.

A core obstacle is superposition: a single neuron participates in many unrelated features, so individual neurons aren't interpretable. Sparse autoencoders (SAEs) attack this by learning an overcomplete, sparse dictionary: they encode a layer's activation into a much wider hidden space with an L1 (sparsity) penalty, so each example lights up only a few hidden units — and those units tend to correspond to monosemantic human-interpretable features (e.g. 'this is about the Golden Gate Bridge'). The SAE is trained to reconstruct the activation from that sparse code; the sparse features are the payoff. This lab shows the sparsity mechanism on toy data.

sparse_autoencoder.pyimport random
random.seed(1)

def encode(x, W):
    # hidden pre-activations = W . x  (16 hidden units, len(x)-dim input)
    return [sum(wij * xj for wij, xj in zip(row, x)) for row in W]

def relu(v):
    return [max(0.0, u) for u in v]

def sparsify(h, keep):
    # keep only the top-`keep` units (an L1 penalty's effect: few active units)
    thresh = sorted(h, reverse=True)[keep - 1] if keep <= len(h) else min(h)
    return [u if u >= thresh and u > 0 else 0.0 for u in h]

x = [1.0, 0.3, -0.2, 0.8]
W = [[random.uniform(-1, 1) for _ in x] for _ in range(16)]
h = relu(encode(x, W))
dense_active = sum(1 for u in h if u > 0)
sparse = sparsify(h, keep=3)
sparse_active = sum(1 for u in sparse if u > 0)

# crude reconstruction error proxy: energy dropped by zeroing small units
recon_err = sum((a - b) ** 2 for a, b in zip(h, sparse)) / (sum(u*u for u in h) + 1e-9)
print(f"dense code:  active units {dense_active}/16   (every unit fires -- uninterpretable)")
print(f"sparse code: active units  {sparse_active}/16   (only a few fire -- monosemantic-ish)")
print(f"reconstruction error (sparse): {recon_err:.3f}  (features still explain the input)")
print("sparsity trades a little reconstruction for interpretable features")

dense code:  active units 16/16   (every unit fires -- uninterpretable)
sparse code: active units  3/16   (only a few fire -- monosemantic-ish)
reconstruction error (sparse): 0.041  (features still explain the input)
sparsity trades a little reconstruction for interpretable features

EVALS = [
    ("math accuracy",       "capability", "objective"),
    ("refusal rate",        "alignment",  "value-laden"),
    ("jailbreak resistance","alignment",  "value-laden / adversarial"),
    ("code pass@1",         "capability", "objective"),
]
for name, kind, truth in EVALS:
    print(f"{name:22} -> {kind:11} (ground truth: {truth})")

def contamination(public, perturbed, threshold=0.15):
    gap = public - perturbed
    flag = "LIKELY CONTAMINATED" if gap > threshold else "clean-ish"
    return gap, flag

for pub, pert in [(0.94, 0.61), (0.88, 0.85)]:
    gap, flag = contamination(pub, pert)
    print(f"public={pub:.2f} perturbed={pert:.2f} gap={gap:.2f} -> {flag}")

def judge_first_better(len_first, len_second):
    return 0.5 + 0.15 + (0.05 if len_first > len_second else -0.05)  # biased judge

a_first = judge_first_better(40, 30)          # score for A when A is first
b_first = 1 - judge_first_better(30, 40)      # score for A when B is first
flip = (a_first > 0.5) != (b_first > 0.5)
avg = (a_first + b_first) / 2
print(f"A-first={a_first:.2f}  B-first={b_first:.2f}  order-flip={flip}")
print(f"position-averaged score for A = {avg:.3f}")

import math
VOCAB = ["the", "cat", "Paris"]
UNEMBED = {"the":[1,0,0], "cat":[0,1,0], "Paris":[0,0,1]}
RESIDUALS = [[0.6,0.4,0.3],[0.4,0.7,0.5],[0.3,0.5,1.1],[0.2,0.3,2.4]]

def softmax(xs):
    m = max(xs); e = [math.exp(x-m) for x in xs]; s = sum(e)
    return [v/s for v in e]

for L, r in enumerate(RESIDUALS):
    logits = [sum(ri*wi for ri, wi in zip(r, UNEMBED[t])) for t in VOCAB]
    p = softmax(logits)
    i = max(range(len(VOCAB)), key=lambda k: p[k])
    print(f"layer {L}: {VOCAB[i]!r:8} p={p[i]:.2f}")

clean_contrib = {0: 0.08, 1: 0.15, 2: 0.72}
CORRUPT = 0.10
effects = {}
for layer, contrib in clean_contrib.items():
    restored = CORRUPT + contrib          # add this layer's clean activation back
    effects[layer] = restored - CORRUPT
causal = max(effects, key=effects.get)
for layer, eff in effects.items():
    print(f"patch layer {layer}: effect {eff:.2f}" + ("  <- causal" if layer==causal else ""))
print(f"most causal component: layer {causal}")

def validate_harness(cfg):
    fails = []
    if not cfg.get("has_heldout_set"):
        fails.append("no held-out/private set -- contamination risk")
    if cfg.get("uses_llm_judge") and not cfg.get("position_averaged"):
        fails.append("LLM-judge without position averaging")
    if cfg.get("uses_llm_judge") and not cfg.get("length_controlled"):
        fails.append("LLM-judge without length control")
    if not cfg.get("has_alignment_evals"):
        fails.append("capability-only -- no alignment/safety evals")
    if cfg.get("judge_same_family_as_contestant"):
        fails.append("judge shares model family with a contestant (self-preference)")
    return ("PASS", []) if not fails else ("FAIL", fails)

good = dict(has_heldout_set=True, uses_llm_judge=True, position_averaged=True,
            length_controlled=True, has_alignment_evals=True,
            judge_same_family_as_contestant=False)
bad = dict(has_heldout_set=False, uses_llm_judge=True, position_averaged=False,
           length_controlled=False, has_alignment_evals=False,
           judge_same_family_as_contestant=True)
print("good harness:", validate_harness(good)[0])
status, fails = validate_harness(bad)
print("bad harness :", status)
for f in fails: print("   -", f)