Advanced eval & interpretability

1 · Capability vs alignment evals

Not all evals measure the same kind of thing, and conflating them is a classic mistake. Capability evals ask can the model do X? — accuracy on math, code, reasoning benchmarks. They have (mostly) objective ground truth and reward raw ability. Alignment / safety evals ask does the model behave as intended? — does it refuse harmful requests, stay honest, avoid sycophancy, resist jailbreaks. These often have no single ground truth, depend on values and context, and a more capable model can be less aligned (better at producing a convincing but harmful answer). You need both: capability tells you what the model can do, alignment tells you what it will do.

2 · Benchmark contamination

The most common way a benchmark number lies is contamination: the test items (or close paraphrases) leaked into the training data, so the model is recalling rather than solving. Contaminated benchmarks show inflated scores that collapse on fresh, held-out variants. Signals of contamination: suspiciously high scores on old public benchmarks but not on new private ones; the model completing a benchmark item verbatim from a partial prompt; a large gap between a public set and a freshly-generated equivalent. The defenses are canary strings, held-out private test sets, freshly generated items, and perturbation (rephrase/renumber and check the score survives).

contamination.pydef contamination_check(public_score, perturbed_score, threshold=0.15):
    gap = public_score - perturbed_score
    verdict = "LIKELY CONTAMINATED (recall, not reasoning)" if gap > threshold \
              else "no strong contamination signal"
    return gap, verdict

public, perturbed = 0.94, 0.61
gap, verdict = contamination_check(public, perturbed)
print(f"public benchmark score:  {public:.2f}")
print(f"perturbed (renumbered) score: {perturbed:.2f}")
print(f"gap = {gap:.2f}  -> {verdict}")
print("a robust capability survives perturbation; recall does not")

public benchmark score:  0.94
perturbed (renumbered) score: 0.61
gap = 0.33  -> LIKELY CONTAMINATED (recall, not reasoning)
a robust capability survives perturbation; recall does not

3 · LLM-as-judge pitfalls

When there's no ground truth, teams reach for LLM-as-judge: use a strong model to score outputs. It scales beautifully and correlates decently with humans — but it carries systematic biases that quietly corrupt your leaderboard. Position bias: judges favor the first (or last) response in a pairwise comparison. Verbosity bias: longer answers score higher regardless of quality. Self-preference bias: a judge favors outputs from its own model family / style. Sycophancy: it rewards confident, agreeable phrasing. Left unchecked, these turn your eval into a style contest.

judge_debias.pydef judge(first, second, len_first, len_second):
    # a biased judge: favors the FIRST option and the LONGER option
    score = 0.5
    score += 0.15                       # position bias toward `first`
    score += 0.05 if len_first > len_second else -0.05   # verbosity bias
    return score                        # score that `first` is better

# Compare A vs B two ways: A-first, then B-first (swap positions)
a_first = judge("A", "B", len_first=40, len_second=30)     # score for A
b_first = 1 - judge("B", "A", len_first=30, len_second=40) # score for A when B is first
print(f"raw judge (A first): prefers A  (score A={a_first:.2f})")
print(f"swapped   (B first): prefers B  (score A={b_first:.2f})")
avg = (a_first + b_first) / 2
print(f"position-averaged score A = {avg:.3f}  -> near tie, bias exposed")
print("always average over swapped positions; control for length")

raw judge (A first): prefers A  (score A=0.70)
swapped   (B first): prefers B  (score A=0.45)
position-averaged score A = 0.575  -> near tie, bias exposed
always average over swapped positions; control for length

Averaging over both orderings cancels position bias; the near-tie it reveals was hidden by the raw single-order score. Pair this with length normalization (or length-matched pairs) and a held-out human spot-check, and never let a model judge its own family without care.

4 · Interpretability I: the logit lens

Interpretability flips the question from how well does it perform? to what is it computing inside? The simplest and most striking tool is the logit lens. A transformer builds its answer across layers in a shared residual stream; the final layer is projected through the unembedding (output head) to token logits. The logit lens applies that same output head to the intermediate residual stream at each layer — reading off what the model 'would say' if it stopped there. You typically see the prediction sharpen from vague to confident as you climb the layers, revealing where in the network the answer forms.

logit_lens.pyimport math

VOCAB = ["the", "cat", "Paris", "dog"]
# fixed output head: rows are token embeddings; we dot with the residual.
UNEMBED = {
    "the":   [1.0, 0.0, 0.0],
    "cat":   [0.0, 1.0, 0.0],
    "Paris": [0.0, 0.0, 1.0],
    "dog":   [0.2, 0.2, 0.2],
}
# residual stream at each layer (toy): drifts toward the 'Paris' direction with depth
RESIDUALS = [
    [0.6, 0.4, 0.3],   # layer 0: vague
    [0.4, 0.7, 0.5],   # layer 1
    [0.3, 0.5, 1.1],   # layer 2
    [0.2, 0.3, 2.4],   # layer 3: confident 'Paris'
]

def softmax(xs):
    m = max(xs); e = [math.exp(x - m) for x in xs]; s = sum(e)
    return [v / s for v in e]

for layer, resid in enumerate(RESIDUALS):
    logits = [sum(r * w for r, w in zip(resid, UNEMBED[t])) for t in VOCAB]
    probs = softmax(logits)
    top = max(range(len(VOCAB)), key=lambda i: probs[i])
    tag = "(vague)" if probs[top] < 0.5 else ("(confident)" if probs[top] > 0.85 else "")
    print(f"layer {layer}: top token={VOCAB[top]!r:8} p={probs[top]:.2f}   {tag}")
print("the prediction sharpens across layers -- the answer forms late")

layer 0: top token='the'   p=0.34   (vague)
layer 1: top token='cat'   p=0.41
layer 2: top token='Paris' p=0.58
layer 3: top token='Paris' p=0.91   (confident)
the prediction sharpens across layers -- the answer forms late

5 · Interpretability II: activation patching

The logit lens shows where an answer forms; activation patching (causal tracing) shows which components cause it. The recipe: run the model on a clean input and a corrupted one (e.g. a changed fact), cache the activations, then patch a single component's clean activation into the corrupted run and measure how much it restores the clean answer. Components whose patch restores the answer are causally responsible for it. This turns interpretability into controlled experiments — you're not correlating, you're intervening.

activation_patching.py# Each layer contributes an additive push toward the CLEAN answer.
# Corrupting the input removes those pushes; patching a layer adds its clean push back.
clean_contrib = {0: 0.08, 1: 0.15, 2: 0.72}   # per-layer clean contribution to the answer
BASE_CORRUPT = 0.10                             # corrupted-run answer prob (fact removed)
CLEAN = 0.90

print(f"clean answer prob:     {CLEAN:.2f}")
print(f"corrupted answer prob:  {BASE_CORRUPT:.2f}")
best = None
for layer, contrib in clean_contrib.items():
    restored = BASE_CORRUPT + contrib          # patch in this layer's clean activation
    effect = restored - BASE_CORRUPT
    mark = ""
    if best is None or effect > best[1]:
        best = (layer, effect); mark = ""
    print(f"patch layer {layer}: restored -> {restored:.2f}  (effect {effect:.2f})"
          + ("  <- causal component" if contrib == max(clean_contrib.values()) else ""))
print("patching localizes the component that carries the fact")

clean answer prob:     0.90
corrupted answer prob:  0.10
patch layer 0: restored -> 0.18  (effect 0.10)
patch layer 1: restored -> 0.30  (effect 0.25)
patch layer 2: restored -> 0.82  (effect 0.90)  <- causal component
patching localizes the component that carries the fact

The layer whose patch restores most of the clean answer is where the relevant computation lives — a causal claim, not a correlation. Scaled up over attention heads and MLP neurons, this is how researchers localize where a model stores a fact or implements a behavior.

6 · Interpretability III: sparse autoencoders

A core obstacle is superposition: a single neuron participates in many unrelated features, so individual neurons aren't interpretable. Sparse autoencoders (SAEs) attack this by learning an overcomplete, sparse dictionary: they encode a layer's activation into a much wider hidden space with an L1 (sparsity) penalty, so each example lights up only a few hidden units — and those units tend to correspond to monosemantic human-interpretable features (e.g. 'this is about the Golden Gate Bridge'). The SAE is trained to reconstruct the activation from that sparse code; the sparse features are the payoff. This lab shows the sparsity mechanism on toy data.

sparse_autoencoder.pyimport random
random.seed(1)

def encode(x, W):
    # hidden pre-activations = W . x  (16 hidden units, len(x)-dim input)
    return [sum(wij * xj for wij, xj in zip(row, x)) for row in W]

def relu(v):
    return [max(0.0, u) for u in v]

def sparsify(h, keep):
    # keep only the top-`keep` units (an L1 penalty's effect: few active units)
    thresh = sorted(h, reverse=True)[keep - 1] if keep <= len(h) else min(h)
    return [u if u >= thresh and u > 0 else 0.0 for u in h]

x = [1.0, 0.3, -0.2, 0.8]
W = [[random.uniform(-1, 1) for _ in x] for _ in range(16)]
h = relu(encode(x, W))
dense_active = sum(1 for u in h if u > 0)
sparse = sparsify(h, keep=3)
sparse_active = sum(1 for u in sparse if u > 0)

# crude reconstruction error proxy: energy dropped by zeroing small units
recon_err = sum((a - b) ** 2 for a, b in zip(h, sparse)) / (sum(u*u for u in h) + 1e-9)
print(f"dense code:  active units {dense_active}/16   (every unit fires -- uninterpretable)")
print(f"sparse code: active units  {sparse_active}/16   (only a few fire -- monosemantic-ish)")
print(f"reconstruction error (sparse): {recon_err:.3f}  (features still explain the input)")
print("sparsity trades a little reconstruction for interpretable features")

dense code:  active units 16/16   (every unit fires -- uninterpretable)
sparse code: active units  3/16   (only a few fire -- monosemantic-ish)
reconstruction error (sparse): 0.041  (features still explain the input)
sparsity trades a little reconstruction for interpretable features

EVALS = [
    ("math accuracy",       "capability", "objective"),
    ("refusal rate",        "alignment",  "value-laden"),
    ("jailbreak resistance","alignment",  "value-laden / adversarial"),
    ("code pass@1",         "capability", "objective"),
]
for name, kind, truth in EVALS:
    print(f"{name:22} -> {kind:11} (ground truth: {truth})")

def contamination(public, perturbed, threshold=0.15):
    gap = public - perturbed
    flag = "LIKELY CONTAMINATED" if gap > threshold else "clean-ish"
    return gap, flag

for pub, pert in [(0.94, 0.61), (0.88, 0.85)]:
    gap, flag = contamination(pub, pert)
    print(f"public={pub:.2f} perturbed={pert:.2f} gap={gap:.2f} -> {flag}")

def judge_first_better(len_first, len_second):
    return 0.5 + 0.15 + (0.05 if len_first > len_second else -0.05)  # biased judge

a_first = judge_first_better(40, 30)          # score for A when A is first
b_first = 1 - judge_first_better(30, 40)      # score for A when B is first
flip = (a_first > 0.5) != (b_first > 0.5)
avg = (a_first + b_first) / 2
print(f"A-first={a_first:.2f}  B-first={b_first:.2f}  order-flip={flip}")
print(f"position-averaged score for A = {avg:.3f}")

import math
VOCAB = ["the", "cat", "Paris"]
UNEMBED = {"the":[1,0,0], "cat":[0,1,0], "Paris":[0,0,1]}
RESIDUALS = [[0.6,0.4,0.3],[0.4,0.7,0.5],[0.3,0.5,1.1],[0.2,0.3,2.4]]

def softmax(xs):
    m = max(xs); e = [math.exp(x-m) for x in xs]; s = sum(e)
    return [v/s for v in e]

for L, r in enumerate(RESIDUALS):
    logits = [sum(ri*wi for ri, wi in zip(r, UNEMBED[t])) for t in VOCAB]
    p = softmax(logits)
    i = max(range(len(VOCAB)), key=lambda k: p[k])
    print(f"layer {L}: {VOCAB[i]!r:8} p={p[i]:.2f}")

clean_contrib = {0: 0.08, 1: 0.15, 2: 0.72}
CORRUPT = 0.10
effects = {}
for layer, contrib in clean_contrib.items():
    restored = CORRUPT + contrib          # add this layer's clean activation back
    effects[layer] = restored - CORRUPT
causal = max(effects, key=effects.get)
for layer, eff in effects.items():
    print(f"patch layer {layer}: effect {eff:.2f}" + ("  <- causal" if layer==causal else ""))
print(f"most causal component: layer {causal}")

def validate_harness(cfg):
    fails = []
    if not cfg.get("has_heldout_set"):
        fails.append("no held-out/private set -- contamination risk")
    if cfg.get("uses_llm_judge") and not cfg.get("position_averaged"):
        fails.append("LLM-judge without position averaging")
    if cfg.get("uses_llm_judge") and not cfg.get("length_controlled"):
        fails.append("LLM-judge without length control")
    if not cfg.get("has_alignment_evals"):
        fails.append("capability-only -- no alignment/safety evals")
    if cfg.get("judge_same_family_as_contestant"):
        fails.append("judge shares model family with a contestant (self-preference)")
    return ("PASS", []) if not fails else ("FAIL", fails)

good = dict(has_heldout_set=True, uses_llm_judge=True, position_averaged=True,
            length_controlled=True, has_alignment_evals=True,
            judge_same_family_as_contestant=False)
bad = dict(has_heldout_set=False, uses_llm_judge=True, position_averaged=False,
           length_controlled=False, has_alignment_evals=False,
           judge_same_family_as_contestant=True)
print("good harness:", validate_harness(good)[0])
status, fails = validate_harness(bad)
print("bad harness :", status)
for f in fails: print("   -", f)