Test doubles & non-functional testing

1 · The test-double taxonomy (Meszaros)

Start with the argument you've probably had in a code review. Someone writes a test, calls every stand-in object a "mock," and asserts a returned value. A colleague objects: "That's not a mock, that's a stub — and you shouldn't be mocking there at all." The words fly, nobody agrees on what they mean, and the review stalls. The frustrating part is that both people are half right: they're using one word — "mock" — for five genuinely different tools, each with a different job. This section gives you the exact vocabulary so those arguments end in ten seconds.

Here's the analogy that makes the five click. Imagine you're rehearsing a play, but the lead actor hasn't arrived. You need stand-ins so rehearsal can continue — and which kind of stand-in you use depends on what the scene needs. A cardboard cutout just fills the spot so the stage isn't empty (nobody looks at it): that's a dummy. A stand-in who reads scripted lines off a card so the scene can move forward is a stub — it feeds canned answers. A stand-in who reads the lines and quietly writes down "the director asked me to enter twice" is a spy — it answers and records what happened for you to check afterward. A stand-in who was told in advance exactly which lines they must be given and storms off if the director gets it wrong is a mock — it carries its expectations and fails itself when they're not met. And a junior actor who actually performs a simplified version of the whole role is a fake — real working behaviour, just not production-grade (think an in-memory dictionary standing in for a database).

The single distinction worth burning into memory is what each one lets you assert. A dummy verifies nothing. A stub and a fake let you check state — the result the code produced or the data it stored. A spy and a mock let you check behaviour — how the code called its collaborator (was send called once, with these arguments?). That is the whole taxonomy in one sentence: state doubles (stub, fake) versus behaviour doubles (spy, mock), with the dummy as the do-nothing placeholder. The common mistake this whole framing exists to prevent is reaching for a mock by reflex: mocks assert the exact shape of your internal calls, so they couple the test to the implementation. Rename a method or reorder two calls and a mock-heavy suite goes red even though nothing broke. That's why the professional default is "prefer stubs and fakes, verify state, and reach for a spy or mock only when the interaction genuinely is the behaviour you're testing" — for example, that a payment gateway was charged exactly once.

“Mock” is used loosely to mean any stand-in, but Gerard Meszaros' xUnit Test Patterns defines five distinct test doubles, and interviews expect the precise distinctions:

The deepest distinction is state verification (assert the result / stored state — stubs and fakes) versus behaviour verification (assert the collaborator was called a certain way — spies and mocks). Over-using mocks couples tests to implementation; prefer stubs/fakes and verify state where you can.

test_doubles.pyimport unittest
from unittest.mock import MagicMock

# The five test doubles (Meszaros), each shown against the SAME collaborator: a
# Notifier that a PasswordReset service depends on.

# --- 1. DUMMY: passed to satisfy a signature, never actually used. ---
class DummyNotifier:
    def send(self, *a, **k): raise AssertionError("dummy must not be called")

# --- 2. STUB: returns canned answers to drive the code under test. ---
class StubClock:
    def now(self): return 1000            # fixed time, no real clock

# --- 3. FAKE: a real working implementation, but not production-grade. ---
class FakeEmailStore:                     # in-memory 'database'
    def __init__(self): self.sent = []
    def send(self, to, body): self.sent.append((to, body)); return True

# --- 4. SPY: records how it was called so the test can assert on it. ---
class SpyNotifier:
    def __init__(self): self.calls = []
    def send(self, to, body): self.calls.append((to, body))

# --- 5. MOCK: pre-programmed with expectations, verifies them (library-backed). ---
def reset_password(user, notifier, clock):
    token = f"tok-{clock.now()}"
    notifier.send(user, f"reset: {token}")
    return token

class TestDoubles(unittest.TestCase):
    def test_stub_and_fake(self):
        fake = FakeEmailStore()
        token = reset_password("ana@x.io", fake, StubClock())
        self.assertEqual(token, "tok-1000")          # stub drove the value
        self.assertEqual(fake.sent, [("ana@x.io", "reset: tok-1000")])  # fake stored it
    def test_spy_records_calls(self):
        spy = SpyNotifier()
        reset_password("bo@x.io", spy, StubClock())
        self.assertEqual(len(spy.calls), 1)          # spy verifies interaction
    def test_mock_verifies_expectation(self):
        mock = MagicMock()
        reset_password("cy@x.io", mock, StubClock())
        mock.send.assert_called_once_with("cy@x.io", "reset: tok-1000")

if __name__ == "__main__":
    unittest.main(verbosity=2)

test_mock_verifies_expectation (__main__.TestDoubles.test_mock_verifies_expectation) ... ok
test_spy_records_calls (__main__.TestDoubles.test_spy_records_calls) ... ok
test_stub_and_fake (__main__.TestDoubles.test_stub_and_fake) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

The FakeEmailStore lets you assert state (what was stored); the SpyNotifier and the MagicMock let you assert interaction (that send was called once with the right arguments). The DummyNotifier exists only to prove it is never touched. Same seam, five roles.

2 · Test smells & flaky-test theory

Picture the CI dashboard on a Tuesday morning. A build failed overnight. You open it, see one red test, sigh, and click "re-run." It goes green. You merge. Nobody investigates. Do this enough times and something quietly catastrophic happens to your team: red stops meaning broken. The next time a test goes red because of a genuine, shipping-blocking bug, the reflex is the same — re-run, it might pass — and the bug sails into production behind a test suite everyone has learned to distrust. That test, the one that passes and fails on identical code, is called flaky, and it is the most corrosive smell in the whole discipline precisely because its damage is social, not technical: it erodes the one thing tests are for, a trustworthy signal.

So where does flakiness come from? It is never magic. A test is a function; run it twice on the same code and it must give the same answer — unless something outside the code changed between the runs. That "something" is nondeterminism, and it leaks in through a small, knowable set of doors. The clock (a test that only fails at midnight, or across a daylight-savings boundary). Randomness (a test that fails roughly one run in fifty because it rolled the unlucky value). The network or an external service (fine until a DNS blip). Concurrency (two threads racing for the same variable, order decided by the OS scheduler). Order dependence (test B passes alone but fails after test A left global state dirty). And iteration over unordered collections (asserting a specific order out of a set or, historically, a dict).

The unifying insight — and the reason the table below has a fix for every row — is this: a reliable test must own every source of nondeterminism it touches. It cannot read the ambient clock, the global random generator, the live network, or shared mutable state; it must be handed a controlled version of each. The move that achieves this is the same dependency injection you saw for the clock in TQ2 and for the network via stubs in TQ3: push the uncontrollable thing in as a parameter so the test can pin it. Seed the RNG, freeze the time, stub the service, give each test fresh state in setUp, sort before you assert. The mistake to name and avoid is the tempting shortcut — wrapping a flaky test in a "retry up to three times" loop. That doesn't fix nondeterminism; it hides it, and worse, it silences the exact intermittent failures (a real race condition, a genuine 1-in-50 bug) that flakiness is often trying to warn you about. Quarantine and fix, or delete — never paper over.

A flaky test passes and fails on the same code — the single most corrosive test smell, because it trains the team to ignore red. Flakiness always traces to nondeterminism leaking into the test. The sources, and the fix for each:

flaky.pyimport random, unittest

# A flaky test: depends on randomness / hidden global state -> nondeterministic.
def pick_greeting_flaky():
    return random.choice(["hi", "hello", "hey"])      # nondeterministic output

# The FIX: inject the source of nondeterminism so the test controls it.
def pick_greeting(rng):
    return rng.choice(["hi", "hello", "hey"])

class TestFlaky(unittest.TestCase):
    def test_deterministic_with_injected_rng(self):
        rng = random.Random(42)               # seeded -> repeatable
        self.assertEqual(pick_greeting(rng), pick_greeting(random.Random(42)))
    def test_no_order_dependence(self):
        # Isolated: does not rely on any other test having run first.
        self.assertIn(pick_greeting(random.Random(1)), ["hi", "hello", "hey"])

if __name__ == "__main__":
    unittest.main(verbosity=2)

test_deterministic_with_injected_rng (__main__.TestFlaky.test_deterministic_with_injected_rng) ... ok
test_no_order_dependence (__main__.TestFlaky.test_no_order_dependence) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

The flaky function reads a global RNG; the fixed version injects the RNG so the test seeds it and gets a repeatable answer — the same dependency-injection move used for the clock in TQ2 and for the network via stubs in TQ3. The rule: a test must own every source of nondeterminism it touches.

3 · Non-functional testing: performance & latency percentiles

Here's the incident that teaches this section. Your service ships. The dashboard says "average response time: 40 ms" — green, well under target, everyone relaxes. Then support tickets trickle in: "the app freezes sometimes." Sometimes. You can't reproduce it, the average is fine, and yet a real slice of users is furious. What's happening is that the average is lying to you. It quietly averages away the handful of requests that took 900 ms, and those requests aren't noise — they're specific users, on specific page loads, having a genuinely bad time. The mean is a single number pretending to describe thousands of very different experiences.

To see why, stop thinking of latency as "a number" and start thinking of it as a distribution — a whole crowd of measurements with a shape. And the shape of real latency is almost always right-skewed: a big pile of fast requests near the median, with a long thin tail of slow ones stretching out to the right (a GC pause, a cold cache, a retry, a noisy neighbour). Because that tail is long, it drags the mean upward and to the right of where most requests actually live — so the mean sits in an empty gap describing no real request. The honest way to describe such a distribution is with percentiles. "p50" (the median) is the value half your requests beat — the typical experience. "p99" is the value 99% of requests beat — which is another way of saying it's how slow your worst 1% are. That worst 1% is not trivia: on a page that makes 100 backend calls, nearly every page load hits someone's p99. This is why real Service Level Objectives are written as "p99 < 300 ms," never "mean < 50 ms."

One more distinction the lab makes concrete, because people conflate them: latency and throughput are different axes, not two words for speed. Latency is how long one request takes (a duration, measured per request). Throughput is how many requests complete per second (a rate, measured across the system). They can even move in opposite directions — batching several requests together to raise throughput often makes each individual request wait longer, worsening tail latency. Optimising one can quietly cost you the other, so a serious performance report always states both, and states latency as a percentile spread rather than a single seductive average.

Everything so far tested functional correctness (does it compute the right answer?). Non-functional testing checks the qualities — speed, capacity, resilience, security, accessibility. The first is performance, and the cardinal rule is: latency is a distribution, not a mean.

Latency is how long one request takes; throughput is how many requests complete per unit time. They are different axes — a batching system can raise throughput while worsening tail latency. Because a few slow requests are what users notice, you report percentiles (p50/p95/p99), never just the mean.

percentiles.py# Non-functional: latency is a DISTRIBUTION, not a mean. Report percentiles.
# p50 = median, p95/p99 = tail. The mean hides the tail that users feel.
def percentile(sorted_data, p):
    # nearest-rank method: smallest value >= p% of the data.
    k = max(0, (len(sorted_data) * p + 99) // 100 - 1)   # ceil(n*p/100)-1, clamped
    return sorted_data[k]

# 100 request latencies in ms: mostly fast, a few very slow (a realistic tail).
latencies = sorted([20] * 90 + [50] * 5 + [200] * 4 + [900] * 1)
mean = sum(latencies) / len(latencies)

print(f"count      : {len(latencies)}")
print(f"mean       : {mean:.1f} ms")
print(f"p50 (median): {percentile(latencies, 50)} ms")
print(f"p95        : {percentile(latencies, 95)} ms")
print(f"p99        : {percentile(latencies, 99)} ms")
print(f"max        : {latencies[-1]} ms")

# Throughput vs latency: throughput = requests / total time (concurrency matters).
total_seconds = 2.0
requests = 1000
print(f"throughput : {requests / total_seconds:.0f} req/s")

count      : 100
mean       : 37.5 ms
p50 (median): 20 ms
p95        : 50 ms
p99        : 200 ms
max        : 900 ms
throughput : 500 req/s

The mean latency is 37.5 ms — but the median (p50) is only 20 ms while p99 is 200 ms and the worst request is 900 ms. The mean is dragged up by the tail and describes no real request. This is why SLOs are written as “p99 < 300 ms”, not “mean < 50 ms”. (Ties to ic1-foundations on latency/throughput/cost.)

4 · Load, stress, soak — and security & accessibility

"We load-tested it" is one of the most misleading sentences in engineering — because it usually means only one of four very different experiments was run, and the other three would have caught the outage. Teams say it, ship, and then get surprised by a memory leak that only shows after six hours, or a Black-Friday traffic spike the service never recovers from. The fix is to realise that "performance testing" isn't one activity; it's a family of tests, and each one asks a different question that the others can't answer.

Think of it like testing a bridge. Load testing asks "does it carry normal rush-hour traffic within spec?" — you ramp up to the expected number of cars, hold steady, and confirm p99 stays under the SLO. Stress testing asks the opposite: "at what weight does it actually break, and does it crack gracefully or collapse all at once?" — you push past capacity on purpose to find the failure point and watch how it degrades (a system that sheds load politely is far better than one that cascades into a crash). Soak testing (endurance) asks "what happens if traffic sits on it for hours or days?" — this is the only one that surfaces slow poisons like memory leaks, growing file-handle counts, or log-disk exhaustion, because they need time, not intensity, to appear. And spike testing asks "if traffic jumps 10× in one second and then drops, does the system absorb it and recover?" — the autoscaling-and-recovery question that a smooth ramp never triggers. Same system, four questions; you need all four because passing one tells you nothing about the others.

Finally, "non-functional" is broader than speed. Two more dimensions separate a hobby project from a professional suite. Security testing probes the qualities an attacker cares about — injection, broken authentication and authorization, leaked secrets, known-vulnerable dependencies (CVEs), and malformed untrusted input (this is exactly where the fuzzing from TQ7 earns its keep). The cheapest, highest-leverage security work is automated and lives in CI: static analysis (SAST tools like Bandit or Semgrep read your source for dangerous patterns), dynamic analysis (DAST tools like OWASP ZAP attack the running app), and dependency scanning (pip-audit checks your installed packages against CVE databases) — manual penetration testing sits on top of that, not instead of it. Accessibility testing (a11y) checks that people using assistive technology can actually use the product, measured against the WCAG standard: sufficient colour contrast, full keyboard navigation, alt text on images, correct ARIA roles, screen-reader-friendly labels — verified with tools like axe-core, Lighthouse, or pa11y. Both dimensions are easy to skip and expensive to bolt on later, which is why the professional instinct is to shift them left — into automated CI checks — long before a release gate.

Performance testing splits into distinct goals, each answering a different question:

Two more non-functional dimensions round out a professional suite:

5 · Testing the untestable: LLM & nondeterministic systems

You've spent this whole track learning that a good test is deterministic — and now you have to test an LLM, which is deterministic's opposite. Ask the same model the same question twice and you get "The order id is 10432." one time and "I found order 10432 in your message." the next. Both are correct. But your entire testing toolkit is built on assertEqual(output, expected), and that assertion just went from a reliable guard to a coin flip — it fails on wording changes that aren't bugs. The naive reaction ("pin temperature to 0 and assert the exact string") is brittle theatre: it breaks on the next model version and tells you nothing about whether the output was actually good.

The mental shift that unlocks this is one you already made when you killed flaky tests: stop asserting the exact answer, and start asserting the properties a correct answer must have. You don't know the precise words the model will use, but you know things that must be true of any acceptable output — and those are testable even when the wording isn't. There are three complementary levels, each borrowed from a technique you've already met:

• Property / invariant checks (from property-based and metamorphic testing, TQ7): assert things that hold for every output regardless of phrasing. The most important for LLMs is grounding — any order id, price, or fact the model returns must actually appear in the input (this is your automated hallucination detector). Other invariants: the output parses as the required format, contains no PII, or shrinks a summary rather than growing it.
• Snapshot / golden checks (from regression testing): pin one representative output as a reference so that when it does change, a human is forced to look and confirm the change was intended, rather than it slipping through silently.
• Rubric / allow-list checks (from acceptance testing): assert the output is one of a small set of accepted shapes — e.g. it either starts with "The order id is " or is exactly "No order id found." — without pinning the exact text.

And when the answer is genuinely open-ended (a paragraph of prose, a plan, an explanation) and no property or shape captures "good," you escalate to LLM-as-judge: a second model scores the output against explicit criteria. That is the bridge out of this software-testing track and into ch05-evaluation, where eval datasets and grader models extend this exact idea — assert properties, not strings — to the messy, subjective outputs that make LLM systems so hard, and so interesting, to test.

An LLM (or any stochastic system) breaks example-based testing: the same prompt can yield different wording, so assertEqual(output, expected) is hopeless. You fall back on the techniques from TQ7 and ch05-evaluation:

test_llm.py# Testing the "untestable": a nondeterministic (e.g. LLM) system where exact
# output can't be asserted. Test PROPERTIES and use SNAPSHOTS, not equality.
import re

def fake_llm_extract(text):
    # Stand-in for a model call: extract an order id. Output wording may vary.
    m = re.search(r"order #?(\d{4,})", text, re.I)
    return f"The order id is {m.group(1)}." if m else "No order id found."

# 1. PROPERTY/INVARIANT check: the extracted id must appear in the input.
def prop_id_grounded(text):
    out = fake_llm_extract(text)
    m = re.search(r"(\d{4,})", out)
    return (m is None) or (m.group(1) in text)   # never hallucinate an id

cases = ["please refund order #10432", "where is ORDER 88888?", "hi there"]
print("grounded on all cases:", all(prop_id_grounded(c) for c in cases))

# 2. SNAPSHOT check: pin the current output; a diff flags an intended change.
snapshot = "The order id is 10432."
print("matches snapshot     :", fake_llm_extract(cases[0]) == snapshot)

# 3. RUBRIC/allow-list check: output must be one of the accepted shapes.
def valid_shape(out):
    return out == "No order id found." or out.startswith("The order id is ")
print("all outputs valid    :", all(valid_shape(fake_llm_extract(c)) for c in cases))

grounded on all cases: True
matches snapshot     : True
all outputs valid    : True

Even though the model's exact wording is not asserted, three checks hold it accountable: a grounding property (any id it returns must appear in the input — no hallucinated ids), a snapshot for regression, and a shape allow-list. This is the bridge from software testing into ch05-evaluation, where LLM-as-judge and eval datasets extend the same idea to open-ended output.

Knowledge check

import unittest

def is_business_hours(clock):
    return 9 <= clock.hour() < 17

class StubClock:
    def __init__(self, h): self._h = h
    def hour(self): return self._h

class TestHours(unittest.TestCase):
    def test_daytime(self):  self.assertTrue(is_business_hours(StubClock(10)))
    def test_night(self):    self.assertFalse(is_business_hours(StubClock(23)))

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 2 tests OK

import unittest
from unittest.mock import MagicMock

def checkout(cart, gateway):
    gateway.charge(sum(cart))

class SpyGateway:
    def __init__(self): self.charges = []
    def charge(self, amt): self.charges.append(amt)

class TestCheckout(unittest.TestCase):
    def test_with_spy(self):
        spy = SpyGateway(); checkout([10, 5], spy)
        self.assertEqual(spy.charges, [15])
    def test_with_mock(self):
        m = MagicMock(); checkout([10, 5], m)
        m.charge.assert_called_once_with(15)

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 2 tests OK

import random, unittest

# BEFORE: reads global rng + real date -> flaky
# AFTER: inject both
def discount_of_the_day(rng, weekday):
    base = 10 if weekday < 5 else 20      # bigger weekend discount
    return base + rng.randint(0, 5)

class TestDiscount(unittest.TestCase):
    def test_repeatable(self):
        a = discount_of_the_day(random.Random(7), weekday=2)
        b = discount_of_the_day(random.Random(7), weekday=2)
        self.assertEqual(a, b)               # deterministic
    def test_weekend_higher_floor(self):
        self.assertGreaterEqual(discount_of_the_day(random.Random(0), 6), 20)

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 2 tests OK, every run

import unittest

def percentile(s, p):
    k = max(0, (len(s) * p + 99) // 100 - 1)     # ceil(n*p/100)-1
    return s[k]

class TestLatency(unittest.TestCase):
    def test_tail(self):
        s = sorted([15] * 95 + [400] * 5)         # 5% slow tail
        mean = sum(s) / len(s)
        p50, p95, p99 = (percentile(s, q) for q in (50, 95, 99))
        self.assertLessEqual(p50, p95)
        self.assertLessEqual(p95, p99)
        self.assertGreater(mean, p50)            # mean dragged up by tail
        self.assertEqual((p50, p99), (15, 400))

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 1 test OK

import random, unittest

def summarize(text, rng):
    words = text.split()
    k = max(1, len(words) // 2)
    return " ".join(rng.sample(words, min(k, len(words))))

class TestSummary(unittest.TestCase):
    TEXT = "the quick brown fox jumps"
    def test_invariants(self):
        for seed in range(50):
            out = summarize(self.TEXT, random.Random(seed))
            self.assertLessEqual(len(out.split()), len(self.TEXT.split()))
            self.assertTrue(set(out.split()) <= set(self.TEXT.split()))
    def test_snapshot(self):
        out = summarize(self.TEXT, random.Random(0))
        self.assertEqual(out, summarize(self.TEXT, random.Random(0)))  # stable

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 2 tests OK

def release_gate(metrics, slos):
    breaches = []
    if metrics["p99_ms"] > slos["p99_ms"]:
        breaches.append(f"p99 {metrics['p99_ms']}ms > {slos['p99_ms']}ms")
    if metrics["error_rate"] > slos["error_rate"]:
        breaches.append(f"errors {metrics['error_rate']} > {slos['error_rate']}")
    if metrics["soak_mem_growth_pct"] > slos["soak_mem_growth_pct"]:
        breaches.append(f"mem growth {metrics['soak_mem_growth_pct']}% (leak?)")
    return (not breaches), breaches

slos = {"p99_ms": 300, "error_rate": 0.01, "soak_mem_growth_pct": 5}
good = {"p99_ms": 240, "error_rate": 0.004, "soak_mem_growth_pct": 2}
bad  = {"p99_ms": 520, "error_rate": 0.03,  "soak_mem_growth_pct": 18}
print(release_gate(good, slos))   # (True, [])
print(release_gate(bad, slos))    # (False, [...three breaches...])