Testing theory & coverage
The theory a practical suite skips: the test taxonomy, the pyramid vs the trophy, rigorous test-case design (equivalence classes, boundary values, decision tables, state machines), and what statement, branch, path and MC-DC coverage really prove — all runnable.
Learning objectives
- Place any test in the taxonomy — unit / integration / system / acceptance, and black / white / grey box.
- Choose the right shape: the testing pyramid vs the testing trophy, and why.
- Design test cases with equivalence partitioning and boundary-value analysis — and find an off-by-one.
- Build decision tables and state-transition tests for logic and stateful code.
- Compute statement, branch, path, condition and MC-DC coverage by hand, and explain why 100% statement coverage is not correctness.
unittest) — no pytest needed. The tools it references (coverage.py, pytest --cov) are noted where a real project would use them.1 · The testing taxonomy
Here's an argument you've probably lost. A teammate says "we have 90% coverage, we're well tested." You have a nagging feeling that's not the whole story, but you can't name why — so the point goes unmade and a wiring bug ships. The problem wasn't your instinct; it was vocabulary. You didn't have the words to say "those are all unit tests — we have almost no integration coverage, which is exactly where this class of bug lives." A taxonomy isn't academic box-ticking; it's the language that lets you diagnose which kind of confidence a test suite actually buys you.
Think of testing like inspecting a new house. You can check a single fixture — does this one tap turn on? That's a unit test: one component in isolation. You can check that the tap, the pipe, and the water heater work together to deliver hot water — that's integration: two or more real parts wired up. You can walk the whole house as a buyer would, flushing toilets and flipping every switch — that's a system (end-to-end) test. And you can ask the entirely different question, "is this even the house I asked the builder for?" — that's acceptance. Each answers a real question, and none substitutes for another: a house with perfect taps can still have the wrong number of bedrooms.
That first axis — scope — is only half the map. The second, independent axis is how much the test can see. A black-box test knows only the interface: inputs go in, outputs come out, and it judges against the spec, never peeking at the source. A white-box test reads the code and is designed around its structure — this branch, that loop — which is how you deliberately chase coverage. Grey box sits between: you know some internals (a database schema, a cache) and use that partial knowledge to write sharper tests. The common mistake is collapsing these two axes into one — assuming "unit = white box, E2E = black box." They're genuinely independent: you can write a black-box unit test (call a pure function, assert its output, never read its body) or a white-box integration test. Naming both axes for any test you write is the skill this section trains, because the two answers together tell you what the test does — and does not — prove.
Before writing a test you should be able to name what kind it is on two independent axes. The first axis is scope — how much of the system it exercises. The second is knowledge of internals — whether the test sees the code or only the interface.
Read the two arrows first: the horizontal axis is scope (how much of the system runs), the vertical axis is how much code the test can see.
- The bottom row of coloured boxes is the scope axis, left to right: a unit pokes one function, integration wires real parts, system runs the whole app, acceptance checks it against business criteria.
- The three stacked bands are the visibility axis: black box sees only inputs/outputs, grey box knows some internals, white box is designed around the source.
- The payoff is that the two axes are independent — nothing forces "unit = white box." You can place a test anywhere on the grid, which is why naming both coordinates is the skill.
In short: Pin every test to a point on both axes — scope and visibility — and you can say exactly what it does and does not prove.
| Level | Scope | Speed / count | Catches |
|---|---|---|---|
| Unit | one function / class in isolation | ms · thousands | logic errors, edge cases |
| Integration | two+ real components together | 10s–100s ms · hundreds | contract / wiring bugs |
| System (E2E) | the whole deployed app | seconds · tens | flow / config / environment bugs |
| Acceptance | system vs business criteria | seconds · few | “wrong thing built” |
The orthogonal axis is how much the test knows about the implementation:
| Approach | Sees | Design driven by | Typical use |
|---|---|---|---|
| Black box | only inputs/outputs | spec & requirements | acceptance, API tests |
| White box | the source code | code structure & coverage | unit tests, coverage design |
| Grey box | partial internals | spec + known internals | integration, security tests |
2 · The testing pyramid & the trophy
You've felt this one on a bad morning. You push a one-line fix, and the CI suite grinds for twenty-five minutes before going red on a test that has nothing to do with your change — it "flaked," someone says, so you click re-run and wait another twenty-five. When a suite is slow and unreliable, people stop trusting it: they merge on a re-run, they skip it locally, they eventually turn the flaky ones off. The suite that was supposed to protect you has become a tax you route around. The shape of your test suite — how many of each kind you have — is what decides whether it feels like a safety net or a toll booth.
The core physics is simple: the wider a test's scope, the more it costs and the flakier it gets. A unit test pokes one function in memory — it runs in a millisecond and either passes or fails for one crisp reason. An end-to-end test spins up the whole app, a real browser, a real database, the network — so it's a thousand times slower and can fail for a hundred unrelated reasons (a slow response, a timing race, a stale fixture). That cost gradient is why the classic testing pyramid is shaped the way it is: a broad base of many fast, stable unit tests, a narrower band of integration tests, and only a thin cap of expensive E2E tests. You want most of your confidence to come from the cheap, reliable layer.
So why does the competing testing trophy put its bulk at the integration layer instead? Because the pyramid quietly assumes your value lives in rich, branchy domain logic — the kind pure unit tests nail. Flip that assumption: a modern web service is often a thin skin of handlers over a database, with the real logic being how the pieces connect. Unit-test that in isolation and you mock the database away — leaving a hundred green units and a broken system, because every actual bug was in the wiring you stubbed out. There the trophy is right: test the handler against a real DB, and stand static analysis (types, lint) as a broad base underneath. The trophy isn't a rebuttal of the pyramid; it's the same cost/return reasoning applied to a different kind of code. The mistake to avoid is treating either shape as dogma — and the specific catastrophe to recognize is the ice-cream cone: the pyramid flipped upside down, mostly manual and E2E, almost no units, which gives you the slowest, flakiest, least-diagnostic suite possible.
How many of each level should you have? Two competing shapes answer this. The testing pyramid (Cohn) says: many fast unit tests at the base, fewer integration tests, very few slow E2E tests at the tip — because cost and flakiness rise as you go up.
Read it top-to-bottom as three stacked layers, with the downward arrows showing that each level rests on the one below it.
- The top box is E2E (few), tagged "slow, brittle" — the whole app end-to-end, so you keep very few of these at the tip.
- The middle box is Integration ("medium"), and the bottom box is Unit (many), tagged "fast, stable" — the arrows point downward toward it because that broad, cheap base is where most of your tests should live.
- The payoff is the cost gradient the labels spell out: as you climb from Unit to E2E the tests get slower and flakier, which is exactly why the reliable layer is widest at the bottom and the expensive layer is a thin cap.
In short: Many fast unit tests at the base, few slow E2E tests at the tip — because cost and flakiness rise as you go up.
The testing trophy (Dodds) argues that for I/O-heavy, well-typed applications the biggest return sits at the integration layer, with static analysis (types, lint) as a broad base. It is not a contradiction — it is a different cost/return profile:
| Shape | Bulk of tests at | Best when | Risk if misapplied |
|---|---|---|---|
| Pyramid | unit | rich domain logic, pure functions | many green units, broken wiring |
| Trophy | integration | thin logic over I/O, strong types | slow suite, harder to localize failures |
3 · Equivalence partitioning & boundary-value analysis
The most expensive bugs hide behind the most reasonable-looking tests. You wrote a function that validates an age between 18 and 65. You tested it with 40 — it returned True, green tick, done. It shipped, and three weeks later a support ticket says a 65-year-old customer can't sign up. The test wasn't wrong; it was uninformative. Picking 40 told you nothing about the two places bugs actually live: the edges. This section is about choosing test inputs so that a small handful does the work of thousands — and so that the ones you pick are the ones that catch real bugs.
Start with the impossible problem: an age is an integer, so "test every input" means billions of cases. You obviously can't. But you don't need to, because most of those inputs are redundant — the function treats 40 exactly the way it treats 41, 42, and 39. That's the insight behind equivalence partitioning: carve the input space into classes where the code should behave identically, then test just one representative from each class. For the age check, the natural partitions are "too young" (below 18), "valid" (18–65), and "too old" (above 65) — three classes, so three well-chosen inputs cover the meaningful behaviours instead of billions of pointless ones. The representative is a stand-in: if 40 works, you trust that 41 does too, because they're in the same class.
But equivalence partitioning has a blind spot, and boundary-value analysis (BVA) exists to fill it. Bugs don't cluster in the calm middle of a class — they cluster at the edges, where one range meets the next, because that's where a programmer typed < when they meant <=, or wrote 18 where the spec said 17. Off-by-one errors are the single most common logic mistake in the industry, and they live exactly on the boundary. So BVA says: for a range [lo, hi], don't just test a middle value — test the six points that straddle both edges: lo-1, lo, lo+1 and hi-1, hi, hi+1. For 18–65 that's 17, 18, 19, 64, 65, 66 — the just-outside, just-on, and just-inside points at each end. This is where the earlier "test with 40" fails and BVA wins: 40 sits deep in the valid class and can never reveal a mishandled boundary, whereas testing 65 directly interrogates the <= vs < decision. The common trap is testing only on the boundary (just 18 and 65) and skipping the neighbours — but a boundary bug can push the edge one step either way, so you need the point just outside and just inside to pin it down. Equivalence partitioning shrinks the input space; BVA aims what's left at the edges where bugs actually breed. The lab below turns exactly this into an off-by-one that a happy-path test sails right past.
You cannot test every input. Equivalence partitioning divides the input space into classes where the program should behave the same, so one representative per class suffices. Boundary-value analysis (BVA) then adds the fact that bugs cluster at the edges of those classes — so for a range [lo, hi] you test lo-1, lo, lo+1, hi-1, hi, hi+1. The lab finds a real off-by-one that a happy-path test misses entirely.
Read the number line left to right: the whole range of ages is chopped into three coloured segments, and small dots sit on the points BVA actually tests.
- The three coloured bars are the equivalence classes: red "too young" and "too old" (reject), green "valid" (accept). One representative from each class is enough to cover the middle of it.
- The six dots are the boundary-value points —
lo-1, lo, lo+1andhi-1, hi, hi+1. Blue dots fall just-inside/just-on a valid edge; amber dots (17, 66) fall just outside. - The payoff is the point at 65: it is exactly where a
<written for<=flips the answer, so a test at 65 interrogates that decision while a comfortable middle value (40) never touches it.
In short: Partitioning shrinks billions of inputs to a handful; BVA aims that handful at the edges where off-by-ones breed.
bva.py# The classic off-by-one: "valid age is 18..65 inclusive". A buggy check uses
# `< 65` instead of `<= 65`, silently rejecting the upper boundary (65).
def eligible_buggy(age):
return 18 <= age < 65 # BUG: excludes 65
def eligible_fixed(age):
return 18 <= age <= 65 # correct: 65 is valid
# Boundary-Value Analysis test set: for a range [lo, hi] test
# lo-1, lo, lo+1, hi-1, hi, hi+1 (the 6 points where bugs cluster).
def bva_points(lo, hi):
return [lo - 1, lo, lo + 1, hi - 1, hi, hi + 1]
points = bva_points(18, 65) # [17, 18, 19, 64, 65, 66]
expected = [False, True, True, True, True, False]
# A naive "happy-path" suite only tries a middle value and both pass:
print("happy-path only :", eligible_buggy(40) == True, eligible_fixed(40) == True)
# BVA catches the bug precisely at the upper boundary (65):
buggy = [eligible_buggy(a) for a in points]
fixed = [eligible_fixed(a) for a in points]
print("boundary points :", points)
print("buggy results :", buggy)
print("fixed results :", fixed)
print("first mismatch :", next(p for p, b, e in zip(points, buggy, expected) if b != e))
happy-path only : True True
boundary points : [17, 18, 19, 64, 65, 66]
buggy results : [False, True, True, True, False, False]
fixed results : [False, True, True, True, True, False]
first mismatch : 65
The happy-path line passes for both the buggy and fixed versions — a middle value like 40 tells you nothing. Only the boundary point 65 exposes that < 65 wrongly rejects the inclusive upper bound. This is the single highest-yield test-design technique for exams and real bugs alike.
pip install hypothesis) will search the boundaries for you by shrinking failing inputs to the minimal case — covered in TQ7.4 · Decision-table testing
Here's the bug that gets through the most careful reviewer. A login rule says "allow if the password is valid, the account is active, and it isn't locked." You test a valid login — works. You test a wrong password — denied. You feel thorough. Then a locked account with a valid password somehow gets in, because nobody tested that particular combination. It's not that the logic was hard; it's that three yes/no conditions produce more combinations than a human tests by intuition, and the one you skip is always the one that bites. Decision-table testing is the discipline that makes skipping a combination impossible.
The trap with combinations is that they grow faster than your patience. One condition has 2 cases. Two conditions have 4. Three have 8. The formula is 2n for n boolean conditions — exponential — and the human brain quietly gives up somewhere around four, testing the "obvious" combinations and assuming the rest. A decision table refuses that shortcut: you write one row per rule, listing every combination of conditions down the left and its expected action on the right, so the table itself guarantees completeness. If there are eight combinations, there are eight rows; a missing row is visible at a glance. Crucially this is a black-box technique — you build the table from the specification, from what the rule is supposed to do, not from reading the code. That's the whole point: it catches the case the code forgot to handle, which no amount of white-box coverage of the existing code can ever reveal.
Real specs, of course, aren't always eight tidy rows — a policy might have six conditions (64 rows) with many combinations that don't matter. That's where don't-care entries earn their keep: once you notice that "if the password is invalid, nothing else matters — it's always DENY," you can collapse all those rows into one, marking the irrelevant conditions with a dash. But the discipline is to derive the full table first and collapse second, so every collapse is a justified decision rather than a combination you simply never thought of. Skip straight to the collapsed table and you're back to guessing which conditions matter — which is exactly the intuition the technique was meant to replace. The lab enumerates all 23 = 8 rules for the login policy so you can see the one true ALLOW row surrounded by seven DENYs.
When behaviour depends on a combination of conditions, a decision table enumerates every rule (2n for n boolean conditions) and its expected action, so you cannot forget a combination. It is a black-box technique: it comes from the spec, not the code.
Read it top-to-bottom as a truth table: each row is one complete combination of the three conditions, and the coloured cell on the right is the action the spec demands.
- The three left columns are the conditions
valid_pw,active,locked, each T or F. Eight rows appear because three booleans give2³ = 8combinations. - The right column is the expected action, coloured: red DENY or green ALLOW. It comes from the specification, not the code — that is what makes this black-box.
- The payoff is the single green row (
T,T,F): valid password, active account, not locked. Every other combination denies — and because there is a row for each, the combination a human would skip is impossible to miss.
In short: Enumerate all 2ⁿ rows first; the completeness is the point, and you only collapse don't-care rows afterwards.
decision_table.py# Decision-table testing: enumerate rules over conditions, derive expected action.
# Login policy: allow only if (valid_password AND account_active AND NOT locked).
from itertools import product
def decide(valid_pw, active, locked):
return "ALLOW" if (valid_pw and active and not locked) else "DENY"
conditions = ["valid_pw", "active", "locked"]
print(f"{'valid_pw':>9} {'active':>7} {'locked':>7} | action")
allow_rows = 0
for valid_pw, active, locked in product([True, False], repeat=3):
action = decide(valid_pw, active, locked)
allow_rows += action == "ALLOW"
print(f"{str(valid_pw):>9} {str(active):>7} {str(locked):>7} | {action}")
print("rules:", 2**len(conditions), "| ALLOW rules:", allow_rows)
valid_pw active locked | action
True True True | DENY
True True False | ALLOW
True False True | DENY
True False False | DENY
False True True | DENY
False True False | DENY
False False True | DENY
False False False | DENY
rules: 8 | ALLOW rules: 1
Three conditions give 23 = 8 rules; exactly one yields ALLOW. In practice you collapse rules whose outcome is independent of a condition (don't-care entries) to shrink the table — but you derive it in full first so the collapse is justified.
5 · State-transition testing
You know this bug from your own bank app. You tap "Pay" twice because the first tap seemed to hang — and get charged twice. Or you hit the browser Back button after checking out and the app cheerfully lets you re-submit the order. The code handled the intended sequence perfectly; what it never considered was the same action arriving in a state where it shouldn't be allowed. Stateful bugs aren't about a single wrong output — they're about the system doing the right thing in the wrong moment. State-transition testing is how you hunt them systematically instead of stumbling on them in production.
The mental model that unlocks this is to stop thinking of your object as "code with variables" and start seeing it as a finite-state machine: a small set of named states, and events that move it from one state to another. A subway turnstile is the textbook example because it's so concrete: it's either LOCKED or UNLOCKED, and two events act on it — inserting a coin and giving it a push. The revealing part is that the same event does different things in different states: a push while UNLOCKED lets you through and re-locks the gate, but a push while LOCKED must be refused. Once you draw the machine — states as boxes, events as arrows — the test cases stop being a matter of imagination and become a matter of enumeration: one test per arrow you drew.
And here's the arrow juniors always forget. It's easy to test the legal path — coin, then push, then through — because that's the story you had in mind when you wrote the code. The bugs live in the illegal transitions: the events that arrive in a state where they make no sense. Pushing a locked turnstile, inserting a second coin, paying an already-paid invoice, cancelling a shipped order. The rule that separates robust code from fragile code is that an illegal event must be explicitly rejected — not silently swallowed. Silent swallowing is the trap: the code doesn't crash, so it looks fine, but it has quietly entered a nonsensical state (charged twice, double-shipped) that surfaces as a data-corruption ticket weeks later. The systematic guard is a transition table with one row for every (state, event) pair — including the illegal pairs, each with its expected rejection — so that no arrow, legal or illegal, goes untested. The lab below encodes exactly the turnstile machine and makes a point of asserting that a push while LOCKED returns BLOCKED rather than passing through.
Stateful code is a finite-state machine: the same event does different things in different states. State-transition testing covers each legal transition and each illegal event (which must be rejected, not silently swallowed). The turnstile is the canonical example.
Read it left to right: two boxes are the states of a turnstile, and the arrow is the event that moves it from one to the other.
- The left box is the LOCKED state — the gate's starting point, where a push should be refused.
- The arrow pointing right is the
coinevent, and the green right box is the UNLOCKED state it leads to — coin always unlocks the gate. - The payoff is what the picture doesn't draw: a push while LOCKED (the illegal event) and the return from UNLOCKED back to LOCKED after a valid push — the exact transitions the tests below have to pin down so none is missed.
In short: Two states, one drawn transition — state-transition testing checks every (state, event) pair, including the illegal ones.
state_transition.pyimport unittest
# State-transition testing: a turnstile FSM. States: LOCKED, UNLOCKED.
# Events: coin, push. Illegal transitions must be rejected, not silently ignored.
class Turnstile:
def __init__(self):
self.state = "LOCKED"
def coin(self):
self.state = "UNLOCKED" # coin always unlocks
def push(self):
if self.state == "UNLOCKED":
self.state = "LOCKED" # a valid push passes through, re-locks
return "PASS"
return "BLOCKED" # pushing a locked gate is blocked
class TestTurnstile(unittest.TestCase):
def setUp(self): self.t = Turnstile()
def test_happy_path(self):
self.t.coin()
self.assertEqual(self.t.push(), "PASS")
self.assertEqual(self.t.state, "LOCKED")
def test_push_while_locked_is_blocked(self):
self.assertEqual(self.t.push(), "BLOCKED") # illegal event handled
self.assertEqual(self.t.state, "LOCKED")
def test_two_coins_still_one_pass(self):
self.t.coin(); self.t.coin() # coin is idempotent-ish
self.assertEqual(self.t.push(), "PASS")
if __name__ == "__main__":
unittest.main(verbosity=2)
test_happy_path (__main__.TestTurnstile.test_happy_path) ... ok
test_push_while_locked_is_blocked (__main__.TestTurnstile.test_push_while_locked_is_blocked) ... ok
test_two_coins_still_one_pass (__main__.TestTurnstile.test_two_coins_still_one_pass) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
test_push_while_locked_is_blocked is the one juniors forget: it asserts the illegal event (push while LOCKED) is handled gracefully. A transition table with one row per (state, event) pair is the systematic way to guarantee none is missed.
6 · Coverage theory: statement → branch → path → MC-DC
"We're at 100% coverage" is one of the most reassuring lies in software. A manager sees the green badge and signs off; a bug ships in a code path that the coverage report swore was tested. The report wasn't lying about what it measured — it was measuring the wrong thing, and nobody knew the difference between the kinds of coverage. This section is about that difference, because "coverage" is not one number: it's a ladder of increasingly strict questions, and knowing which rung you're standing on is the difference between real confidence and a green badge that means almost nothing.
The weakest rung is statement coverage: did every line of code run at least once? It sounds thorough until you meet an if with no else. Consider if user.is_admin: grant_access(). A single test with an admin user runs both lines — 100% statement coverage — while never once checking what happens to a non-admin. The dangerous branch (the one that should deny access) was never taken, yet the report is a perfect green. That's why the next rung up, branch coverage (also called decision coverage), asks the stronger question: did every decision go both True and False? Branch coverage subsumes statement coverage — take every branch and you've necessarily run every line — which is why it's the target most CI pipelines actually enforce. The relationship matters: the ladder is nested, each rung guaranteeing the ones below it.
Above branch coverage the questions get subtler and aimed at compound decisions like if x > 0 and y > 0. Condition coverage asks that each atomic condition (x>0, y>0) be seen both true and false — but, surprisingly, that alone doesn't guarantee branch coverage, because short-circuit evaluation can flip individual conditions without ever flipping the overall decision. MC-DC (Modified Condition/Decision Coverage) closes that gap with a demanding rule: every condition must be shown to independently flip the final decision, holding the others fixed. It's expensive enough that it's mandated only where lives are at stake — the DO-178C avionics standard. At the top sits path coverage — every possible route through the code — which is complete but usually infeasible, since loops and branches multiply paths exponentially. The one lesson to carry out of the whole ladder is the trap the lab makes concrete: coverage is necessary but not sufficient. It proves a line ran; it never proves you asserted the right thing — a test with no assertion at all still counts as full coverage — and it's blind to code you forgot to write. Chase it to find untested gaps, never to certify correctness; measuring the strength of your assertions is a different tool (mutation testing, in TQ7).
Coverage is a white-box measure of what your tests execute. The hierarchy, from weakest to strongest:
| Criterion | Requires | Subsumes | Note |
|---|---|---|---|
| Statement | every line executed once | — | weakest; misses untaken branches |
| Branch (decision) | every decision True and False | statement | the usual CI target |
| Condition | each atomic condition both ways | — | alone doesn't imply branch |
| Condition/decision (C/DC) | conditions + decisions both ways | branch | stronger combo |
| MC-DC | each condition independently flips the decision | C/DC | DO-178C avionics standard |
| Path | every path through the code | all above | exponential; usually infeasible |
The lab makes the key point concrete: a suite can hit every statement (100% statement coverage) yet leave a branch outcome untaken — so 100% statement coverage is not correctness.
Read it from the inside out: the innermost box is the weakest, cheapest criterion, and every ring around it is a strictly stronger one that contains the weaker one.
- The green core is statement coverage — did every line run once? Weakest, because an
ifwith noelsecan be "100%" while the untaken path is never checked. - Moving out, branch (every decision True and False) encloses statement; C/DC and then MC-DC (each condition independently flips the decision) add strictly more for compound conditions.
- The payoff is the nesting itself: an outer ring subsumes every inner one, so achieving it guarantees the weaker ones for free. The outermost path ring is complete but usually infeasible — loops make routes explode.
In short: Bigger ring = stronger proof but higher cost; branch is the practical CI target, and even the outer rings only prove code ran, not that you asserted the right thing.
coverage_theory.pyimport unittest
# The function under test. Two conditions in one `if` (a compound decision),
# plus a branch that returns early.
def classify(x, y):
if x > 0 and y > 0: # decision D1 with conditions C1:(x>0) C2:(y>0)
return "both-pos"
if x == y: # decision D2
return "equal"
return "other"
# --- Coverage models (computed by hand so you see exactly what each counts) ---
# Statements: the 4 executable lines inside classify.
# Branches: each decision's True AND False outcome -> 4 branch outcomes.
# Conditions: each atomic condition True AND False -> C1,C2 each way + D2.
STATEMENTS = {"if1", "ret_both", "if2", "ret_equal_or_other"}
BRANCHES = {"D1-T", "D1-F", "D2-T", "D2-F"}
def run(tests):
stmt, brch = set(), set()
for x, y in tests:
stmt.add("if1")
if x > 0 and y > 0:
stmt.add("ret_both"); brch.add("D1-T")
else:
brch.add("D1-F")
stmt.add("if2")
if x == y:
stmt.add("ret_equal_or_other"); brch.add("D2-T")
else:
stmt.add("ret_equal_or_other"); brch.add("D2-F")
return stmt, brch
def pct(got, whole):
return f"{len(got & whole)}/{len(whole)} = {100*len(got & whole)//len(whole)}%"
# Suite A: two tests. Reaches every *statement* -> 100% statement coverage...
suite_a = [(1, 1), (-2, -2)]
sa_stmt, sa_brch = run(suite_a)
print("suite A statement:", pct(sa_stmt, STATEMENTS))
print("suite A branch :", pct(sa_brch, BRANCHES), "-> misses", sorted(BRANCHES - sa_brch))
# ...yet it never takes the False branch of D2, so a bug there hides.
# Suite B adds a non-equal negative case to cover every branch outcome.
suite_b = [(1, 1), (-2, -2), (-2, -3)]
sb_stmt, sb_brch = run(suite_b)
print("suite B branch :", pct(sb_brch, BRANCHES))
print("100% statement != correct:", len(sa_stmt) == len(STATEMENTS), "yet branch only", pct(sa_brch, BRANCHES))
suite A statement: 4/4 = 100%
suite A branch : 3/4 = 75% -> misses ['D2-F']
suite B branch : 4/4 = 100%
100% statement != correct: True yet branch only 3/4 = 75%
Suite A executes all 4 statements — a green 100% on a statement-coverage report — but never takes the False branch of D2 (the x != y case), so it sits at 75% branch coverage. A bug in that untaken path is invisible until Suite B adds the non-equal negative case. In a real project you would get these numbers from coverage.py / pytest --cov --cov-branch; MC-DC needs a dedicated tool.
✓ Checkpoint — you can move on when you can…
- Classify a test by scope and by black/white/grey box.
- Choose pyramid vs trophy for a given system and justify it.
- Derive a BVA test set for a range and explain which point catches an off-by-one.
- Build a decision table and a state-transition test set.
- Compute statement and branch coverage by hand and state why one test can give 100% statement but 25% branch.
Knowledge check
The coverage lab shows Suite A reaching 4/4 statements (100% statement coverage) but only 3/4 branches. Explain in coverage terms why “100% statement coverage” still does not prove classify is correct.
Show answer
D2 (the x != y path), so an error there is never exercised — hence 75% branch coverage despite 100% statement coverage. Branch coverage subsumes statement coverage precisely because it demands both outcomes of every decision. And even 100% branch coverage only shows lines ran, not that assertions checked the right result — which is why mutation testing (TQ7) exists.You have a well-typed web service that is mostly thin handlers over a database, with little domain logic. Which test shape does the theory favour — pyramid or trophy — and what is the main risk of the other?
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Partitioning the input space is the first move in test design — one representative per class beats random guessing.
Your task: Given letter_grade(score) mapping 0–100 to A/B/C/D/F, list the equivalence classes and write one unittest assertion per class plus the invalid-input class.
Requirements:
- Identify each score band as a class (A:90–100, B:80–89, …, F:0–59) plus an invalid class (<0 or >100)
- One representative value per class, asserted with
assertEqual - The invalid class asserts a raised
ValueErrorviaassertRaises - Runs under
unittest.mainwith no external deps
💡 Hint: Pick a mid-band representative for each class; you are testing the class, not the boundary yet (that is the next rung).
Show solution
One representative per equivalence class, plus the invalid class:
import unittest
def letter_grade(score):
if not 0 <= score <= 100:
raise ValueError("score must be 0-100")
if score >= 90: return "A"
if score >= 80: return "B"
if score >= 70: return "C"
if score >= 60: return "D"
return "F"
class TestGrade(unittest.TestCase):
def test_classes(self):
for score, grade in [(95,"A"),(85,"B"),(75,"C"),(65,"D"),(30,"F")]:
self.assertEqual(letter_grade(score), grade)
def test_invalid_class(self):
with self.assertRaises(ValueError): letter_grade(150)
if __name__ == "__main__":
unittest.main(verbosity=2) # 2 tests OK
Five valid classes plus one invalid class = six representatives. Boundaries (89 vs 90) are a separate technique — next rung.
Context: Bugs cluster at class edges; BVA tests lo-1, lo, lo+1, hi-1, hi, hi+1.
Your task: Write bva_points(lo, hi) and use it to test a range check in_range(x) for 1–100 inclusive, then show the set catches a < 100 off-by-one.
Requirements:
bva_points(lo, hi)returns the six boundary points- Test both a buggy (
< 100) and a correct (<= 100) implementation over those points - Assert the two disagree exactly at the upper boundary
100 - Runs under
unittest
💡 Hint: The mismatch will be at hi itself; a middle value never reveals it.
Show solution
The six points, and the disagreement pinned at the upper boundary:
import unittest
def bva_points(lo, hi): return [lo-1, lo, lo+1, hi-1, hi, hi+1]
def in_range_buggy(x): return 1 <= x < 100 # BUG
def in_range_ok(x): return 1 <= x <= 100
class TestBVA(unittest.TestCase):
def test_boundary_mismatch(self):
pts = bva_points(1, 100) # [0,1,2,99,100,101]
diffs = [p for p in pts if in_range_buggy(p) != in_range_ok(p)]
self.assertEqual(diffs, [100]) # only 100 differs
if __name__ == "__main__":
unittest.main(verbosity=2) # 1 test OK
The buggy < 100 rejects 100; the correct version accepts it. BVA localizes the off-by-one to a single point.
Context: Combinatorial logic needs every rule enumerated so none is forgotten.
Your task: A shipping-cost rule: free shipping if member AND (cart ≥ 50 OR coupon). Build the full decision table over the three booleans and assert the number of FREE rules.
Requirements:
- Enumerate all 23 = 8 rules with
itertools.product - Compute the action per rule from the policy
- Assert the count of FREE-shipping rules matches the truth table
- Runs under
unittest
💡 Hint: member must be True in every FREE row; then either cart≥50 or coupon.
Show solution
Enumerate, then assert the FREE count against the derived truth table:
import unittest
from itertools import product
def free_shipping(member, big_cart, coupon):
return member and (big_cart or coupon)
class TestDecisionTable(unittest.TestCase):
def test_free_rules(self):
rules = list(product([True, False], repeat=3))
free = [r for r in rules if free_shipping(*r)]
# member=T and (big_cart or coupon): (T,T,T),(T,T,F),(T,F,T) -> 3
self.assertEqual(len(free), 3)
self.assertEqual(len(rules), 8)
if __name__ == "__main__":
unittest.main(verbosity=2) # 1 test OK
Deriving the table in full guarantees no combination is missed; you may then collapse don't-care rows for the final suite.
Context: Every legal transition and every illegal event must be tested for a stateful object.
Your task: Model a player FSM (STOPPED/PLAYING/PAUSED) with events play/pause/stop, and write tests covering each legal transition plus one illegal event (pause while STOPPED).
Requirements:
- States STOPPED, PLAYING, PAUSED with play/pause/stop events
- One test per legal transition asserting the resulting state
- One test asserting an illegal event (pause while STOPPED) leaves state unchanged and is rejected
- Use
setUpfor a fresh player; runs underunittest
💡 Hint: Make illegal events raise or return a sentinel — either way, assert the state did not change.
Show solution
Cover legal transitions and the illegal event explicitly:
import unittest
class Player:
def __init__(self): self.state = "STOPPED"
def play(self): self.state = "PLAYING"
def pause(self):
if self.state != "PLAYING":
raise RuntimeError("can only pause while playing")
self.state = "PAUSED"
def stop(self): self.state = "STOPPED"
class TestPlayer(unittest.TestCase):
def setUp(self): self.p = Player()
def test_play_then_pause(self):
self.p.play(); self.p.pause()
self.assertEqual(self.p.state, "PAUSED")
def test_stop_from_playing(self):
self.p.play(); self.p.stop()
self.assertEqual(self.p.state, "STOPPED")
def test_illegal_pause_when_stopped(self):
with self.assertRaises(RuntimeError): self.p.pause()
self.assertEqual(self.p.state, "STOPPED") # unchanged
if __name__ == "__main__":
unittest.main(verbosity=2) # 3 tests OK
A transition table (row per state×event) is the systematic checklist ensuring no legal or illegal edge is missed.
Context: Teams gate on branch coverage precisely because statement coverage over-reports safety.
Your task: Instrument a two-branch function and show a suite that reaches 100% statement but under 100% branch coverage, then add the test that closes the branch gap.
Requirements:
- Track executed statements and branch outcomes in sets
- Suite 1 achieves 100% statement but <100% branch
- Suite 2 adds the case that reaches the missing branch outcome to hit 100% branch
- Print both percentages; runs under
unittestor as a script
💡 Hint: A guard clause that returns early is the classic place statement coverage lies about branch coverage.
Show solution
Statement coverage can read 100% while a branch outcome is still missing:
def run(tests):
stmt, brch = set(), set()
for n in tests:
stmt.add("enter")
if n < 0:
stmt.add("neg"); brch.add("T")
else:
brch.add("F")
stmt.add("exit")
return stmt, brch
STMT = {"enter", "neg", "exit"}; BR = {"T", "F"}
s1, b1 = run([-1]) # only the negative case
print("suite1 stmt", f"{len(s1&STMT)}/{len(STMT)}", "branch", f"{len(b1&BR)}/{len(BR)}")
s2, b2 = run([-1, 5]) # add a non-negative case
print("suite2 branch", f"{len(b2&BR)}/{len(BR)}")
# suite1 stmt 3/3 branch 1/2 -> suite2 branch 2/2
Suite 1 runs every statement (3/3) yet only the True branch (1/2). Adding a non-negative input closes the gap — exactly what a branch-coverage gate forces.
Context: A tech lead sets a coverage policy that is risk-weighted, not a vanity 100% — and can defend it in review.
Your task: Write a coverage_policy(module) that returns the required branch-coverage floor for a module given its risk tier, and a checker that flags modules below their floor; then justify why MC-DC is required only for the safety-critical tier.
Requirements:
- Map risk tiers (safety-critical / core / peripheral) to branch-coverage floors (e.g. 100 / 85 / 60)
check(modules)returns the list of modules below their tier floor with the shortfall- Safety-critical tier additionally requires MC-DC (flag it in the policy)
- Demonstrate on a realistic module list; explain the cost/benefit of the tiers
- Note that coverage is necessary-not-sufficient and pair it with mutation testing (TQ7)
💡 Hint: Floors should track blast radius: a payment/auth module earns a higher floor than a log formatter. 100% everywhere wastes effort on low-risk code.
Show solution
A risk-weighted policy — higher floors where failure hurts most:
FLOORS = {"safety": 100, "core": 85, "peripheral": 60}
MCDC_REQUIRED = {"safety"}
def check(modules):
problems = []
for m in modules:
floor = FLOORS[m["tier"]]
if m["branch_cov"] < floor:
problems.append((m["name"], m["tier"], floor - m["branch_cov"]))
if m["tier"] in MCDC_REQUIRED and not m.get("mcdc"):
problems.append((m["name"], "MC-DC missing", 0))
return problems
mods = [
{"name": "auth", "tier": "safety", "branch_cov": 96, "mcdc": False},
{"name": "pricing","tier": "core", "branch_cov": 88, "mcdc": False},
{"name": "logfmt", "tier": "peripheral", "branch_cov": 55, "mcdc": False},
]
for p in check(mods): print(p)
# ('auth', 'safety', 4)
# ('auth', 'MC-DC missing', 0)
# ('logfmt', 'peripheral', 5)
The floor tracks blast radius: auth must hit 100% branch and MC-DC (the DO-178C discipline); a log formatter earns 60%. Demanding 100% everywhere wastes effort and breeds assertion-free tests — which is why you pair the floor with mutation testing (TQ7) to check the assertions actually bite.