Testing theory & coverage
The theory a practical suite skips: the test taxonomy, the pyramid vs the trophy, rigorous test-case design (equivalence classes, boundary values, decision tables, state machines), and what statement, branch, path and MC-DC coverage really prove — all runnable.
Learning objectives
- Place any test in the taxonomy — unit / integration / system / acceptance, and black / white / grey box.
- Choose the right shape: the testing pyramid vs the testing trophy, and why.
- Design test cases with equivalence partitioning and boundary-value analysis — and find an off-by-one.
- Build decision tables and state-transition tests for logic and stateful code.
- Compute statement, branch, path, condition and MC-DC coverage by hand, and explain why 100% statement coverage is not correctness.
unittest) — no pytest needed. The tools it references (coverage.py, pytest --cov) are noted where a real project would use them.1 · The testing taxonomy
Before writing a test you should be able to name what kind it is on two independent axes. The first axis is scope — how much of the system it exercises. The second is knowledge of internals — whether the test sees the code or only the interface.
| Level | Scope | Speed / count | Catches |
|---|---|---|---|
| Unit | one function / class in isolation | ms · thousands | logic errors, edge cases |
| Integration | two+ real components together | 10s–100s ms · hundreds | contract / wiring bugs |
| System (E2E) | the whole deployed app | seconds · tens | flow / config / environment bugs |
| Acceptance | system vs business criteria | seconds · few | “wrong thing built” |
The orthogonal axis is how much the test knows about the implementation:
| Approach | Sees | Design driven by | Typical use |
|---|---|---|---|
| Black box | only inputs/outputs | spec & requirements | acceptance, API tests |
| White box | the source code | code structure & coverage | unit tests, coverage design |
| Grey box | partial internals | spec + known internals | integration, security tests |
2 · The testing pyramid & the trophy
How many of each level should you have? Two competing shapes answer this. The testing pyramid (Cohn) says: many fast unit tests at the base, fewer integration tests, very few slow E2E tests at the tip — because cost and flakiness rise as you go up.
The testing trophy (Dodds) argues that for I/O-heavy, well-typed applications the biggest return sits at the integration layer, with static analysis (types, lint) as a broad base. It is not a contradiction — it is a different cost/return profile:
| Shape | Bulk of tests at | Best when | Risk if misapplied |
|---|---|---|---|
| Pyramid | unit | rich domain logic, pure functions | many green units, broken wiring |
| Trophy | integration | thin logic over I/O, strong types | slow suite, harder to localize failures |
3 · Equivalence partitioning & boundary-value analysis
You cannot test every input. Equivalence partitioning divides the input space into classes where the program should behave the same, so one representative per class suffices. Boundary-value analysis (BVA) then adds the fact that bugs cluster at the edges of those classes — so for a range [lo, hi] you test lo-1, lo, lo+1, hi-1, hi, hi+1. The lab finds a real off-by-one that a happy-path test misses entirely.
bva.py# The classic off-by-one: "valid age is 18..65 inclusive". A buggy check uses
# `< 65` instead of `<= 65`, silently rejecting the upper boundary (65).
def eligible_buggy(age):
return 18 <= age < 65 # BUG: excludes 65
def eligible_fixed(age):
return 18 <= age <= 65 # correct: 65 is valid
# Boundary-Value Analysis test set: for a range [lo, hi] test
# lo-1, lo, lo+1, hi-1, hi, hi+1 (the 6 points where bugs cluster).
def bva_points(lo, hi):
return [lo - 1, lo, lo + 1, hi - 1, hi, hi + 1]
points = bva_points(18, 65) # [17, 18, 19, 64, 65, 66]
expected = [False, True, True, True, True, False]
# A naive "happy-path" suite only tries a middle value and both pass:
print("happy-path only :", eligible_buggy(40) == True, eligible_fixed(40) == True)
# BVA catches the bug precisely at the upper boundary (65):
buggy = [eligible_buggy(a) for a in points]
fixed = [eligible_fixed(a) for a in points]
print("boundary points :", points)
print("buggy results :", buggy)
print("fixed results :", fixed)
print("first mismatch :", next(p for p, b, e in zip(points, buggy, expected) if b != e))
happy-path only : True True
boundary points : [17, 18, 19, 64, 65, 66]
buggy results : [False, True, True, True, False, False]
fixed results : [False, True, True, True, True, False]
first mismatch : 65
The happy-path line passes for both the buggy and fixed versions — a middle value like 40 tells you nothing. Only the boundary point 65 exposes that < 65 wrongly rejects the inclusive upper bound. This is the single highest-yield test-design technique for exams and real bugs alike.
pip install hypothesis) will search the boundaries for you by shrinking failing inputs to the minimal case — covered in TQ7.4 · Decision-table testing
When behaviour depends on a combination of conditions, a decision table enumerates every rule (2n for n boolean conditions) and its expected action, so you cannot forget a combination. It is a black-box technique: it comes from the spec, not the code.
decision_table.py# Decision-table testing: enumerate rules over conditions, derive expected action.
# Login policy: allow only if (valid_password AND account_active AND NOT locked).
from itertools import product
def decide(valid_pw, active, locked):
return "ALLOW" if (valid_pw and active and not locked) else "DENY"
conditions = ["valid_pw", "active", "locked"]
print(f"{'valid_pw':>9} {'active':>7} {'locked':>7} | action")
allow_rows = 0
for valid_pw, active, locked in product([True, False], repeat=3):
action = decide(valid_pw, active, locked)
allow_rows += action == "ALLOW"
print(f"{str(valid_pw):>9} {str(active):>7} {str(locked):>7} | {action}")
print("rules:", 2**len(conditions), "| ALLOW rules:", allow_rows)
valid_pw active locked | action
True True True | DENY
True True False | ALLOW
True False True | DENY
True False False | DENY
False True True | DENY
False True False | DENY
False False True | DENY
False False False | DENY
rules: 8 | ALLOW rules: 1
Three conditions give 23 = 8 rules; exactly one yields ALLOW. In practice you collapse rules whose outcome is independent of a condition (don't-care entries) to shrink the table — but you derive it in full first so the collapse is justified.
5 · State-transition testing
Stateful code is a finite-state machine: the same event does different things in different states. State-transition testing covers each legal transition and each illegal event (which must be rejected, not silently swallowed). The turnstile is the canonical example.
state_transition.pyimport unittest
# State-transition testing: a turnstile FSM. States: LOCKED, UNLOCKED.
# Events: coin, push. Illegal transitions must be rejected, not silently ignored.
class Turnstile:
def __init__(self):
self.state = "LOCKED"
def coin(self):
self.state = "UNLOCKED" # coin always unlocks
def push(self):
if self.state == "UNLOCKED":
self.state = "LOCKED" # a valid push passes through, re-locks
return "PASS"
return "BLOCKED" # pushing a locked gate is blocked
class TestTurnstile(unittest.TestCase):
def setUp(self): self.t = Turnstile()
def test_happy_path(self):
self.t.coin()
self.assertEqual(self.t.push(), "PASS")
self.assertEqual(self.t.state, "LOCKED")
def test_push_while_locked_is_blocked(self):
self.assertEqual(self.t.push(), "BLOCKED") # illegal event handled
self.assertEqual(self.t.state, "LOCKED")
def test_two_coins_still_one_pass(self):
self.t.coin(); self.t.coin() # coin is idempotent-ish
self.assertEqual(self.t.push(), "PASS")
if __name__ == "__main__":
unittest.main(verbosity=2)
test_happy_path (__main__.TestTurnstile.test_happy_path) ... ok
test_push_while_locked_is_blocked (__main__.TestTurnstile.test_push_while_locked_is_blocked) ... ok
test_two_coins_still_one_pass (__main__.TestTurnstile.test_two_coins_still_one_pass) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
test_push_while_locked_is_blocked is the one juniors forget: it asserts the illegal event (push while LOCKED) is handled gracefully. A transition table with one row per (state, event) pair is the systematic way to guarantee none is missed.
6 · Coverage theory: statement → branch → path → MC-DC
Coverage is a white-box measure of what your tests execute. The hierarchy, from weakest to strongest:
| Criterion | Requires | Subsumes | Note |
|---|---|---|---|
| Statement | every line executed once | — | weakest; misses untaken branches |
| Branch (decision) | every decision True and False | statement | the usual CI target |
| Condition | each atomic condition both ways | — | alone doesn't imply branch |
| Condition/decision (C/DC) | conditions + decisions both ways | branch | stronger combo |
| MC-DC | each condition independently flips the decision | C/DC | DO-178C avionics standard |
| Path | every path through the code | all above | exponential; usually infeasible |
The lab makes the key point concrete: a suite can hit every statement (100% statement coverage) yet leave a branch outcome untaken — so 100% statement coverage is not correctness.
coverage_theory.pyimport unittest
# The function under test. Two conditions in one `if` (a compound decision),
# plus a branch that returns early.
def classify(x, y):
if x > 0 and y > 0: # decision D1 with conditions C1:(x>0) C2:(y>0)
return "both-pos"
if x == y: # decision D2
return "equal"
return "other"
# --- Coverage models (computed by hand so you see exactly what each counts) ---
# Statements: the 4 executable lines inside classify.
# Branches: each decision's True AND False outcome -> 4 branch outcomes.
# Conditions: each atomic condition True AND False -> C1,C2 each way + D2.
STATEMENTS = {"if1", "ret_both", "if2", "ret_equal_or_other"}
BRANCHES = {"D1-T", "D1-F", "D2-T", "D2-F"}
def run(tests):
stmt, brch = set(), set()
for x, y in tests:
stmt.add("if1")
if x > 0 and y > 0:
stmt.add("ret_both"); brch.add("D1-T")
else:
brch.add("D1-F")
stmt.add("if2")
if x == y:
stmt.add("ret_equal_or_other"); brch.add("D2-T")
else:
stmt.add("ret_equal_or_other"); brch.add("D2-F")
return stmt, brch
def pct(got, whole):
return f"{len(got & whole)}/{len(whole)} = {100*len(got & whole)//len(whole)}%"
# Suite A: two tests. Reaches every *statement* -> 100% statement coverage...
suite_a = [(1, 1), (-2, -2)]
sa_stmt, sa_brch = run(suite_a)
print("suite A statement:", pct(sa_stmt, STATEMENTS))
print("suite A branch :", pct(sa_brch, BRANCHES), "-> misses", sorted(BRANCHES - sa_brch))
# ...yet it never takes the False branch of D2, so a bug there hides.
# Suite B adds a non-equal negative case to cover every branch outcome.
suite_b = [(1, 1), (-2, -2), (-2, -3)]
sb_stmt, sb_brch = run(suite_b)
print("suite B branch :", pct(sb_brch, BRANCHES))
print("100% statement != correct:", len(sa_stmt) == len(STATEMENTS), "yet branch only", pct(sa_brch, BRANCHES))
suite A statement: 4/4 = 100%
suite A branch : 3/4 = 75% -> misses ['D2-F']
suite B branch : 4/4 = 100%
100% statement != correct: True yet branch only 3/4 = 75%
Suite A executes all 4 statements — a green 100% on a statement-coverage report — but never takes the False branch of D2 (the x != y case), so it sits at 75% branch coverage. A bug in that untaken path is invisible until Suite B adds the non-equal negative case. In a real project you would get these numbers from coverage.py / pytest --cov --cov-branch; MC-DC needs a dedicated tool.
✓ Checkpoint — you can move on when you can…
- Classify a test by scope and by black/white/grey box.
- Choose pyramid vs trophy for a given system and justify it.
- Derive a BVA test set for a range and explain which point catches an off-by-one.
- Build a decision table and a state-transition test set.
- Compute statement and branch coverage by hand and state why one test can give 100% statement but 25% branch.
Knowledge check
The coverage lab shows Suite A reaching 4/4 statements (100% statement coverage) but only 3/4 branches. Explain in coverage terms why “100% statement coverage” still does not prove classify is correct.
Show answer
D2 (the x != y path), so an error there is never exercised — hence 75% branch coverage despite 100% statement coverage. Branch coverage subsumes statement coverage precisely because it demands both outcomes of every decision. And even 100% branch coverage only shows lines ran, not that assertions checked the right result — which is why mutation testing (TQ7) exists.You have a well-typed web service that is mostly thin handlers over a database, with little domain logic. Which test shape does the theory favour — pyramid or trophy — and what is the main risk of the other?
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Partitioning the input space is the first move in test design — one representative per class beats random guessing.
Your task: Given letter_grade(score) mapping 0–100 to A/B/C/D/F, list the equivalence classes and write one unittest assertion per class plus the invalid-input class.
Requirements:
- Identify each score band as a class (A:90–100, B:80–89, …, F:0–59) plus an invalid class (<0 or >100)
- One representative value per class, asserted with
assertEqual - The invalid class asserts a raised
ValueErrorviaassertRaises - Runs under
unittest.mainwith no external deps
💡 Hint: Pick a mid-band representative for each class; you are testing the class, not the boundary yet (that is the next rung).
Show solution
One representative per equivalence class, plus the invalid class:
import unittest
def letter_grade(score):
if not 0 <= score <= 100:
raise ValueError("score must be 0-100")
if score >= 90: return "A"
if score >= 80: return "B"
if score >= 70: return "C"
if score >= 60: return "D"
return "F"
class TestGrade(unittest.TestCase):
def test_classes(self):
for score, grade in [(95,"A"),(85,"B"),(75,"C"),(65,"D"),(30,"F")]:
self.assertEqual(letter_grade(score), grade)
def test_invalid_class(self):
with self.assertRaises(ValueError): letter_grade(150)
if __name__ == "__main__":
unittest.main(verbosity=2) # 2 tests OK
Five valid classes plus one invalid class = six representatives. Boundaries (89 vs 90) are a separate technique — next rung.
Context: Bugs cluster at class edges; BVA tests lo-1, lo, lo+1, hi-1, hi, hi+1.
Your task: Write bva_points(lo, hi) and use it to test a range check in_range(x) for 1–100 inclusive, then show the set catches a < 100 off-by-one.
Requirements:
bva_points(lo, hi)returns the six boundary points- Test both a buggy (
< 100) and a correct (<= 100) implementation over those points - Assert the two disagree exactly at the upper boundary
100 - Runs under
unittest
💡 Hint: The mismatch will be at hi itself; a middle value never reveals it.
Show solution
The six points, and the disagreement pinned at the upper boundary:
import unittest
def bva_points(lo, hi): return [lo-1, lo, lo+1, hi-1, hi, hi+1]
def in_range_buggy(x): return 1 <= x < 100 # BUG
def in_range_ok(x): return 1 <= x <= 100
class TestBVA(unittest.TestCase):
def test_boundary_mismatch(self):
pts = bva_points(1, 100) # [0,1,2,99,100,101]
diffs = [p for p in pts if in_range_buggy(p) != in_range_ok(p)]
self.assertEqual(diffs, [100]) # only 100 differs
if __name__ == "__main__":
unittest.main(verbosity=2) # 1 test OK
The buggy < 100 rejects 100; the correct version accepts it. BVA localizes the off-by-one to a single point.
Context: Combinatorial logic needs every rule enumerated so none is forgotten.
Your task: A shipping-cost rule: free shipping if member AND (cart ≥ 50 OR coupon). Build the full decision table over the three booleans and assert the number of FREE rules.
Requirements:
- Enumerate all 23 = 8 rules with
itertools.product - Compute the action per rule from the policy
- Assert the count of FREE-shipping rules matches the truth table
- Runs under
unittest
💡 Hint: member must be True in every FREE row; then either cart≥50 or coupon.
Show solution
Enumerate, then assert the FREE count against the derived truth table:
import unittest
from itertools import product
def free_shipping(member, big_cart, coupon):
return member and (big_cart or coupon)
class TestDecisionTable(unittest.TestCase):
def test_free_rules(self):
rules = list(product([True, False], repeat=3))
free = [r for r in rules if free_shipping(*r)]
# member=T and (big_cart or coupon): (T,T,T),(T,T,F),(T,F,T) -> 3
self.assertEqual(len(free), 3)
self.assertEqual(len(rules), 8)
if __name__ == "__main__":
unittest.main(verbosity=2) # 1 test OK
Deriving the table in full guarantees no combination is missed; you may then collapse don't-care rows for the final suite.
Context: Every legal transition and every illegal event must be tested for a stateful object.
Your task: Model a player FSM (STOPPED/PLAYING/PAUSED) with events play/pause/stop, and write tests covering each legal transition plus one illegal event (pause while STOPPED).
Requirements:
- States STOPPED, PLAYING, PAUSED with play/pause/stop events
- One test per legal transition asserting the resulting state
- One test asserting an illegal event (pause while STOPPED) leaves state unchanged and is rejected
- Use
setUpfor a fresh player; runs underunittest
💡 Hint: Make illegal events raise or return a sentinel — either way, assert the state did not change.
Show solution
Cover legal transitions and the illegal event explicitly:
import unittest
class Player:
def __init__(self): self.state = "STOPPED"
def play(self): self.state = "PLAYING"
def pause(self):
if self.state != "PLAYING":
raise RuntimeError("can only pause while playing")
self.state = "PAUSED"
def stop(self): self.state = "STOPPED"
class TestPlayer(unittest.TestCase):
def setUp(self): self.p = Player()
def test_play_then_pause(self):
self.p.play(); self.p.pause()
self.assertEqual(self.p.state, "PAUSED")
def test_stop_from_playing(self):
self.p.play(); self.p.stop()
self.assertEqual(self.p.state, "STOPPED")
def test_illegal_pause_when_stopped(self):
with self.assertRaises(RuntimeError): self.p.pause()
self.assertEqual(self.p.state, "STOPPED") # unchanged
if __name__ == "__main__":
unittest.main(verbosity=2) # 3 tests OK
A transition table (row per state×event) is the systematic checklist ensuring no legal or illegal edge is missed.
Context: Teams gate on branch coverage precisely because statement coverage over-reports safety.
Your task: Instrument a two-branch function and show a suite that reaches 100% statement but under 100% branch coverage, then add the test that closes the branch gap.
Requirements:
- Track executed statements and branch outcomes in sets
- Suite 1 achieves 100% statement but <100% branch
- Suite 2 adds the case that reaches the missing branch outcome to hit 100% branch
- Print both percentages; runs under
unittestor as a script
💡 Hint: A guard clause that returns early is the classic place statement coverage lies about branch coverage.
Show solution
Statement coverage can read 100% while a branch outcome is still missing:
def run(tests):
stmt, brch = set(), set()
for n in tests:
stmt.add("enter")
if n < 0:
stmt.add("neg"); brch.add("T")
else:
brch.add("F")
stmt.add("exit")
return stmt, brch
STMT = {"enter", "neg", "exit"}; BR = {"T", "F"}
s1, b1 = run([-1]) # only the negative case
print("suite1 stmt", f"{len(s1&STMT)}/{len(STMT)}", "branch", f"{len(b1&BR)}/{len(BR)}")
s2, b2 = run([-1, 5]) # add a non-negative case
print("suite2 branch", f"{len(b2&BR)}/{len(BR)}")
# suite1 stmt 3/3 branch 1/2 -> suite2 branch 2/2
Suite 1 runs every statement (3/3) yet only the True branch (1/2). Adding a non-negative input closes the gap — exactly what a branch-coverage gate forces.
Context: A tech lead sets a coverage policy that is risk-weighted, not a vanity 100% — and can defend it in review.
Your task: Write a coverage_policy(module) that returns the required branch-coverage floor for a module given its risk tier, and a checker that flags modules below their floor; then justify why MC-DC is required only for the safety-critical tier.
Requirements:
- Map risk tiers (safety-critical / core / peripheral) to branch-coverage floors (e.g. 100 / 85 / 60)
check(modules)returns the list of modules below their tier floor with the shortfall- Safety-critical tier additionally requires MC-DC (flag it in the policy)
- Demonstrate on a realistic module list; explain the cost/benefit of the tiers
- Note that coverage is necessary-not-sufficient and pair it with mutation testing (TQ7)
💡 Hint: Floors should track blast radius: a payment/auth module earns a higher floor than a log formatter. 100% everywhere wastes effort on low-risk code.
Show solution
A risk-weighted policy — higher floors where failure hurts most:
FLOORS = {"safety": 100, "core": 85, "peripheral": 60}
MCDC_REQUIRED = {"safety"}
def check(modules):
problems = []
for m in modules:
floor = FLOORS[m["tier"]]
if m["branch_cov"] < floor:
problems.append((m["name"], m["tier"], floor - m["branch_cov"]))
if m["tier"] in MCDC_REQUIRED and not m.get("mcdc"):
problems.append((m["name"], "MC-DC missing", 0))
return problems
mods = [
{"name": "auth", "tier": "safety", "branch_cov": 96, "mcdc": False},
{"name": "pricing","tier": "core", "branch_cov": 88, "mcdc": False},
{"name": "logfmt", "tier": "peripheral", "branch_cov": 55, "mcdc": False},
]
for p in check(mods): print(p)
# ('auth', 'safety', 4)
# ('auth', 'MC-DC missing', 0)
# ('logfmt', 'peripheral', 5)
The floor tracks blast radius: auth must hit 100% branch and MC-DC (the DO-178C discipline); a log formatter earns 60%. Demanding 100% everywhere wastes effort and breeds assertion-free tests — which is why you pair the floor with mutation testing (TQ7) to check the assertions actually bite.