AI EngineeringZero to ProductionHome·About·Contact
Software Testing · Chapter TQ6

Testing theory & coverage

The theory a practical suite skips: the test taxonomy, the pyramid vs the trophy, rigorous test-case design (equivalence classes, boundary values, decision tables, state machines), and what statement, branch, path and MC-DC coverage really prove — all runnable.

⏱️ ~3 hours🧪 6 labs🎯 Beginner→Industry

Learning objectives

  • Place any test in the taxonomy — unit / integration / system / acceptance, and black / white / grey box.
  • Choose the right shape: the testing pyramid vs the testing trophy, and why.
  • Design test cases with equivalence partitioning and boundary-value analysis — and find an off-by-one.
  • Build decision tables and state-transition tests for logic and stateful code.
  • Compute statement, branch, path, condition and MC-DC coverage by hand, and explain why 100% statement coverage is not correctness.
▶ Runnable companionEvery Python block here runs on the standard library (unittest) — no pytest needed. The tools it references (coverage.py, pytest --cov) are noted where a real project would use them.

1 · The testing taxonomy

Before writing a test you should be able to name what kind it is on two independent axes. The first axis is scope — how much of the system it exercises. The second is knowledge of internals — whether the test sees the code or only the interface.

LevelScopeSpeed / countCatches
Unitone function / class in isolationms · thousandslogic errors, edge cases
Integrationtwo+ real components together10s–100s ms · hundredscontract / wiring bugs
System (E2E)the whole deployed appseconds · tensflow / config / environment bugs
Acceptancesystem vs business criteriaseconds · few“wrong thing built”

The orthogonal axis is how much the test knows about the implementation:

ApproachSeesDesign driven byTypical use
Black boxonly inputs/outputsspec & requirementsacceptance, API tests
White boxthe source codecode structure & coverageunit tests, coverage design
Grey boxpartial internalsspec + known internalsintegration, security tests
Verification vs validationVerification asks “did we build the thing right?” (unit/integration correctness). Validation asks “did we build the right thing?” (acceptance). A suite can be 100% green (verified) and still ship the wrong product (invalidated).

2 · The testing pyramid & the trophy

How many of each level should you have? Two competing shapes answer this. The testing pyramid (Cohn) says: many fast unit tests at the base, fewer integration tests, very few slow E2E tests at the tip — because cost and flakiness rise as you go up.

E2E (few) slow, brittle Integration medium Unit (many) fast, stable

The testing trophy (Dodds) argues that for I/O-heavy, well-typed applications the biggest return sits at the integration layer, with static analysis (types, lint) as a broad base. It is not a contradiction — it is a different cost/return profile:

ShapeBulk of tests atBest whenRisk if misapplied
Pyramidunitrich domain logic, pure functionsmany green units, broken wiring
Trophyintegrationthin logic over I/O, strong typesslow suite, harder to localize failures
The ice-cream cone anti-patternInverting the pyramid — mostly manual/E2E tests, almost no unit tests — gives a slow, flaky suite that is expensive to run and useless at localizing a failure. Recognizing it is an exam favourite.

3 · Equivalence partitioning & boundary-value analysis

You cannot test every input. Equivalence partitioning divides the input space into classes where the program should behave the same, so one representative per class suffices. Boundary-value analysis (BVA) then adds the fact that bugs cluster at the edges of those classes — so for a range [lo, hi] you test lo-1, lo, lo+1, hi-1, hi, hi+1. The lab finds a real off-by-one that a happy-path test misses entirely.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.
Python · BVA finds the off-by-one (runs)
bva.py# The classic off-by-one: "valid age is 18..65 inclusive". A buggy check uses
# `< 65` instead of `<= 65`, silently rejecting the upper boundary (65).
def eligible_buggy(age):
    return 18 <= age < 65            # BUG: excludes 65

def eligible_fixed(age):
    return 18 <= age <= 65           # correct: 65 is valid

# Boundary-Value Analysis test set: for a range [lo, hi] test
# lo-1, lo, lo+1, hi-1, hi, hi+1  (the 6 points where bugs cluster).
def bva_points(lo, hi):
    return [lo - 1, lo, lo + 1, hi - 1, hi, hi + 1]

points = bva_points(18, 65)          # [17, 18, 19, 64, 65, 66]
expected = [False, True, True, True, True, False]

# A naive "happy-path" suite only tries a middle value and both pass:
print("happy-path only  :", eligible_buggy(40) == True, eligible_fixed(40) == True)

# BVA catches the bug precisely at the upper boundary (65):
buggy = [eligible_buggy(a) for a in points]
fixed = [eligible_fixed(a) for a in points]
print("boundary points  :", points)
print("buggy  results   :", buggy)
print("fixed  results   :", fixed)
print("first mismatch   :", next(p for p, b, e in zip(points, buggy, expected) if b != e))
happy-path only  : True True
boundary points  : [17, 18, 19, 64, 65, 66]
buggy  results   : [False, True, True, True, False, False]
fixed  results   : [False, True, True, True, True, False]
first mismatch   : 65

The happy-path line passes for both the buggy and fixed versions — a middle value like 40 tells you nothing. Only the boundary point 65 exposes that < 65 wrongly rejects the inclusive upper bound. This is the single highest-yield test-design technique for exams and real bugs alike.

Property tools generalize thisA property-based tool such as Hypothesis (needs pip install hypothesis) will search the boundaries for you by shrinking failing inputs to the minimal case — covered in TQ7.

4 · Decision-table testing

When behaviour depends on a combination of conditions, a decision table enumerates every rule (2n for n boolean conditions) and its expected action, so you cannot forget a combination. It is a black-box technique: it comes from the spec, not the code.

Python · enumerate every rule (runs)
decision_table.py# Decision-table testing: enumerate rules over conditions, derive expected action.
# Login policy: allow only if (valid_password AND account_active AND NOT locked).
from itertools import product

def decide(valid_pw, active, locked):
    return "ALLOW" if (valid_pw and active and not locked) else "DENY"

conditions = ["valid_pw", "active", "locked"]
print(f"{'valid_pw':>9} {'active':>7} {'locked':>7} | action")
allow_rows = 0
for valid_pw, active, locked in product([True, False], repeat=3):
    action = decide(valid_pw, active, locked)
    allow_rows += action == "ALLOW"
    print(f"{str(valid_pw):>9} {str(active):>7} {str(locked):>7} | {action}")
print("rules:", 2**len(conditions), "| ALLOW rules:", allow_rows)
 valid_pw  active  locked | action
     True    True    True | DENY
     True    True   False | ALLOW
     True   False    True | DENY
     True   False   False | DENY
    False    True    True | DENY
    False    True   False | DENY
    False   False    True | DENY
    False   False   False | DENY
rules: 8 | ALLOW rules: 1

Three conditions give 23 = 8 rules; exactly one yields ALLOW. In practice you collapse rules whose outcome is independent of a condition (don't-care entries) to shrink the table — but you derive it in full first so the collapse is justified.

5 · State-transition testing

Stateful code is a finite-state machine: the same event does different things in different states. State-transition testing covers each legal transition and each illegal event (which must be rejected, not silently swallowed). The turnstile is the canonical example.

LOCKED UNLOCKED
Python · FSM transitions under unittest (runs)
state_transition.pyimport unittest

# State-transition testing: a turnstile FSM. States: LOCKED, UNLOCKED.
# Events: coin, push. Illegal transitions must be rejected, not silently ignored.
class Turnstile:
    def __init__(self):
        self.state = "LOCKED"
    def coin(self):
        self.state = "UNLOCKED"        # coin always unlocks
    def push(self):
        if self.state == "UNLOCKED":
            self.state = "LOCKED"      # a valid push passes through, re-locks
            return "PASS"
        return "BLOCKED"               # pushing a locked gate is blocked

class TestTurnstile(unittest.TestCase):
    def setUp(self): self.t = Turnstile()
    def test_happy_path(self):
        self.t.coin()
        self.assertEqual(self.t.push(), "PASS")
        self.assertEqual(self.t.state, "LOCKED")
    def test_push_while_locked_is_blocked(self):
        self.assertEqual(self.t.push(), "BLOCKED")   # illegal event handled
        self.assertEqual(self.t.state, "LOCKED")
    def test_two_coins_still_one_pass(self):
        self.t.coin(); self.t.coin()                 # coin is idempotent-ish
        self.assertEqual(self.t.push(), "PASS")

if __name__ == "__main__":
    unittest.main(verbosity=2)
test_happy_path (__main__.TestTurnstile.test_happy_path) ... ok
test_push_while_locked_is_blocked (__main__.TestTurnstile.test_push_while_locked_is_blocked) ... ok
test_two_coins_still_one_pass (__main__.TestTurnstile.test_two_coins_still_one_pass) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

test_push_while_locked_is_blocked is the one juniors forget: it asserts the illegal event (push while LOCKED) is handled gracefully. A transition table with one row per (state, event) pair is the systematic way to guarantee none is missed.

6 · Coverage theory: statement → branch → path → MC-DC

Coverage is a white-box measure of what your tests execute. The hierarchy, from weakest to strongest:

CriterionRequiresSubsumesNote
Statementevery line executed onceweakest; misses untaken branches
Branch (decision)every decision True and Falsestatementthe usual CI target
Conditioneach atomic condition both waysalone doesn't imply branch
Condition/decision (C/DC)conditions + decisions both waysbranchstronger combo
MC-DCeach condition independently flips the decisionC/DCDO-178C avionics standard
Pathevery path through the codeall aboveexponential; usually infeasible

The lab makes the key point concrete: a suite can hit every statement (100% statement coverage) yet leave a branch outcome untaken — so 100% statement coverage is not correctness.

Python · statement vs branch coverage, computed (runs)
coverage_theory.pyimport unittest

# The function under test. Two conditions in one `if` (a compound decision),
# plus a branch that returns early.
def classify(x, y):
    if x > 0 and y > 0:      # decision D1 with conditions C1:(x>0) C2:(y>0)
        return "both-pos"
    if x == y:               # decision D2
        return "equal"
    return "other"

# --- Coverage models (computed by hand so you see exactly what each counts) ---
# Statements: the 4 executable lines inside classify.
# Branches:   each decision's True AND False outcome  -> 4 branch outcomes.
# Conditions: each atomic condition True AND False     -> C1,C2 each way + D2.
STATEMENTS = {"if1", "ret_both", "if2", "ret_equal_or_other"}
BRANCHES   = {"D1-T", "D1-F", "D2-T", "D2-F"}

def run(tests):
    stmt, brch = set(), set()
    for x, y in tests:
        stmt.add("if1")
        if x > 0 and y > 0:
            stmt.add("ret_both"); brch.add("D1-T")
        else:
            brch.add("D1-F")
            stmt.add("if2")
            if x == y:
                stmt.add("ret_equal_or_other"); brch.add("D2-T")
            else:
                stmt.add("ret_equal_or_other"); brch.add("D2-F")
    return stmt, brch

def pct(got, whole):
    return f"{len(got & whole)}/{len(whole)} = {100*len(got & whole)//len(whole)}%"

# Suite A: two tests. Reaches every *statement* -> 100% statement coverage...
suite_a = [(1, 1), (-2, -2)]
sa_stmt, sa_brch = run(suite_a)
print("suite A statement:", pct(sa_stmt, STATEMENTS))
print("suite A branch   :", pct(sa_brch, BRANCHES), "-> misses", sorted(BRANCHES - sa_brch))

# ...yet it never takes the False branch of D2, so a bug there hides.
# Suite B adds a non-equal negative case to cover every branch outcome.
suite_b = [(1, 1), (-2, -2), (-2, -3)]
sb_stmt, sb_brch = run(suite_b)
print("suite B branch   :", pct(sb_brch, BRANCHES))
print("100% statement != correct:", len(sa_stmt) == len(STATEMENTS), "yet branch only", pct(sa_brch, BRANCHES))
suite A statement: 4/4 = 100%
suite A branch   : 3/4 = 75% -> misses ['D2-F']
suite B branch   : 4/4 = 100%
100% statement != correct: True yet branch only 3/4 = 75%

Suite A executes all 4 statements — a green 100% on a statement-coverage report — but never takes the False branch of D2 (the x != y case), so it sits at 75% branch coverage. A bug in that untaken path is invisible until Suite B adds the non-equal negative case. In a real project you would get these numbers from coverage.py / pytest --cov --cov-branch; MC-DC needs a dedicated tool.

The limits of coverageCoverage is necessary but not sufficient. It proves a line ran, never that you asserted the right thing (a test with no assertion still counts as covered), and it cannot see missing code (a branch you forgot to write). Use it to find gaps, not to certify quality. TQ7's mutation testing measures assertion strength instead.

✓ Checkpoint — you can move on when you can…

  • Classify a test by scope and by black/white/grey box.
  • Choose pyramid vs trophy for a given system and justify it.
  • Derive a BVA test set for a range and explain which point catches an off-by-one.
  • Build a decision table and a state-transition test set.
  • Compute statement and branch coverage by hand and state why one test can give 100% statement but 25% branch.

Knowledge check

✓ Knowledge check

The coverage lab shows Suite A reaching 4/4 statements (100% statement coverage) but only 3/4 branches. Explain in coverage terms why “100% statement coverage” still does not prove classify is correct.

Show answer
Statement coverage only requires each line to run once; Suite A does that, but it never takes the False branch of D2 (the x != y path), so an error there is never exercised — hence 75% branch coverage despite 100% statement coverage. Branch coverage subsumes statement coverage precisely because it demands both outcomes of every decision. And even 100% branch coverage only shows lines ran, not that assertions checked the right result — which is why mutation testing (TQ7) exists.
✓ Knowledge check

You have a well-typed web service that is mostly thin handlers over a database, with little domain logic. Which test shape does the theory favour — pyramid or trophy — and what is the main risk of the other?

Show answer
The trophy: most value sits at the integration layer (handler + real DB) with static typing/lint as the base, because the logic is thin and the bugs live in wiring and I/O. Forcing a classic pyramid here yields many trivial unit tests that mock the database away and still let contract/wiring bugs through — lots of green units, broken system. (Conversely, a rich-domain-logic system favours the pyramid.)

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Equivalence classes for a graderBeginner

Context: Partitioning the input space is the first move in test design — one representative per class beats random guessing.

Your task: Given letter_grade(score) mapping 0–100 to A/B/C/D/F, list the equivalence classes and write one unittest assertion per class plus the invalid-input class.

Requirements:

  • Identify each score band as a class (A:90–100, B:80–89, …, F:0–59) plus an invalid class (<0 or >100)
  • One representative value per class, asserted with assertEqual
  • The invalid class asserts a raised ValueError via assertRaises
  • Runs under unittest.main with no external deps

💡 Hint: Pick a mid-band representative for each class; you are testing the class, not the boundary yet (that is the next rung).

Show solution

One representative per equivalence class, plus the invalid class:

import unittest

def letter_grade(score):
    if not 0 <= score <= 100:
        raise ValueError("score must be 0-100")
    if score >= 90: return "A"
    if score >= 80: return "B"
    if score >= 70: return "C"
    if score >= 60: return "D"
    return "F"

class TestGrade(unittest.TestCase):
    def test_classes(self):
        for score, grade in [(95,"A"),(85,"B"),(75,"C"),(65,"D"),(30,"F")]:
            self.assertEqual(letter_grade(score), grade)
    def test_invalid_class(self):
        with self.assertRaises(ValueError): letter_grade(150)

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 2 tests OK

Five valid classes plus one invalid class = six representatives. Boundaries (89 vs 90) are a separate technique — next rung.

Exercise 2 · Boundary-value test set that finds the bugIntermediate

Context: Bugs cluster at class edges; BVA tests lo-1, lo, lo+1, hi-1, hi, hi+1.

Your task: Write bva_points(lo, hi) and use it to test a range check in_range(x) for 1–100 inclusive, then show the set catches a < 100 off-by-one.

Requirements:

  • bva_points(lo, hi) returns the six boundary points
  • Test both a buggy (< 100) and a correct (<= 100) implementation over those points
  • Assert the two disagree exactly at the upper boundary 100
  • Runs under unittest

💡 Hint: The mismatch will be at hi itself; a middle value never reveals it.

Show solution

The six points, and the disagreement pinned at the upper boundary:

import unittest

def bva_points(lo, hi): return [lo-1, lo, lo+1, hi-1, hi, hi+1]
def in_range_buggy(x): return 1 <= x < 100     # BUG
def in_range_ok(x):    return 1 <= x <= 100

class TestBVA(unittest.TestCase):
    def test_boundary_mismatch(self):
        pts = bva_points(1, 100)                  # [0,1,2,99,100,101]
        diffs = [p for p in pts if in_range_buggy(p) != in_range_ok(p)]
        self.assertEqual(diffs, [100])           # only 100 differs

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 1 test OK

The buggy < 100 rejects 100; the correct version accepts it. BVA localizes the off-by-one to a single point.

Exercise 3 · Decision table for a discount policyAdvanced

Context: Combinatorial logic needs every rule enumerated so none is forgotten.

Your task: A shipping-cost rule: free shipping if member AND (cart ≥ 50 OR coupon). Build the full decision table over the three booleans and assert the number of FREE rules.

Requirements:

  • Enumerate all 23 = 8 rules with itertools.product
  • Compute the action per rule from the policy
  • Assert the count of FREE-shipping rules matches the truth table
  • Runs under unittest

💡 Hint: member must be True in every FREE row; then either cart≥50 or coupon.

Show solution

Enumerate, then assert the FREE count against the derived truth table:

import unittest
from itertools import product

def free_shipping(member, big_cart, coupon):
    return member and (big_cart or coupon)

class TestDecisionTable(unittest.TestCase):
    def test_free_rules(self):
        rules = list(product([True, False], repeat=3))
        free = [r for r in rules if free_shipping(*r)]
        # member=T and (big_cart or coupon): (T,T,T),(T,T,F),(T,F,T) -> 3
        self.assertEqual(len(free), 3)
        self.assertEqual(len(rules), 8)

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 1 test OK

Deriving the table in full guarantees no combination is missed; you may then collapse don't-care rows for the final suite.

Exercise 4 · State-transition coverage of a media playerExpert

Context: Every legal transition and every illegal event must be tested for a stateful object.

Your task: Model a player FSM (STOPPED/PLAYING/PAUSED) with events play/pause/stop, and write tests covering each legal transition plus one illegal event (pause while STOPPED).

Requirements:

  • States STOPPED, PLAYING, PAUSED with play/pause/stop events
  • One test per legal transition asserting the resulting state
  • One test asserting an illegal event (pause while STOPPED) leaves state unchanged and is rejected
  • Use setUp for a fresh player; runs under unittest

💡 Hint: Make illegal events raise or return a sentinel — either way, assert the state did not change.

Show solution

Cover legal transitions and the illegal event explicitly:

import unittest

class Player:
    def __init__(self): self.state = "STOPPED"
    def play(self):  self.state = "PLAYING"
    def pause(self):
        if self.state != "PLAYING":
            raise RuntimeError("can only pause while playing")
        self.state = "PAUSED"
    def stop(self):  self.state = "STOPPED"

class TestPlayer(unittest.TestCase):
    def setUp(self): self.p = Player()
    def test_play_then_pause(self):
        self.p.play(); self.p.pause()
        self.assertEqual(self.p.state, "PAUSED")
    def test_stop_from_playing(self):
        self.p.play(); self.p.stop()
        self.assertEqual(self.p.state, "STOPPED")
    def test_illegal_pause_when_stopped(self):
        with self.assertRaises(RuntimeError): self.p.pause()
        self.assertEqual(self.p.state, "STOPPED")   # unchanged

if __name__ == "__main__":
    unittest.main(verbosity=2)   # 3 tests OK

A transition table (row per state×event) is the systematic checklist ensuring no legal or illegal edge is missed.

Exercise 5 · Branch coverage vs statement coverage, measuredProfessional

Context: Teams gate on branch coverage precisely because statement coverage over-reports safety.

Your task: Instrument a two-branch function and show a suite that reaches 100% statement but under 100% branch coverage, then add the test that closes the branch gap.

Requirements:

  • Track executed statements and branch outcomes in sets
  • Suite 1 achieves 100% statement but <100% branch
  • Suite 2 adds the case that reaches the missing branch outcome to hit 100% branch
  • Print both percentages; runs under unittest or as a script

💡 Hint: A guard clause that returns early is the classic place statement coverage lies about branch coverage.

Show solution

Statement coverage can read 100% while a branch outcome is still missing:

def run(tests):
    stmt, brch = set(), set()
    for n in tests:
        stmt.add("enter")
        if n < 0:
            stmt.add("neg"); brch.add("T")
        else:
            brch.add("F")
        stmt.add("exit")
    return stmt, brch

STMT = {"enter", "neg", "exit"}; BR = {"T", "F"}
s1, b1 = run([-1])            # only the negative case
print("suite1 stmt", f"{len(s1&STMT)}/{len(STMT)}", "branch", f"{len(b1&BR)}/{len(BR)}")
s2, b2 = run([-1, 5])         # add a non-negative case
print("suite2 branch", f"{len(b2&BR)}/{len(BR)}")
# suite1 stmt 3/3 branch 1/2  ->  suite2 branch 2/2

Suite 1 runs every statement (3/3) yet only the True branch (1/2). Adding a non-negative input closes the gap — exactly what a branch-coverage gate forces.

Exercise 6 · Design a coverage policy and defend itIndustry scenario

Context: A tech lead sets a coverage policy that is risk-weighted, not a vanity 100% — and can defend it in review.

Your task: Write a coverage_policy(module) that returns the required branch-coverage floor for a module given its risk tier, and a checker that flags modules below their floor; then justify why MC-DC is required only for the safety-critical tier.

Requirements:

  • Map risk tiers (safety-critical / core / peripheral) to branch-coverage floors (e.g. 100 / 85 / 60)
  • check(modules) returns the list of modules below their tier floor with the shortfall
  • Safety-critical tier additionally requires MC-DC (flag it in the policy)
  • Demonstrate on a realistic module list; explain the cost/benefit of the tiers
  • Note that coverage is necessary-not-sufficient and pair it with mutation testing (TQ7)

💡 Hint: Floors should track blast radius: a payment/auth module earns a higher floor than a log formatter. 100% everywhere wastes effort on low-risk code.

Show solution

A risk-weighted policy — higher floors where failure hurts most:

FLOORS = {"safety": 100, "core": 85, "peripheral": 60}
MCDC_REQUIRED = {"safety"}

def check(modules):
    problems = []
    for m in modules:
        floor = FLOORS[m["tier"]]
        if m["branch_cov"] < floor:
            problems.append((m["name"], m["tier"], floor - m["branch_cov"]))
        if m["tier"] in MCDC_REQUIRED and not m.get("mcdc"):
            problems.append((m["name"], "MC-DC missing", 0))
    return problems

mods = [
    {"name": "auth",   "tier": "safety",     "branch_cov": 96, "mcdc": False},
    {"name": "pricing","tier": "core",       "branch_cov": 88, "mcdc": False},
    {"name": "logfmt", "tier": "peripheral", "branch_cov": 55, "mcdc": False},
]
for p in check(mods): print(p)
# ('auth', 'safety', 4)
# ('auth', 'MC-DC missing', 0)
# ('logfmt', 'peripheral', 5)

The floor tracks blast radius: auth must hit 100% branch and MC-DC (the DO-178C discipline); a log formatter earns 60%. Demanding 100% everywhere wastes effort and breeds assertion-free tests — which is why you pair the floor with mutation testing (TQ7) to check the assertions actually bite.

© 2026 studybydoing.in · AI Engineering: Zero to Production · All rights reserved. · About · Privacy Policy · Terms · Contact
Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.
© studybydoing.in