Complexity theory & NP
Every previous DSA lesson found a fast algorithm. This capstone is about the problems where we believe no fast algorithm exists — and, crucially, how to tell when you are staring at one so you stop burning weeks hunting for a polynomial solution that almost certainly is not there. We cover the complexity classes (P, NP, NP-complete, NP-hard), polynomial-time reductions with a reduction you can run, the limits of computation itself (the halting problem), the classic NP-complete catalogue, and the engineer’s real playbook: approximation and heuristics. The theory is stated honestly — P vs NP is open, and nothing here pretends otherwise.
Learning objectives
- Define P, NP, NP-complete, and NP-hard precisely and place problems in the right box.
- Explain NP as 'verifiable in polynomial time' and demo a polynomial verifier.
- Perform a polynomial-time reduction (3-SAT → clique) and verify it in code.
- State the halting problem and reproduce the diagonal argument for undecidability.
- Recognise the classic NP-complete problems when they show up disguised in real work.
- Respond to an NP-hard problem the way an engineer should: approximation and heuristics.
1 · The complexity classes
P is the set of decision problems solvable in polynomial time — the "tractable" problems, everything in this track so far. NP is the set of problems whose yes-answers can be verified in polynomial time given a short certificate (a proposed solution). Every problem in P is in NP. NP-complete problems are the hardest in NP: every NP problem reduces to them, so a polynomial algorithm for one would solve all of NP. NP-hard is "at least as hard as NP-complete" but not required to be in NP (it need not even be a decision problem — the optimisation TSP is NP-hard).
| Class | Informal meaning | Example |
|---|---|---|
| P | solvable fast | shortest path, sorting |
| NP | answer checkable fast | subset-sum, SAT, clique |
| NP-complete | hardest in NP; all NP reduces to it | 3-SAT, clique, Hamiltonian cycle |
| NP-hard | ≥ every NP problem (maybe not in NP) | TSP optimisation, halting problem |
2 · NP = 'easy to verify' — a runnable verifier
The defining feature of NP is not that problems are hard to solve but that solutions are easy to check. Subset-sum is the cleanest illustration: given a set of numbers and a target, does some subset sum to it? Finding that subset seems to need exponential search; verifying a proposed subset (the certificate) is a trivial linear-time addition. That asymmetry is NP.
import itertools
def verify_subset_sum(numbers, target, certificate):
"""A VERIFIER for subset-sum: given a proposed subset (the 'certificate'),
check it in polynomial time. Fast verification is the essence of NP."""
if not set(certificate).issubset(set(range(len(numbers)))):
return False
return sum(numbers[i] for i in certificate) == target
def solve_subset_sum_bruteforce(numbers, target):
"""FINDING the certificate is the hard part: 2^n subsets to try."""
for r in range(len(numbers) + 1):
for combo in itertools.combinations(range(len(numbers)), r):
if sum(numbers[i] for i in combo) == target:
return list(combo)
return None
nums = [3, 34, 4, 12, 5, 2]
target = 9
cert = solve_subset_sum_bruteforce(nums, target) # slow: explores 2^n
print("found subset indices :", cert) # e.g. [2, 4] -> 4+5=9
print("verifies? :", verify_subset_sum(nums, target, cert)) # True
print("bogus cert verifies? :", verify_subset_sum(nums, target, [0, 1])) # False
found subset indices : [2, 4]
verifies? : True
bogus cert verifies? : False
3 · Polynomial-time reductions: 3-SAT → clique
A reduction transforms problem A into problem B in polynomial time so that a solver for B solves A. Reductions are how we prove NP-completeness: since Cook-Levin showed 3-SAT is NP-complete, reducing 3-SAT to a new problem proves that problem is NP-hard too. The classic 3-SAT → clique reduction: make one vertex per literal-occurrence, connect vertices in different clauses that are not contradictory; the formula is satisfiable iff the graph has a clique of size k (one true literal per clause). The demo builds the graph and confirms the equivalence on a tiny instance.
import itertools
def three_sat_to_clique(clauses):
"""Reduce 3-SAT to CLIQUE (the textbook Karp reduction).
Build a graph with one vertex per literal-occurrence. Connect two vertices
if they are in DIFFERENT clauses AND are not contradictory (x and NOT x).
A satisfying assignment exists <=> the graph has a clique of size k
(k = number of clauses): pick one true literal per clause, all mutually
consistent -> a clique. literals are ints: 2 means x2, -2 means NOT x2."""
verts = [] # (clause_index, literal)
for ci, clause in enumerate(clauses):
for lit in clause:
verts.append((ci, lit))
edges = set()
for a, b in itertools.combinations(range(len(verts)), 2):
(ci, la), (cj, lb) = verts[a], verts[b]
if ci != cj and la != -lb: # different clause, not contradictory
edges.add((a, b))
return verts, edges
def has_clique(n, edges, k):
"""Brute-force clique check (itself exponential) — used only to CONFIRM the
reduction is faithful on tiny instances."""
adj = {i: set() for i in range(n)}
for a, b in edges:
adj[a].add(b); adj[b].add(a)
for combo in itertools.combinations(range(n), k):
if all(b in adj[a] for a, b in itertools.combinations(combo, 2)):
return True
return False
# (x1 OR x2 OR x3) AND (NOT x1 OR x2 OR x3) — satisfiable (e.g. x2=True)
clauses = [[1, 2, 3], [-1, 2, 3]]
verts, edges = three_sat_to_clique(clauses)
k = len(clauses)
print("literal-vertices :", len(verts))
print("has clique of size k=%d :" % k, has_clique(len(verts), edges, k)) # True
literal-vertices : 6
has clique of size k=2 : True
4 · The limits of computation — the halting problem
NP-hardness is about time. A deeper limit is decidability: some problems have no algorithm at all, at any cost. The halting problem — "does program P halt on input I?" — is the canonical undecidable problem. Turing’s diagonal argument: assume a halts oracle exists, build a program that halts iff the oracle says it loops, then feed it itself — a contradiction. So no such oracle can exist. This is conceptual; the demo sketches the paradox, it does not (and cannot) implement a real oracle.
def halting_paradox_sketch():
"""The halting problem is UNDECIDABLE: no program can decide, for every
(program, input) pair, whether it halts. The classic diagonal argument,
shown as a contradiction (this does NOT run a real 'halts' oracle)."""
def halts(program, inp):
raise NotImplementedError("assume this oracle exists, then derive absurdity")
def paradox(program):
if halts(program, program):
while True: # if it halts, loop forever
pass
else:
return # if it loops, halt
# Now ask: does paradox(paradox) halt?
# - If halts(paradox, paradox) == True -> paradox loops -> it does NOT halt.
# - If halts(paradox, paradox) == False -> paradox halts -> it DOES halt.
# Both cases contradict the oracle. Therefore 'halts' cannot exist.
return "No consistent 'halts' oracle can exist — halting is undecidable."
print(halting_paradox_sketch())
No consistent 'halts' oracle can exist — halting is undecidable.
5 · The classic NP-complete catalogue
You do not need to memorise proofs, but you must recognise these when they appear in disguise — a scheduling ticket that is secretly graph colouring, a routing feature that is secretly TSP. Spotting the pattern saves you from promising a fast exact solution you cannot deliver.
| Problem | Question | Shows up as |
|---|---|---|
| 3-SAT | is a boolean formula satisfiable? | config validation, circuit design |
| Clique / Independent set | k mutually connected / unconnected nodes? | social clusters, conflict-free scheduling |
| Graph colouring | colour nodes with k colours, no clash? | register allocation, timetabling |
| Hamiltonian cycle | visit every node exactly once, return? | route/tour planning |
| TSP (decision) | tour under cost B? | logistics, PCB drilling |
| Subset-sum / Knapsack | hit a target / maximise under a budget? | resource packing, budgeting |
| Set cover / Vertex cover | cover everything with k sets/nodes? | sensor placement, feature selection |
6 · The engineer's playbook: approximation & heuristics
NP-completeness is a statement about worst-case exact solutions on arbitrary inputs. Real work rarely needs that. The practical responses: (1) exact on small n — brute force or DP is fine when n is tiny or the number is small (pseudo-polynomial knapsack); (2) approximation algorithms with a provable ratio (2-approx vertex cover, ln n set cover); (3) heuristics (greedy, local search, simulated annealing) that are fast and usually good but carry no guarantee; (4) solvers — modern SAT/ILP solvers crush huge "hard" instances because real inputs are not adversarial. The first demo compares an exact TSP to a greedy heuristic; the second gives a guaranteed 2-approximation.
import itertools, random
def tsp_bruteforce(dist):
"""Exact travelling-salesman (return to start). O(n!) — only tiny n."""
n = len(dist)
best, best_tour = float("inf"), None
for perm in itertools.permutations(range(1, n)):
tour = (0,) + perm + (0,)
cost = sum(dist[tour[i]][tour[i + 1]] for i in range(n))
if cost < best:
best, best_tour = cost, tour
return best, best_tour
def tsp_nearest_neighbour(dist):
"""A greedy HEURISTIC: always go to the nearest unvisited city. O(n^2).
Fast but NOT optimal — a bounded, practical answer instead of an exact one."""
n = len(dist)
unvisited = set(range(1, n))
tour, cur, cost = [0], 0, 0
while unvisited:
nxt = min(unvisited, key=lambda c: dist[cur][c])
cost += dist[cur][nxt]
cur = nxt; tour.append(cur); unvisited.discard(cur)
cost += dist[cur][0]; tour.append(0)
return cost, tour
random.seed(7)
n = 8
pts = [(random.random(), random.random()) for _ in range(n)]
dist = [[((a[0]-b[0])**2 + (a[1]-b[1])**2) ** 0.5 for b in pts] for a in pts]
exact, _ = tsp_bruteforce(dist)
approx, _ = tsp_nearest_neighbour(dist)
print("exact cost : %.3f" % exact)
print("greedy cost : %.3f" % approx)
print("greedy within %.1f%% of optimal" % (100 * (approx - exact) / exact))
exact cost : 2.320
greedy cost : 2.389
greedy within 3.0% of optimal
def vertex_cover_2approx(edges):
"""2-approximation for minimum vertex cover (NP-hard to solve exactly).
Repeatedly pick any uncovered edge and take BOTH endpoints. The result is
at most 2x the optimum — a provable guarantee, not just a hope."""
cover = set()
remaining = set(map(tuple, (map(lambda e: tuple(sorted(e)), edges))))
while remaining:
u, v = next(iter(remaining))
cover.add(u); cover.add(v)
remaining = {(a, b) for (a, b) in remaining if a not in (u, v) and b not in (u, v)}
return cover
edges = [(0, 1), (1, 2), (2, 3), (3, 0), (1, 3)]
cover = vertex_cover_2approx(edges)
covered = all(a in cover or b in cover for a, b in edges)
print("cover :", sorted(cover))
print("covers all edges :", covered)
print("size <= 2x optimal (guaranteed) :", True)
cover : [0, 1, 2, 3]
covers all edges : True
size <= 2x optimal (guaranteed) : True
✓ Checkpoint — you can move on when you can…
- Give a precise definition of each of P, NP, NP-complete, and NP-hard.
- Explain, with subset-sum, why NP is about verification not solving.
- Sketch a reduction and say what proving 'B is NP-hard' via 3-SAT → B means.
- Reproduce the halting-problem contradiction and distinguish undecidable from intractable.
- Name three real tasks that are secretly NP-complete problems.
- Choose exact / approximation / heuristic / solver for a given NP-hard problem and justify it.
A teammate says: 'This scheduling feature is NP-complete, so it is impossible to build.' What is wrong with that statement, and what should you do instead?
Show answer
Why does proving a reduction from 3-SAT to problem X establish that X is NP-hard, and what extra fact would additionally make X NP-complete?
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: The gap between checking and finding is the heart of NP. A Hamiltonian-path verifier makes it concrete: verification is trivial, finding is NP-complete.
Your task: Write a polynomial-time verifier that checks whether a given vertex ordering is a valid Hamiltonian path of a graph.
Requirements:
- The ordering must be a permutation of all n vertices
- Every consecutive pair must be an actual edge
- Return True/False in polynomial time
- Show one valid and one invalid ordering
💡 Hint: Two checks: the ordering is a permutation of range(n), and each adjacent pair is in the edge set.
Show solution
def verify_hamiltonian_path(n, edges, order):
"""VERIFY a proposed Hamiltonian path (visits every vertex exactly once).
Verification is polynomial; finding one is NP-complete."""
if sorted(order) != list(range(n)):
return False # must be a permutation of all vertices
edge_set = {tuple(sorted(e)) for e in edges}
return all(tuple(sorted((order[i], order[i + 1]))) in edge_set
for i in range(n - 1))
edges = [(0, 1), (1, 2), (2, 3), (0, 3)]
print("0-1-2-3 valid? :", verify_hamiltonian_path(4, edges, [0, 1, 2, 3])) # True
print("0-2-1-3 valid? :", verify_hamiltonian_path(4, edges, [0, 2, 1, 3])) # False
Context: Before appreciating why SAT is hard, implement the exponential baseline — and get the ground truth heuristics are later measured against.
Your task: Decide CNF satisfiability by trying all 2ⁿ truth assignments; return a satisfying assignment or None.
Requirements:
- Enumerate every assignment of n variables
- A clause is satisfied if any literal is true
- The formula holds when all clauses are satisfied
- Return a witness assignment, or None for UNSAT
💡 Hint: Represent a literal as a signed int (-2 = ¬x₂); itertools.product([False,True], repeat=n) enumerates assignments.
Show solution
import itertools
def brute_force_sat(clauses, n_vars):
"""Decide CNF-SAT by trying all 2^n truth assignments. Exponential, but the
ground truth against which heuristics are measured."""
for bits in itertools.product([False, True], repeat=n_vars):
assign = {i + 1: bits[i] for i in range(n_vars)}
def lit_true(lit):
return assign[abs(lit)] if lit > 0 else not assign[abs(lit)]
if all(any(lit_true(l) for l in clause) for clause in clauses):
return assign
return None
clauses = [[1, 2], [-1, 2], [-2, 3]]
sol = brute_force_sat(clauses, 3)
print("satisfying assignment :", sol) # e.g. {1:False,2:True,3:True}
print("unsat example :", brute_force_sat([[1], [-1]], 1)) # None
Context: Reductions are the currency of complexity theory; the Independent-Set ↔ Clique pair via graph complement is the tidiest one to implement.
Your task: Reduce INDEPENDENT SET to CLIQUE and verify the correspondence on a small graph.
Requirements:
- Build the complement graph
- An independent set of size k in G ⇔ a clique of size k in the complement
- Confirm with a brute-force clique check on the complement
- Demonstrate on a path graph where the answer is known
💡 Hint: Two non-adjacent vertices in G become adjacent in the complement — so 'no edges among them' turns into 'all edges among them.'
Show solution
def independent_set_to_clique(n, edges):
"""Reduce INDEPENDENT SET to CLIQUE by complementing the graph:
an independent set in G is exactly a clique in the complement of G."""
present = {tuple(sorted(e)) for e in edges}
complement = [(a, b) for a in range(n) for b in range(a + 1, n)
if (a, b) not in present]
return complement
import itertools
def has_clique(n, edges, k):
adj = {i: set() for i in range(n)}
for a, b in edges:
adj[a].add(b); adj[b].add(a)
return any(all(b in adj[a] for a, b in itertools.combinations(c, 2))
for c in itertools.combinations(range(n), k))
# path 0-1-2-3: {0,2} and {1,3} are independent sets of size 2
edges = [(0, 1), (1, 2), (2, 3)]
comp = independent_set_to_clique(4, edges)
print("independent set size 2 <=> clique size 2 in complement :",
has_clique(4, comp, 2)) # True
Context: Set cover is NP-hard, yet the greedy algorithm has a clean ln n guarantee — and it powers real feature-selection and sensor-placement decisions.
Your task: Implement greedy set cover: repeatedly take the subset covering the most still-uncovered elements.
Requirements:
- Track the uncovered universe
- Each step pick the subset with the largest uncovered intersection
- Stop when everything is covered (or report impossible)
- Return the chosen subsets
💡 Hint: The greedy choice is max(sets, key=len(set & uncovered)) each round; it is at most H(n) ≈ ln n times optimal.
Show solution
def greedy_set_cover(universe, subsets):
"""Greedy approximation for SET COVER (NP-hard). Repeatedly take the subset
covering the most still-uncovered elements. Guarantee: at most H(n) ~ ln n
times the optimal number of subsets."""
universe = set(universe)
uncovered = set(universe)
chosen = []
sets = {k: set(v) for k, v in subsets.items()}
while uncovered:
best = max(sets, key=lambda k: len(sets[k] & uncovered))
if not (sets[best] & uncovered):
break # cannot cover the rest
chosen.append(best)
uncovered -= sets[best]
return chosen, not uncovered
universe = [1, 2, 3, 4, 5]
subsets = {"A": [1, 2, 3], "B": [2, 4], "C": [3, 4], "D": [4, 5]}
chosen, ok = greedy_set_cover(universe, subsets)
print("chosen subsets :", chosen)
print("fully covered :", ok)
Context: Max-Cut is NP-hard but a workhorse in VLSI and clustering; local search is the simplest heuristic and already beats the naive random baseline reliably.
Your task: Solve Max-Cut with local search: start from a random 2-colouring and flip any vertex that increases the number of cut edges until stable.
Requirements:
- Assign each vertex to side 0 or 1 at random
- Flip a vertex when more of its neighbours are on its own side
- Iterate to a local optimum
- Return the cut value and the partition
💡 Hint: A vertex helps the cut by flipping exactly when its same-side neighbour count exceeds its cross-side count.
Show solution
import random
def maxcut_local_search(n, edges, seed=0):
"""MAX-CUT is NP-hard. Local search: start random, flip any vertex that
increases the cut, repeat until stable. A simple 0.5-approximation-quality
heuristic (random assignment alone already expects half the edges cut)."""
rnd = random.Random(seed)
side = [rnd.randint(0, 1) for _ in range(n)]
adj = {i: [] for i in range(n)}
for a, b in edges:
adj[a].append(b); adj[b].append(a)
def cut_value():
return sum(1 for a, b in edges if side[a] != side[b])
improved = True
while improved:
improved = False
for v in range(n):
same = sum(1 for w in adj[v] if side[w] == side[v])
diff = len(adj[v]) - same
if same > diff: # flipping v cuts more edges
side[v] ^= 1
improved = True
return cut_value(), side
edges = [(0, 1), (1, 2), (2, 3), (3, 0), (0, 2)]
value, side = maxcut_local_search(5 if False else 4, edges)
print("cut value :", value)
print("sides :", side)
Context: Resource packing under a budget is 0/1 knapsack — NP-hard in general, yet pseudo-polynomial by DP. Comparing exact DP with a greedy density heuristic is the real trade-off engineers make.
Your task: Implement exact 0/1 knapsack (DP) and a greedy value-density heuristic, then compare their results on a random instance.
Requirements:
- Exact: DP over capacity, O(n · cap) — pseudo-polynomial
- Greedy: sort by value/weight and pack while it fits
- Report both values and the ratio
- State honestly that greedy has no constant-factor guarantee for 0/1 knapsack
💡 Hint: 'Pseudo-polynomial' means polynomial in the capacity value but exponential in its bit-length — which is why huge budgets still hurt.
Show solution
import itertools, random, time
def knapsack_exact(items, cap):
"""0/1 knapsack via DP: O(n * cap) — pseudo-polynomial (polynomial in the
NUMBER cap, exponential in its bit-length). The optimum baseline."""
n = len(items)
dp = [0] * (cap + 1)
for w, v in items:
for c in range(cap, w - 1, -1):
dp[c] = max(dp[c], dp[c - w] + v)
return dp[cap]
def knapsack_greedy(items, cap):
"""Greedy by value density (v/w). Fast heuristic, NOT optimal for 0/1."""
order = sorted(items, key=lambda it: it[1] / it[0], reverse=True)
total = w = 0
for wi, vi in order:
if w + wi <= cap:
w += wi; total += vi
return total
random.seed(1)
items = [(random.randint(1, 10), random.randint(1, 20)) for _ in range(15)]
cap = 30
opt = knapsack_exact(items, cap)
approx = knapsack_greedy(items, cap)
print("exact value :", opt)
print("greedy value :", approx)
print("greedy ratio : %.2f" % (approx / opt))
# Honest note: for 0/1 knapsack the greedy density heuristic has NO constant
# approximation guarantee (unlike fractional knapsack). A true FPTAS exists.