Design under constraints

Each scenario below is a real staff-interview / design-review shape: a set of constraints that cannot all be satisfied. Work them in order. For each one, the pattern is the same — find the constraint that binds first (the one that forces your hand), design to it, then write down, out loud, what you traded away. A design with no stated tradeoff is not finished; it just hasn't found its cost yet.

The move: which constraint binds first advanced

Here's the design review that goes wrong every week. A team is asked for "a fast, cheap, private, always-on assistant," so they open a whiteboard and start drawing the ideal system — best model, live retrieval, full observability — then spend the next hour discovering, one requirement at a time, that the ideal system violates every constraint they were given. By the end everyone is exhausted, the design is a compromise nobody chose on purpose, and the tradeoff that actually decided the architecture was never said out loud. The staff engineer in the room avoids all of that by asking one question first: which constraint binds?

Think of it like packing a suitcase for a flight. You have a weight limit, a size limit, and a list of things you want to bring. The amateur packs everything they'd like and then panics at the gate. The experienced traveler asks up front: which limit will I hit first? If it's weight, that decides what goes in — the heavy boots come out no matter how much you love them, and everything else is arranged around that one hard number. The binding constraint is exactly that: of all the requirements pulling against each other, it's the one that reaches its limit first and therefore forces a structural decision. Everything else is a dial you can still turn; the binding constraint is a wall you cannot move.

Why does naming it matter so much? Because a system has usually one — occasionally two — constraints that genuinely remove options from the board, and a long tail of constraints that are merely uncomfortable. Latency binding means you physically cannot afford a second network hop; it isn't "try to be fast," it's "the multi-hop architecture is dead." A hard cost ceiling means the expensive model can't be your default path, full stop. Data residency means the frontier API is not a candidate at any price. Confuse a dial for a wall — "let's just optimize cost later" — and you'll design the whole system around the wrong axis and rebuild it after the first review. The common mistake is treating every requirement as equally negotiable and trying to optimize all of them at once; you end up with a design that's mediocre on each and decisively good on none.

The discipline, then, is a chain, and it runs in one direction: the binding constraint forces a design, and that design costs you a capability. Naming the constraint isn't the finish line — it's what lets you predict, before you build, exactly what you'll have to give up, so you can decide on purpose whether that price is worth paying. Read the diagram below left-to-right as that chain.

Before any boxes-and-arrows, ask which single constraint you'd break first if you had to. That constraint binds — it forces a structural choice, and everything else bends around it. Latency binds → you can't multi-hop. A cost ceiling binds → you route cheap-first. Data residency binds → the frontier API is off the table. Read the chain below left-to-right: the binding constraint forces a design, which costs you a capability.

Scenario 1 · A 300 ms p99 latency budget advanced

You've felt this one as a user, even if you've never designed it. Type into a search box that answers instantly and you keep typing; type into one that stalls for half a second per keystroke and you give up and close the tab. For an interactive feature, slowness isn't a quality problem — it's an existence problem. That's why a latency budget, when it's real, outranks accuracy: a slightly-less-perfect answer that arrives is worth infinitely more than a perfect one the user never waited for.

The first thing to internalize is what p99 actually means, because the whole scenario hinges on it. "p99 = 300 ms" does not mean "usually about 300 ms." It means 99 out of every 100 requests must finish within 300 ms — so the number you're being graded on is the slowest common request, not the typical one. Picture a restaurant that promises "seated within 5 minutes." Averaging the wait tells you nothing useful; the promise is about the person at the back of the line on a busy night. In a request path, the person at the back is the request that misses the cache, hits the big model, and waits on a retrieval round-trip. If that path is over budget, your p99 is over budget — no matter how fast the easy requests are.

The second idea is that a latency budget is a fixed pot of milliseconds you must allocate, not a wish. Every stage on the path spends from the same 300 ms: the network round-trip, TLS and serialization, an optional retrieval hop, and finally the model's own inference time. They add up in series, so the model doesn't get 300 ms — it gets whatever is left after the unavoidable costs are subtracted. This is the crucial framing: you reserve the fixed costs first, and only then ask "does the model I want fit in the remainder?" A frontier model that takes 350 ms by itself can't fit a 300 ms total no matter what you do to the rest.

That reframing is what makes the design decisions feel forced rather than arbitrary. Once the budget is allocated, a small/distilled model, a single hop with no agentic loop, and a front cache stop being "optimizations" and become the only shapes that fit the pot. The common mistake here is optimizing the average — pointing at a high cache-hit rate and declaring victory — when the budget is defined by the miss. Below, we make the allocation concrete with a tiny model so the tradeoff is a number, not a hope.

Constraints. A typeahead-style feature must answer within a 300 ms p99 budget, end to end, including network. Accuracy matters but the product is unusable if it's slow. Volume is moderate; cost is not the tight constraint here.

Binding constraint: latency. The tension: the most accurate setup — a large frontier model doing multi-hop reasoning with a retrieval round-trip — is exactly what blows a 300 ms budget. Every accuracy lever you'd reach for adds milliseconds.

Worked design. A 300 ms p99 budget has to be allocated, not hoped for. Spend it: leave headroom for network + serialization, cap the model's share, and make retrieval optional. That forces a small/distilled model, a single hop (no agentic loop), and an exact-match cache in front so hot queries never touch the model at all. Model the allocation before you commit to it.

latency_budget.pydef allocate(budget_ms, fixed, model_p99):
    """fixed: dict of unavoidable costs (network, serialize, retrieval).
    Returns the ms left for the model and whether the chosen model fits."""
    reserved = sum(fixed.values())
    for_model = budget_ms - reserved
    return {"reserved_ms": reserved, "left_for_model_ms": for_model,
            "model_fits": model_p99 <= for_model,
            "over_by_ms": max(0, model_p99 - for_model)}

fixed = {"network_rt": 40, "tls_serialize": 20, "retrieval": 60}
print("frontier (350ms):", allocate(300, fixed, model_p99=350))
print("distilled (140ms):", allocate(300, fixed, model_p99=140))
# Drop retrieval to buy the model more room:
print("distilled, no RAG:", allocate(300, {"network_rt": 40, "tls_serialize": 20}, 140))

frontier (350ms): {'reserved_ms': 120, 'left_for_model_ms': 180, 'model_fits': False, 'over_by_ms': 170}
distilled (140ms): {'reserved_ms': 120, 'left_for_model_ms': 180, 'model_fits': True, 'over_by_ms': 0}
distilled, no RAG: {'reserved_ms': 60, 'left_for_model_ms': 240, 'model_fits': True, 'over_by_ms': 0}

The tradeoff, explicit. The frontier model overshoots by 170 ms — it is simply not an option at this budget. You ship the distilled model and trade peak accuracy for the budget: on the hardest queries the small model will be worse, and you accept that. What you give up: multi-hop reasoning, and — if you drop retrieval to buy headroom — freshness/grounding on the long tail. Failure mode: a cache miss on a cold query still has to fit 300 ms; size the cache and the model for the miss path, not the hit path, or your p99 lies to you.

Scenario 2 · $500/month hard ceiling at 1M requests advanced

This is the scenario that quietly kills promising AI features. The demo is a hit, leadership says "ship it," and three weeks after launch someone in finance forwards a $4,000 invoice with a single question mark in the subject line. The feature worked perfectly; it was just never affordable at the volume it succeeded at. A hard cost ceiling is the constraint that turns "make it good" into "make it good for less than a fraction of a cent," and the humbling part is that the math is unforgiving in a way latency never is — you can't cache your way out of arithmetic you got wrong.

The move that separates a real cost design from wishful thinking is to divide first. A $500/month ceiling over 1,000,000 requests isn't an abstract budget — it's $0.0005 per request, and that per-request number is the wall you actually design against. Do that division before anything else, because it instantly tells you which models are even candidates. If a single frontier-model summarization call costs several tenths of a cent, then sending every request to it is mathematically impossible under the ceiling, and no amount of engineering cleverness changes that. The ceiling isn't a target you aim at; it's a line the blended average of your entire traffic must land under.

The architectural answer to a per-request wall is a cascade, and it's worth understanding why a cascade is the natural shape rather than just accepting it. Real traffic is not uniform: many requests repeat (a cache serves those for free), most of the rest are easy enough for a cheap small model to handle well, and only a minority are genuinely hard enough to justify the expensive model. A cascade matches the cost of each request to its difficulty — free for repeats, cheap for the easy majority, expensive only for the escalated tail. The blended cost is a weighted average of those tiers, and the whole design succeeds or fails on whether that weighted average clears $0.0005.

Here's the trap, and it's why this scenario teaches quantification harder than any other: the escalation rate — the fraction you send to the expensive model — feels like a small knob, but it dominates the blended cost. Guess it too generously and you sail past the ceiling while believing you're fine. So you don't guess; you compute the blended cost, read the number honestly even when it says "over budget," and then tighten cache rate or escalation until it fits. The common mistake is designing the cascade, feeling clever, and never doing the arithmetic that proves it actually fits. The lab below does exactly that — and, tellingly, its first plausible-looking cascade comes out over the wall.

Constraints. A summarization endpoint serves 1,000,000 requests/month under a hard $500/month ceiling — not a target, a wall. Quality should be as high as the money allows. Latency is comfortable.

Binding constraint: cost. The tension: $500 / 1M = $0.0005 per request. A frontier model on a real summarization prompt costs multiples of that. You cannot send every request to the good model and stay under the wall.

Worked design. A cascade: a cache absorbs repeats for free; a cheap small model handles the easy majority; only low-confidence cases escalate to the expensive model. The whole design lives or dies on the blended per-request cost landing under $0.0005 — so compute it, don't guess.

routing_cost.pydef blended_cost(n, ceiling, cache_hit, esc_rate, cheap_cost, exp_cost):
    """n requests. cache hits are free; misses go cheap, a fraction escalate to expensive."""
    misses = n * (1 - cache_hit)
    cheap = misses * (1 - esc_rate) * cheap_cost
    expensive = misses * esc_rate * exp_cost
    total = cheap + expensive
    per_req = total / n
    return {"monthly_$": round(total, 2), "per_req_$": round(per_req, 6),
            "under_ceiling": total <= ceiling, "headroom_$": round(ceiling - total, 2)}

# 1M req, $500 wall. 30% cache, 15% escalate; cheap=$0.0002, expensive=$0.004
print("cascade:", blended_cost(1_000_000, 500, 0.30, 0.15, 0.0002, 0.004))
# Naive "everything to the expensive model":
print("all-expensive:", blended_cost(1_000_000, 500, 0.0, 1.0, 0.0002, 0.004))

cascade: {'monthly_$': 539.0, 'per_req_$': 0.000539, 'under_ceiling': False, 'headroom_$': -39.0}
all-expensive: {'monthly_$': 4000.0, 'per_req_$': 0.004, 'under_ceiling': False, 'headroom_$': -3500.0}

Read the numbers. All-expensive is 8× over the wall — never in contention. But the first cascade is also over, by $39: a 15% escalation rate is too generous. This is the honest part of the exercise — the model tells you the plan doesn't fit yet. You have three knobs: raise the cache hit rate, lower the escalation threshold (send fewer to the expensive model), or find a cheaper 'expensive' tier. Tightening escalation to ~10% or cache to ~35% brings it under.

The tradeoff, explicit. You trade quality on the escalation margin for the ceiling: the borderline cases that would have gone to the strong model now get the cheap one, and some of those answers are worse. What you give up: a slice of top-end quality, and the simplicity of one model — a cascade is more moving parts to monitor. Failure mode: the escalation rate is not fixed. If input difficulty drifts up, escalation climbs, and you silently blow the ceiling — so the escalation rate must be a monitored, alerted metric, and you need a load-shedding fallback (serve cheap-only) when the month's budget is spent.

Scenario 3 · No data may leave our network expert

The first two scenarios gave you dials; this one gives you a wall — and the difference is the whole lesson. Latency and cost are constraints you optimize toward: you can be a little faster, a little cheaper, and "better" is always meaningful. Data residency is binary. The data either stays inside the boundary or it doesn't; there is no "mostly compliant." A latency miss is a slow response and an apologetic Slack message. A residency miss is a regulated-data breach, a legal event, possibly a fine and a headline. Once you feel that asymmetry, the design stops being a negotiation and becomes a filter.

What a binary wall does — and what makes this an expert-level scenario — is remove options from the board entirely, before you evaluate them on merit. The most capable models on the market are closed APIs; using them means the request payload travels to a vendor's servers. Under a no-egress rule, that single fact disqualifies the frontier API completely, no matter how much better it is at the task. This is a genuinely different mental motion from optimization: you're not comparing candidates and picking the best, you're crossing out every candidate that touches the network and picking the best of what remains. The forced move is a self-hosted open-weight model on infrastructure you control, and everything downstream — owning GPUs, running a serving stack, quantizing to fit the hardware, carrying an on-call — follows mechanically from that one non-negotiable decision.

Now the part that trips up even strong engineers, and the reason this scenario is worth its own section: a binding constraint constrains the whole system, not just the component you were thinking about. When teams hear "no data leaves the network," they picture the model call and stop there. But residency is a property of the entire request path — and request data flows through far more than inference. It flows into logs, into distributed traces, into error payloads sent to a crash reporter, into an APM dashboard, into backups. Every one of those is an egress point. Self-host the model, then pipe your prompts to a hosted logging SaaS for "observability," and you've defeated the constraint through the plumbing — the data left through a side door you forgot you'd built.

The common mistake, then, isn't choosing the wrong model; it's declaring the constraint satisfied the moment inference is in-house, while telemetry quietly ships the very data you self-hosted to protect. The design isn't finished until you can trace every byte of request data and confirm none of it reaches a network you don't control. The comparison table and the worked design below make the forced tradeoff — and the leak you have to hunt for — explicit.

Constraints. Regulated data (health / financial). No request payload may leave the corporate network — a compliance/legal wall, not a preference. The task is hard reasoning over sensitive documents.

Binding constraint: data residency. The tension: the most capable models are closed APIs that require sending the payload out. Residency removes them from the board entirely, regardless of how much better they are.

Worked design. A self-hosted open-weight model on infrastructure you control — the data never crosses the boundary. Everything follows from that one forced move: you now own GPUs, a serving stack (vLLM/TGI), quantization to fit the hardware, capacity planning, and an on-call for the model itself. Route only the non-sensitive metadata (if any) to a stronger API; the sensitive path stays in-house, full stop.

The tradeoff, explicit. You accept a capability gap to keep the data in-house. The open model is measurably weaker on the hardest reasoning, and you pay a standing ops cost that an API doesn't have. That is not a bug in the design — it is the price of the constraint, and it's the right price to pay because the alternative is illegal. What you give up: peak reasoning quality and zero-ops simplicity. Failure mode: a subtle exfiltration path — logs, traces, error payloads, or a third-party observability SaaS — quietly ships the very data you self-hosted to protect. The design isn't done until the whole telemetry path is inside the boundary too.

Scenario 4 · Must work offline / air-gapped expert

If residency was a wall, air-gap is the same wall built one layer higher — and the extra layer changes everything. "No data leaves the network" still assumes a network exists; you just keep the data inside it. Air-gapped removes the network itself. Picture a system on a submarine, in a classified facility, or on a field laptop in a place with no signal: there is no API to call, no web search to run, no pip install to reach for when a dependency is missing, no license server to phone. Everything the system will ever need at runtime must already be on the box before it goes dark. Engineers who've only ever built connected systems consistently underestimate how many quiet network dependencies they've been leaning on — and each one becomes a crash the moment the cable is pulled.

The reframing that makes air-gap tractable is this: the deliverable stops being a running service and becomes a self-contained artifact. In a normal deployment, "the app" is code that reaches out to models, indexes, and package registries as it runs. Air-gapped, "the app" is a single bundle that contains all of those — quantized model weights, the tokenizer, a pre-built retrieval index, every pinned dependency — vendored in ahead of time. Anything that would normally be fetched on demand has to be either baked in or deleted from the design. A "tool" the agent can call is only allowed if it's a local function: a file reader, a calculator, a local SQLite query. The instant a tool would hit a URL, it's off the table.

That constraint reshapes retrieval in a way worth naming, because it's the least obvious casualty. RAG normally implies a live index you can query and update. Air-gapped, the index becomes a frozen snapshot shipped on disk — built offline, carried across the gap, and stale until a human physically re-images the machine. There is no "refresh from source"; freshness is traded away wholesale for autonomy. Similarly, model quality is capped by what fits the on-box hardware after quantization, which on a tight machine may mean accepting a 4-bit model and its accuracy hit. You are choosing a smaller, older, frozen system on purpose because the alternative — needing the network — simply doesn't exist in this world.

The common mistake is assuming "offline" is just "online minus the API calls," and shipping something that looks self-contained but isn't: a library that lazy-downloads a model file on first use, a tokenizer that fetches vocab from a CDN, a "local" tool that quietly resolves a hostname. Those hidden assumptions don't error politely — they hang or crash in the field where nobody can debug them. The only honest test is to pull the network physically, not just firewall it, and see if the whole thing still answers. The build checklist below is the discipline that forces every runtime dependency into the bundle before it's too late to add one.

Constraints. The system runs on a machine with no internet — a ship, a secure facility, a field laptop. No API calls, no web search, no package pulls at runtime. It must answer usefully with what's on the box.

Binding constraint: connectivity (there is none). The tension: almost every modern LLM convenience — hosted models, server-side tool use, live retrieval, remote MCP servers — assumes a network. Air-gap deletes all of them at once.

Worked design. Everything bundled, local, ahead of time. A quantized open-weight model sized to the on-box hardware (GGUF via llama.cpp/Ollama, or a small vLLM). Any 'tools' are local functions only — file readers, a calculator, a local SQLite — never a network call. RAG becomes a pre-built local index shipped on disk (no live search). Model weights, index, and dependencies are all vendored into the image before it goes offline.

Air-gap build checklist

1. Pick a model that fits the hardware quantized — 4-bit if VRAM is tight (accept the quality hit).
2. Vendor the weights + tokenizer into the image; no download at runtime.
3. Replace every server/tool call with a local equivalent, or drop the feature.
4. Pre-build the retrieval index offline and ship it on disk; refresh only on a manual re-image.
5. Pin and vendor all dependencies — assume pip install will fail at runtime.

The tradeoff, explicit. You trade freshness and top-end capability for autonomy. The model's knowledge is frozen at bundle time; the index goes stale until someone physically re-images the box; the local model is smaller than anything you'd call via API. What you give up: live information, server-side tools, and easy updates. Failure mode: hidden network assumptions — a library that phones home, a tokenizer that lazy-downloads, a 'tool' that quietly hits a URL — turn into hangs or crashes in the field. Test the whole thing with the network physically pulled, not just firewalled.

Scenario 5 · A 10× traffic spike is coming tech-lead

Every engineer has watched a service die at exactly the moment it mattered most. The launch goes viral, the flash sale hits, the news picks up the product — the thing you wanted to happen happens — and the system falls over precisely because it succeeded. What makes this a tech-lead scenario rather than an engineering one is that the naïve fixes are both wrong in instructive ways. Provision for the 10× peak permanently and you burn money on idle GPUs 99% of the time. Provision for the average and you collapse at the peak. And unlike a stateless web tier you can autoscale in seconds, GPU capacity is slow and expensive to add, so "just autoscale" doesn't save you when the spike arrives faster than replicas can warm.

The mental shift that resolves it: stop trying to pick a single point on the capacity curve, and instead design a system that degrades in defined stages. The product owner already told you the shape of the answer — they'll accept degraded service but not an outage. That single sentence is permission to trade quality for availability, and it turns an impossible sizing problem into a tractable sequencing problem: not "how much capacity?" but "in what order do I give things up as load climbs?"

The order is not arbitrary — it runs from least painful to most, and each rung buys time for the next to be a last resort rather than a first reflex:

• Queue first — trade latency. A request queue absorbs a burst by making users wait instead of fail. Latency is the cheapest thing to spend, so it goes first. (A queue with no bounded depth, though, is a memory bomb — that caveat matters later.)
• Autoscale next — add capacity within a pre-warmed ceiling. Spin up more GPU replicas up to a cap you provisioned in advance. It genuinely helps, but it's slow and bounded, which is exactly why it can't be your only defense.
• Degrade quality — trade accuracy for throughput. Shrink max_tokens, drop the optional RAG hop, route everything to the small/cheap model. Users get worse answers rather than no answers.
• Shed load — the last resort. Reject or 503 the lowest-priority traffic to keep the core alive. An outage for the least-important 10% beats an outage for everyone.

The reason this is a design and not a reflex is that every one of those trades is pre-decided. When the system is drowning at 2 a.m., you do not want a human choosing which corners to cut — you want the system to already know, because you decided in calm daylight and load-tested it. The common mistake is drawing only the happy path and treating the degradation ladder as something you'll "figure out under pressure"; the paths you never rehearse rot silently — the small-model route starts 500ing, the queue OOMs, load-shedding drops the wrong tier — and you discover it during the incident. The diagram below lays out the ladder in order so the sacrifice sequence is explicit and defensible in the post-incident review.

Constraints. A launch/seasonal event will push 10× normal traffic for a few hours. You cannot provision 10× GPUs permanently (cost), and you cannot fall over. The product owner will accept degraded service but not an outage.

Binding constraint: peak capacity vs standing cost. The tension: sizing for the peak wastes money 99% of the time; sizing for the average falls over at the peak. GPU capacity is slow and expensive to scale, so you can't just autoscale your way out in seconds.

Worked design — the tech-lead move. Don't pick a single point on the curve; design a system that degrades in defined stages instead of collapsing. Layer four defenses: a request queue to absorb bursts (trade latency for survival), autoscaling on GPU replicas within a pre-warmed ceiling (accept it's slow), graceful degradation (shrink max_tokens, drop optional RAG, route everything to the cheap/small model under load), and finally load-shedding — reject or 503 the lowest-priority traffic to keep the core alive. Each layer is a tradeoff you pre-decide, so the system chooses correctly under stress instead of a human panicking at 2 a.m.

The tradeoff, explicit. You trade latency, then quality, then completeness — in that order — to never trade availability. Under the spike, users wait longer, get shorter answers from a smaller model, and the least-important requests get turned away. That is a chosen, defensible sequence, not a random failure. What you give up: uniform quality during the peak and some standing cost for the pre-warmed ceiling and queue infra. Failure mode: if you never rehearse it, the degradation paths rot — the 'small model' route 500s, the queue has no max depth and OOMs, load-shedding drops the wrong tier. Degradation is only real if you load-test it before the event.

Grade your design tech-lead

Score any design you produced for these scenarios against the dimensions below. This is the same rubric a design review uses — the goal is not a 'correct' architecture but a defensible one.