Design under constraints

Each scenario below is a real staff-interview / design-review shape: a set of constraints that cannot all be satisfied. Work them in order. For each one, the pattern is the same — find the constraint that binds first (the one that forces your hand), design to it, then write down, out loud, what you traded away. A design with no stated tradeoff is not finished; it just hasn't found its cost yet.

The move: which constraint binds first advanced

Before any boxes-and-arrows, ask which single constraint you'd break first if you had to. That constraint binds — it forces a structural choice, and everything else bends around it. Latency binds → you can't multi-hop. A cost ceiling binds → you route cheap-first. Data residency binds → the frontier API is off the table. Read the chain below left-to-right: the binding constraint forces a design, which costs you a capability.

Scenario 1 · A 300 ms p99 latency budget advanced

Constraints. A typeahead-style feature must answer within a 300 ms p99 budget, end to end, including network. Accuracy matters but the product is unusable if it's slow. Volume is moderate; cost is not the tight constraint here.

Binding constraint: latency. The tension: the most accurate setup — a large frontier model doing multi-hop reasoning with a retrieval round-trip — is exactly what blows a 300 ms budget. Every accuracy lever you'd reach for adds milliseconds.

Worked design. A 300 ms p99 budget has to be allocated, not hoped for. Spend it: leave headroom for network + serialization, cap the model's share, and make retrieval optional. That forces a small/distilled model, a single hop (no agentic loop), and an exact-match cache in front so hot queries never touch the model at all. Model the allocation before you commit to it.

latency_budget.pydef allocate(budget_ms, fixed, model_p99):
    """fixed: dict of unavoidable costs (network, serialize, retrieval).
    Returns the ms left for the model and whether the chosen model fits."""
    reserved = sum(fixed.values())
    for_model = budget_ms - reserved
    return {"reserved_ms": reserved, "left_for_model_ms": for_model,
            "model_fits": model_p99 <= for_model,
            "over_by_ms": max(0, model_p99 - for_model)}

fixed = {"network_rt": 40, "tls_serialize": 20, "retrieval": 60}
print("frontier (350ms):", allocate(300, fixed, model_p99=350))
print("distilled (140ms):", allocate(300, fixed, model_p99=140))
# Drop retrieval to buy the model more room:
print("distilled, no RAG:", allocate(300, {"network_rt": 40, "tls_serialize": 20}, 140))

frontier (350ms): {'reserved_ms': 120, 'left_for_model_ms': 180, 'model_fits': False, 'over_by_ms': 170}
distilled (140ms): {'reserved_ms': 120, 'left_for_model_ms': 180, 'model_fits': True, 'over_by_ms': 0}
distilled, no RAG: {'reserved_ms': 60, 'left_for_model_ms': 240, 'model_fits': True, 'over_by_ms': 0}

The tradeoff, explicit. The frontier model overshoots by 170 ms — it is simply not an option at this budget. You ship the distilled model and trade peak accuracy for the budget: on the hardest queries the small model will be worse, and you accept that. What you give up: multi-hop reasoning, and — if you drop retrieval to buy headroom — freshness/grounding on the long tail. Failure mode: a cache miss on a cold query still has to fit 300 ms; size the cache and the model for the miss path, not the hit path, or your p99 lies to you.

Scenario 2 · $500/month hard ceiling at 1M requests advanced

Constraints. A summarization endpoint serves 1,000,000 requests/month under a hard $500/month ceiling — not a target, a wall. Quality should be as high as the money allows. Latency is comfortable.

Binding constraint: cost. The tension: $500 / 1M = $0.0005 per request. A frontier model on a real summarization prompt costs multiples of that. You cannot send every request to the good model and stay under the wall.

Worked design. A cascade: a cache absorbs repeats for free; a cheap small model handles the easy majority; only low-confidence cases escalate to the expensive model. The whole design lives or dies on the blended per-request cost landing under $0.0005 — so compute it, don't guess.

routing_cost.pydef blended_cost(n, ceiling, cache_hit, esc_rate, cheap_cost, exp_cost):
    """n requests. cache hits are free; misses go cheap, a fraction escalate to expensive."""
    misses = n * (1 - cache_hit)
    cheap = misses * (1 - esc_rate) * cheap_cost
    expensive = misses * esc_rate * exp_cost
    total = cheap + expensive
    per_req = total / n
    return {"monthly_$": round(total, 2), "per_req_$": round(per_req, 6),
            "under_ceiling": total <= ceiling, "headroom_$": round(ceiling - total, 2)}

# 1M req, $500 wall. 30% cache, 15% escalate; cheap=$0.0002, expensive=$0.004
print("cascade:", blended_cost(1_000_000, 500, 0.30, 0.15, 0.0002, 0.004))
# Naive "everything to the expensive model":
print("all-expensive:", blended_cost(1_000_000, 500, 0.0, 1.0, 0.0002, 0.004))

cascade: {'monthly_$': 539.0, 'per_req_$': 0.000539, 'under_ceiling': False, 'headroom_$': -39.0}
all-expensive: {'monthly_$': 4000.0, 'per_req_$': 0.004, 'under_ceiling': False, 'headroom_$': -3500.0}

Read the numbers. All-expensive is 8× over the wall — never in contention. But the first cascade is also over, by $39: a 15% escalation rate is too generous. This is the honest part of the exercise — the model tells you the plan doesn't fit yet. You have three knobs: raise the cache hit rate, lower the escalation threshold (send fewer to the expensive model), or find a cheaper 'expensive' tier. Tightening escalation to ~10% or cache to ~35% brings it under.

The tradeoff, explicit. You trade quality on the escalation margin for the ceiling: the borderline cases that would have gone to the strong model now get the cheap one, and some of those answers are worse. What you give up: a slice of top-end quality, and the simplicity of one model — a cascade is more moving parts to monitor. Failure mode: the escalation rate is not fixed. If input difficulty drifts up, escalation climbs, and you silently blow the ceiling — so the escalation rate must be a monitored, alerted metric, and you need a load-shedding fallback (serve cheap-only) when the month's budget is spent.

Scenario 3 · No data may leave our network expert

Constraints. Regulated data (health / financial). No request payload may leave the corporate network — a compliance/legal wall, not a preference. The task is hard reasoning over sensitive documents.

Binding constraint: data residency. The tension: the most capable models are closed APIs that require sending the payload out. Residency removes them from the board entirely, regardless of how much better they are.

Worked design. A self-hosted open-weight model on infrastructure you control — the data never crosses the boundary. Everything follows from that one forced move: you now own GPUs, a serving stack (vLLM/TGI), quantization to fit the hardware, capacity planning, and an on-call for the model itself. Route only the non-sensitive metadata (if any) to a stronger API; the sensitive path stays in-house, full stop.

The tradeoff, explicit. You accept a capability gap to keep the data in-house. The open model is measurably weaker on the hardest reasoning, and you pay a standing ops cost that an API doesn't have. That is not a bug in the design — it is the price of the constraint, and it's the right price to pay because the alternative is illegal. What you give up: peak reasoning quality and zero-ops simplicity. Failure mode: a subtle exfiltration path — logs, traces, error payloads, or a third-party observability SaaS — quietly ships the very data you self-hosted to protect. The design isn't done until the whole telemetry path is inside the boundary too.

Scenario 4 · Must work offline / air-gapped expert

Constraints. The system runs on a machine with no internet — a ship, a secure facility, a field laptop. No API calls, no web search, no package pulls at runtime. It must answer usefully with what's on the box.

Binding constraint: connectivity (there is none). The tension: almost every modern LLM convenience — hosted models, server-side tool use, live retrieval, remote MCP servers — assumes a network. Air-gap deletes all of them at once.

Worked design. Everything bundled, local, ahead of time. A quantized open-weight model sized to the on-box hardware (GGUF via llama.cpp/Ollama, or a small vLLM). Any 'tools' are local functions only — file readers, a calculator, a local SQLite — never a network call. RAG becomes a pre-built local index shipped on disk (no live search). Model weights, index, and dependencies are all vendored into the image before it goes offline.

Air-gap build checklist

1. Pick a model that fits the hardware quantized — 4-bit if VRAM is tight (accept the quality hit).
2. Vendor the weights + tokenizer into the image; no download at runtime.
3. Replace every server/tool call with a local equivalent, or drop the feature.
4. Pre-build the retrieval index offline and ship it on disk; refresh only on a manual re-image.
5. Pin and vendor all dependencies — assume pip install will fail at runtime.

The tradeoff, explicit. You trade freshness and top-end capability for autonomy. The model's knowledge is frozen at bundle time; the index goes stale until someone physically re-images the box; the local model is smaller than anything you'd call via API. What you give up: live information, server-side tools, and easy updates. Failure mode: hidden network assumptions — a library that phones home, a tokenizer that lazy-downloads, a 'tool' that quietly hits a URL — turn into hangs or crashes in the field. Test the whole thing with the network physically pulled, not just firewalled.

Scenario 5 · A 10× traffic spike is coming tech-lead

Constraints. A launch/seasonal event will push 10× normal traffic for a few hours. You cannot provision 10× GPUs permanently (cost), and you cannot fall over. The product owner will accept degraded service but not an outage.

Binding constraint: peak capacity vs standing cost. The tension: sizing for the peak wastes money 99% of the time; sizing for the average falls over at the peak. GPU capacity is slow and expensive to scale, so you can't just autoscale your way out in seconds.

Worked design — the tech-lead move. Don't pick a single point on the curve; design a system that degrades in defined stages instead of collapsing. Layer four defenses: a request queue to absorb bursts (trade latency for survival), autoscaling on GPU replicas within a pre-warmed ceiling (accept it's slow), graceful degradation (shrink max_tokens, drop optional RAG, route everything to the cheap/small model under load), and finally load-shedding — reject or 503 the lowest-priority traffic to keep the core alive. Each layer is a tradeoff you pre-decide, so the system chooses correctly under stress instead of a human panicking at 2 a.m.

The tradeoff, explicit. You trade latency, then quality, then completeness — in that order — to never trade availability. Under the spike, users wait longer, get shorter answers from a smaller model, and the least-important requests get turned away. That is a chosen, defensible sequence, not a random failure. What you give up: uniform quality during the peak and some standing cost for the pre-warmed ceiling and queue infra. Failure mode: if you never rehearse it, the degradation paths rot — the 'small model' route 500s, the queue has no max depth and OOMs, load-shedding drops the wrong tier. Degradation is only real if you load-test it before the event.

Grade your design tech-lead

Score any design you produced for these scenarios against the dimensions below. This is the same rubric a design review uses — the goal is not a 'correct' architecture but a defensible one.