System Design & Production Debugging
The technical round of a forward-deployed-engineer interview rarely says "design Twitter." It hands you a client's half-working system and asks what you'd do on Monday morning. This page works eight of those questions end to end — the weak answer that sinks candidates, the strong answer that gets offers, and links into the lessons that teach each piece in depth.
Learning objectives
- Use one repeatable frame — clarify, slice, design, break, measure — for any FDE technical prompt.
- Debug a failing RAG or agent system by isolating the stage that broke, not by tweaking prompts.
- Design the integration pieces FDEs own: eval harnesses, webhook receivers, auth bridges, per-tenant monitoring.
- Tell the difference between a weak and a strong answer, and say why out loud.
30-second recap
- The FDE method: Work backward from the client's outcome, ship the thinnest end-to-end slice, and iterate in a tight feedback loop with real users.
- RAG foundations: Retrieval-augmented generation: find the relevant passages first, then give them to the model as context, so answers come from your documents instead of memory.
- Evaluation & testing: A golden set of real questions plus cheap deterministic checks and an LLM judge, run as a regression gate before every change ships.
1 · How the FDE technical round works essential
Most candidates prepare for the wrong interview. They drill classic system-design prompts — URL shorteners, news feeds, rate limiters — and then sit down in front of an FDE panel that says: "A client's support bot started giving wrong answers last week. Go." There's no blank whiteboard and no clean requirements doc. There is a real-looking system, a symptom, a frustrated customer in the story, and a clock. Candidates who reach for a memorised architecture diagram lose the room within five minutes, because the question was never "can you draw boxes" — it was "can you find what's broken in someone else's environment and fix it without making things worse."
That's the job, so that's the interview. A forward-deployed engineer sits between a product and a customer's messy reality: their data, their legacy systems, their security team, their deadlines. The technical round tests the same instincts the FDE method teaches — work backward from the outcome the client needs, ship the thinnest slice that proves it, and keep a tight feedback loop. Interviewers are listening for judgment under messy constraints: do you ask the right first question, do you isolate before you fix, do you think about how your fix fails, and do you leave behind a way to know it's still working?
One frame handles almost every prompt on this page. Clarify the outcome and the constraints (who's affected, since when, what changed). Slice the problem down to the smallest thing you can test or ship. Design the fix or the system. Break it — say out loud how your design fails and what catches that. Measure — name the number that proves it works and the check that keeps it working. Say the frame's steps as you go; it keeps you and the interviewer on the same map, exactly like walking the pillars in the system-design interview.
- Read left to right: each box is one step you say out loud, in order.
- The blue boxes are where everyone spends their time — understanding and designing.
- The amber "Break" box is the step most candidates skip. Saying "here's how this fails, and here's what catches it" turns a good answer into a senior one.
- The green "Measure" box closes the loop: a number that proves it works and a check that keeps it working after you leave the client.
In short: don't jump to the design. Clarify and slice first, and always end on how it fails and how you'll know.
2 · "Since the re-upload, the bot is confidently wrong" essential
This is the most common FDE debugging prompt, and it's a trap for anyone who starts by editing the prompt. What it really tests is whether you can isolate a fault in a multi-stage pipeline. A RAG system has two very different places to fail — retrieval (the right text never reached the model) and generation (the right text arrived and the model mishandled it) — and the fix for one does nothing for the other. The interviewer also planted a clue: the problem started with a re-upload. A strong candidate grabs that clue immediately.
- Reproduce and pin it down. Collect 5–10 concrete questions that now fail, with the answer the client expected. These become regression tests later.
- Split the pipeline. For each failing question, look at the retrieved chunks. Is the correct passage in the top-k? If not, it's a retrieval/ingestion problem. If it is and the answer is still wrong, it's a generation/grounding problem.
- Follow the clue — ingestion. The failure started with a re-upload, so check what the re-upload changed: did the parser flatten tables or drop headings? Are chunk boundaries now cutting clauses in half? Are the old and new versions of each policy both indexed, so a stale rule outranks the current one? Were documents re-embedded with a different embedding model than the query side uses? (A mismatch quietly turns similarity scores into noise.)
- Fix the root cause. Typical fixes: dedupe by document ID and version, add an
is_currentmetadata filter, re-chunk on section boundaries, re-embed everything with one model. - Leave a guard. Add the failing questions to an eval set that runs before every future re-ingestion, so the next upload can't silently regress the bot.
Why this lands: you bisected the system instead of guessing, you tied the symptom to the change that caused it, and you left the client with a check that prevents a repeat. That last step is what separates an FDE from a firefighter. If the interviewer pushes — "the right chunk is retrieved" — switch cleanly to the generation side: is the answer buried mid-context, does the prompt allow outside knowledge, does the model cite the chunk it used?
Go deeper: chunking strategies · metadata filtering · production RAG failure modes · evals in CI
3 · Hybrid search for contracts intermediate
The interviewer is checking that you know why dense search fails here, not just that "hybrid" is the buzzword. Embeddings compress meaning; they're excellent at paraphrase ("late delivery" ≈ "failure to deliver on time") and poor at exact tokens like clause numbers, part codes, or party names, which all look semantically alike. Keyword search has the opposite profile. The design question is how to combine them without one drowning out the other, and how to keep it fast.
- Two retrievers, one per query type. BM25 (keyword) for exact identifiers; dense vectors for meaning. Run both on every query — you often can't tell in advance which kind it is.
- Fuse by rank, not score. BM25 and cosine scores live on different scales, so adding them is meaningless. Use Reciprocal Rank Fusion: each document scores the sum of
1 / (k + rank)across lists (k ≈ 60). It's simple, needs no tuning data, and is hard to break. - Re-rank the fused shortlist. Fusion is tuned for recall. Pass the top ~50 through a cross-encoder re-ranker, which reads query and passage together, and keep the top 5 for the model.
- Filter before ranking. Contract ID, party, effective date, jurisdiction as metadata filters — a lawyer asking about the current MSA must never get a superseded one.
- Chunk on clause boundaries. Keep the clause number in both the chunk text and its metadata, so "14.3(b)" is findable either way.
- Measure each query type separately. Recall@5 on a labelled set of exact queries and fuzzy ones — an average can hide a collapse in one. Budget the re-ranker's latency (say, +150 ms) and confirm it pays for itself.
Why this lands: you named the root cause (embeddings blur exact tokens), you fused correctly (rank, not score), and you measured the two populations separately. Mentioning that you'd evaluate them separately is a small detail that signals real production experience.
Go deeper: hybrid search with RRF · cross-encoder re-ranking · when re-ranking is worth the latency
4 · The agent that's "randomly" wrong intermediate
"Same input, different output" is the line that tempts people into folklore. The interviewer wants to see you treat non-determinism as an engineering problem: get evidence, find what actually differs between good and bad runs, constrain the action space, and measure a failure rate rather than trusting a single run. There's also a safety angle — an agent that writes to a client's system needs guardrails at the point of action, not just a better prompt.
- Get traces of good and bad runs. The full prompt, the tool list, every tool call and result, the model version. Diff a good run against a bad one. Most "random" failures turn out to be a different input: an extra history turn, a tool that returned an error string the agent misread, a stale project list.
- Fix the tool surface. Overlapping tool names and loose parameters invite wrong choices. Make
projectan enum built from the client's real project list, tighten descriptions so tools don't overlap, and use strict schemas so arguments are always valid. - Guard the action. Validate arguments server-side before anything is written. Require confirmation for irreversible or high-impact writes. Cap steps and retries so a confused agent can't loop.
- Measure a rate, not a run. Turn the failing requests into an eval and run each one, say, 10 times. "Wrong project in 3 of 10" is a baseline you can fix and re-measure; "it worked when I tried it" proves nothing about a stochastic system.
Why this lands: evidence first (traces), structural fixes over prompt tweaks (tool design and validation), safety at the action, and an honest statistical way to prove the fix. If you mention that you'd keep those traces flowing in production so the next "random" failure arrives already diagnosed, you've also answered the follow-up before it's asked.
Go deeper: determinism, honestly · gating input, tools & output · agent observability · tracing an app
5 · Prove it works before the client sees it advanced
This question separates people who have shipped LLM features from people who have demoed them. A demo proves the happy path exists; an evaluation proves how often the feature does the right thing on the client's actual questions — and catches the day a prompt edit or model upgrade quietly breaks it. The interviewer is listening for the layered structure (cheap checks before expensive ones), where the test data comes from, how you trust an LLM judge, and how the evaluation becomes a gate rather than a one-off report.
- Build a golden set from real data. 50–200 real (anonymised) questions from the client's tickets or subject-matter experts, each with the facts a correct answer must contain. Synthetic questions miss what real users actually ask.
- Layer the checks, cheapest first. Deterministic checks (format, required facts present, citations point at real documents) catch a lot for free. Then an LLM-as-judge grades groundedness and whether the question was answered, returning a structured verdict.
- Calibrate the judge. Have a human grade a sample of 30–50 and measure agreement with the judge. A judge you haven't checked against people is just another unverified model.
- Agree on the bar up front. With the client, before the build: for example "≥ 90% grounded on the golden set, zero policy violations." Now "does it work?" has an answer you both signed.
- Make it a gate. Run the suite on every prompt, model, or index change, in CI. A drop below the bar blocks the release.
The judge is a few lines of code in either SDK. The key is a structured verdict — booleans you can count — instead of free-text opinions you'd have to read.
A groundedness judge — in Claude or OpenAI
Same rubric, same Pydantic verdict; only the SDK call changes. Toggle the tab to switch.
judge.pyfrom anthropic import Anthropic
from pydantic import BaseModel
client = Anthropic()
class Verdict(BaseModel):
grounded: bool # every claim supported by the context?
answers_question: bool
reason: str
JUDGE = (
"You grade a support answer against the retrieved context. "
"grounded is false if ANY claim in the answer is not supported by the context."
)
def judge(question: str, context: str, answer: str) -> Verdict:
resp = client.messages.parse(
model="claude-opus-5-5", max_tokens=1024,
system=JUDGE,
messages=[{"role": "user", "content":
f"<question>{question}</question>\n"
f"<context>{context}</context>\n"
f"<answer>{answer}</answer>"}],
output_format=Verdict,
)
if resp.parsed_output is None: # e.g. a refusal — never count it as a pass
raise RuntimeError(f"judge returned no verdict: {resp.stop_reason}")
return resp.parsed_output
judge.pyfrom openai import OpenAI
from pydantic import BaseModel
client = OpenAI()
class Verdict(BaseModel):
grounded: bool # every claim supported by the context?
answers_question: bool
reason: str
JUDGE = (
"You grade a support answer against the retrieved context. "
"grounded is false if ANY claim in the answer is not supported by the context."
)
def judge(question: str, context: str, answer: str) -> Verdict:
resp = client.responses.parse(
model="gpt-5.5",
instructions=JUDGE,
input=(
f"<question>{question}</question>\n"
f"<context>{context}</context>\n"
f"<answer>{answer}</answer>"
),
text_format=Verdict,
)
if resp.output_parsed is None: # e.g. a refusal — never count it as a pass
raise RuntimeError("judge returned no verdict")
return resp.output_parsed
Verdictis the contract: two booleans you can count, plus areasonyou read only when a case fails.- The rubric is strict on purpose — any unsupported claim fails groundedness. A lenient judge reports a high score that doesn't mean anything.
- Question, context, and answer go in tagged sections so the judge can't confuse the answer it's grading with the evidence.
- The SDK parses the reply straight into
Verdict(messages.parse→parsed_outputon Claude,responses.parse→output_parsedon OpenAI). If no verdict comes back, the case raises instead of silently counting as a pass.
The harness around it is vendor-neutral. Deterministic checks run first because they're free; the judge only sees answers that already passed them.
run_eval.pyimport json
from judge import judge
def run_eval(answer_fn, path="golden.jsonl", bar=0.90) -> bool:
cases = [json.loads(line) for line in open(path)]
passed = 0
for case in cases:
answer, context = answer_fn(case["question"])
# 1. cheap deterministic check first: required facts present?
if not all(f.lower() in answer.lower() for f in case["must_contain"]):
continue
# 2. only then pay for the judge
v = judge(case["question"], context, answer)
passed += v.grounded and v.answers_question
rate = passed / len(cases)
print(f"pass rate {rate:.0%} (bar {bar:.0%})")
return rate >= bar # CI fails the build when this is False
Why this lands: real data, a layered and costed design, a judge you've verified against humans, a bar the client agreed to, and a gate that keeps working after you leave. That's the whole evaluation story in five sentences — practise saying it that compactly.
Go deeper: the evaluation pyramid · LLM-as-judge · evaluating production RAG · gating deploys on evals
6 · A webhook receiver that never loses an order advanced
Integration work is half of an FDE's job, and webhooks are where it goes wrong most often. The hidden fact the interviewer wants you to surface: webhook senders retry when they don't get a fast success response, so duplicates aren't an edge case — they're guaranteed. "Exactly once" doesn't exist on the wire; you build it from at-least-once delivery plus idempotent processing. Then they want to hear what happens when processing fails, and what happens when the sender simply never sends.
- Receive thin, acknowledge fast. Verify the signature (usually an HMAC of the body with a shared secret), write the raw event to a durable queue or table, return 2xx. No business logic in the request path.
- Make processing idempotent. Use the sender's event ID as an idempotency key, stored under a unique constraint. Processing becomes "insert key, then upsert the order" in one transaction, so a duplicate event is a harmless no-op.
- Retry with backoff, then dead-letter. Workers retry failed events with exponential backoff and jitter. After N attempts the event goes to a dead-letter queue, which raises an alert and comes with a replay tool once the bug is fixed.
- Don't trust ordering. Events can arrive out of order. Store a version or timestamp per order and ignore updates older than what you already have.
- Reconcile. On a schedule, pull the ERP's recent order list and compare it with yours. That's the only way to catch events the sender never delivered at all.
- Watch it. Received vs processed counts, processing lag, DLQ depth — with alerts on the last two.
Why this lands: you named the at-least-once reality, built idempotency on a real key, handled poison messages, and closed the "sender never sent it" gap with reconciliation — the step most candidates miss. Clients remember the engineer whose integration never lost an order.
Go deeper: message queues · the fallacies of distributed computing · error handling & resiliency
7 · "Your dashboard says we're fine. We're not." professional
The question is really about averages lying. An aggregate p95 across 40 tenants is dominated by the biggest tenants' traffic; a small tenant can be suffering badly and barely move the global number. The interviewer wants to hear that you'd measure what each customer actually experiences, find the cause behind this tenant's pain, and fix it in a way that protects every tenant — while answering the client with their own numbers, not yours.
- Name the flaw. A global percentile hides per-tenant experience. The SLA is a promise to each tenant, so it has to be measured per tenant.
- Tag everything. Every request carries tenant ID, feature, model, and prompt version. Compute per-tenant p50/p95, error rate, and cost; set SLOs per tenant tier with burn-rate alerts.
- Find this tenant's cause. Common culprits: their documents are much larger, so prompts are longer and slower to process; or they share a rate limit with a big tenant whose nightly batch starves everyone else (a noisy neighbour).
- Fix with isolation. Per-tenant quotas and fair queueing, a separate lane for batch work, and context budgets for oversized documents. Apply the same isolation to data: tenant filters enforced server-side in retrieval, never trusted from the client.
- Close the loop with the client. Show them their own latency history before and after the fix. That's what rebuilds trust.
Why this lands: you questioned the metric instead of the customer, you designed for per-tenant truth, and you fixed the shared-resource cause structurally. Mentioning data isolation alongside performance isolation shows you think about multi-tenancy as a whole, not just latency.
Go deeper: the four monitoring signals · cost attribution — tag everything · budgets, quotas & routing · tenancy isolation
8 · Bridging OAuth 1.0a to an OAuth 2.0 platform professional
Legacy auth shows up in almost every enterprise deployment, and this prompt checks whether you understand the protocols or just the acronyms. The two aren't versions of the same thing. OAuth 2.0 mostly uses bearer tokens over TLS — whoever holds the token can call the API. OAuth 1.0a signs every request: each call carries a nonce, a timestamp, and a signature (typically HMAC-SHA1) computed from the request plus the consumer secret and token secret. You can't "convert" one into the other; you have to translate between them at a boundary you control.
- Put an adapter at the boundary. A small integration service: our platform calls it with our normal OAuth 2.0 / internal auth, and it calls the client's system with OAuth 1.0a. Neither side changes.
- Keep secrets in one place. The adapter holds the consumer key/secret and the per-user token/secret in a secrets manager. They never reach our platform or the browser.
- Sign with a proven library. Never hand-roll signing. The classic bugs are parameter encoding and ordering in the signature base string, clock skew (the server rejects old timestamps), and reused nonces — good libraries handle all three.
- Expose only what's needed. The adapter offers just the handful of endpoints the integration uses, with rate limits and an audit log (with secrets redacted). It's least privilege applied to a legacy system.
- Plan the token lifecycle. OAuth 1.0a access tokens are often long-lived, so agree a rotation and revocation process with the client's IT team, and test end to end against their sandbox before production.
Why this lands: you showed you know why the protocols differ (signing vs bearer), chose the adapter pattern so neither side rewrites, and covered the operational details — secrets, signing pitfalls, least privilege, rotation — that make legacy integrations survive contact with a security review.
Go deeper: API gateways — the front door · vendor security review · LLM security & safety
9 · Six months later, quality is sliding tech-lead
This is the tech-lead question, and the word that matters is "nobody can say why." A slow decline with no explanation is an observability failure before it's a model failure. The interviewer wants to hear that you'd design the system so a decline is detected early, explained by data, and owned by someone — and that you know the usual causes: the client's content drifted, an upstream data feed broke quietly, or an unlogged change (model upgrade, prompt edit, index rebuild) shifted behaviour.
- Trace every request. Prompt/template version, model ID, retrieved document IDs and scores, token counts, latency, cost, and user feedback — with PII redacted before anything is stored.
- Check the data before it reaches users. On the feed into the index: freshness (last successful sync), volume against a baseline, schema changes, empty or duplicate documents. Alert when a sync silently delivers half the usual documents.
- Measure quality online. Each week, sample production traffic through the same judge used before launch (§5) and track the trend per category, alongside thumbs-down and escalation rates.
- Log every change. Model versions, prompt edits, and index rebuilds go on a timeline next to the quality chart, so a dip lines up with its cause instead of becoming a mystery.
- Hand over ownership. A named owner on the client side, a runbook for "quality dropped," and a monthly review. FDEs rotate off accounts; the system has to outlive your involvement.
Why this lands: you reframed "the model got worse" as "we couldn't see why," covered data, quality, and change tracking as one system, and finished on ownership after you leave — the tech-lead concern. It also ties back to everything above: the regression set from §2 and the judge from §5 become the production monitors here.
Go deeper: detecting drift & live regressions · tracing an agent · online monitoring · observability
🪜 Practice ladder beginner → industry
- Beginner: answer §2 out loud in three minutes, naming each step of the Clarify → Slice → Design → Break → Measure frame as you go.
- Easy: for §3 (hybrid search), write down three follow-ups an interviewer would ask after your answer, and your reply to each.
- Core: build a 20-question golden set for a RAG app you've written, run the Lab FI1.1 judge over it, and record the pass rate.
- Stretch: whiteboard §6 (webhooks) in 15 minutes, including the dead-letter replay tool and the reconciliation job.
- Hard: run a mock with a peer who plays a client and changes a key requirement halfway through your answer to §5 (the evaluation design). Practise adapting without starting over.
- Industry: write a one-page "day-one observability" spec (§9) for a real system you've worked on — what's traced, what alerts, who owns it.
✓ Checkpoint — you can move on when you can…
- Apply Clarify → Slice → Design → Break → Measure to an unfamiliar prompt without notes.
- Isolate a RAG failure to retrieval or generation, and explain why the re-upload clue points to ingestion.
- Design an evaluation with a golden set, layered checks, a calibrated judge, and a CI gate.
- Design a webhook receiver that survives duplicates, failures, and missed events.
- Explain why aggregate percentiles hide per-tenant pain, and what to measure instead.
Knowledge check check yourself
In an FDE interview, a RAG assistant started giving wrong answers right after the client re-uploaded their documents. What is the first diagnostic split you make, and why?
Show answer
Why is "acknowledge fast, process from a queue with idempotency keys" the standard design for a webhook receiver that must not lose or duplicate orders?