Big Tech AI-Engineering Patterns

1 · Implement softmax & cross-entropy (A4) very common

Here's the moment this question decides your loop. You write softmax in four confident lines, it passes the tiny example, and the interviewer quietly types one big number into the input — say a logit of 1000. Your code returns nan. That single nan is the whole point of the round: it separates people who memorized the softmax formula from people who understand that a formula on paper and a formula on a real machine are different animals. The maths is exact; floating-point arithmetic is not.

So what is softmax, in plain terms? Think of it as a vote-counter that has to turn raw enthusiasm into percentages. Each candidate (each class, each next-token) shouts a number — the logit — and those numbers can be anything: 2.0, -17, 1000. Softmax's job is to convert that noisy shouting into a clean probability distribution: everything positive, everything summing to exactly 1, and the loudest shout getting the biggest slice. It does this by exponentiating (which makes big numbers enormously bigger, exaggerating the winner) and then dividing by the total (which rescales the whole thing back to a tidy 1.0).

The trap — and the entire reason this is an interview question rather than a one-liner — is that exp grows terrifyingly fast. exp(1000) is a number with over 400 digits; your computer can't hold it, so it becomes inf, and inf/inf is nan. The fix is a small piece of algebra that feels like cheating but isn't: subtract the largest logit from every logit before exponentiating. Because that same constant appears in every term of the numerator and the denominator, it cancels perfectly in the ratio — the answer is mathematically identical — but now the biggest exponent is exp(0) = 1 and nothing overflows. The common mistake is to skip this line because "it works on my test." It does work, right up until a real model hands you a confident logit and the whole thing silently returns garbage. Its partner, cross-entropy, is just "how surprised were we by the right answer" — the negative log of the probability softmax assigned to the true class — which is why the two always travel together.

Asked as: "code softmax — now make it numerically stable." The stability trick (subtract the max) is the whole point; forgetting it overflows on large logits.

pythonimport numpy as np

def softmax(logits):
    z = logits - np.max(logits)          # STABILITY: avoid exp overflow
    e = np.exp(z)
    return e / e.sum()

def cross_entropy(logits, label):
    p = softmax(logits)
    return -np.log(p[label] + 1e-12)     # negative log-prob of the true class

print(softmax(np.array([2.0, 1.0, 0.1])).round(2))   # [0.66 0.24 0.1 ]

2 · Implement scaled dot-product attention (A4) the signature question advanced

If you learn to write exactly one thing from scratch for a GenAI interview, make it this. "Implement attention" is to LLM roles what "reverse a linked list" is to general software roles — the question that comes up so reliably that not being able to write it is a red flag on its own. The good news: the mechanism that powers every transformer, the thing behind ChatGPT and Claude and every model you've used, fits in about six lines. The intimidating name hides an idea you already understand.

Here's the plain-language version. Imagine every word in a sentence is a person at a networking event, and each one is holding two cards: a question card (the query: "I'm looking for someone who knows about X") and a badge (the key: "here's what I'm about"). To figure out who a given person should listen to, you compare their question card against everyone else's badge and score the matches — a strong match means "pay a lot of attention to that person." Those scores get turned into percentages (that's softmax again, from §1), and finally each person walks away with a blend of what the people they cared about were actually saying (the values). That blend is the output. Attention is, at its core, nothing more exotic than a weighted average — the cleverness is entirely in where the weights come from.

Two details earn the check-mark, and both have intuitive reasons. First, the ÷√d scaling: when your vectors are long (high dimension d), the dot-product scores naturally get bigger just because you're summing more terms — and huge scores push softmax into a corner where it hands almost all the weight to one word and the gradients flatten to nothing during training. Dividing by √d is the volume knob that keeps scores in a sane range no matter the dimension. The common mistake here is dropping that scale entirely because the code "runs fine" on a toy example — it does, until you train at real dimension and the model won't learn. Second, the causal mask: in a decoder, a word must not peek at words that come after it (they haven't been generated yet), so you set those future positions to a huge negative number before softmax, which drives their weight to essentially zero. Get the weighted-average intuition, the √d reason, and the mask, and you've answered the signature question completely.

Asked as: "implement attention" — the single most common GenAI-role question. It's three matmuls, a scale, a softmax, and a weighted sum.

pythonimport numpy as np

def softmax_rows(x):
    x = x - x.max(axis=-1, keepdims=True)
    e = np.exp(x)
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V, mask=None):
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)         # (seq, seq) affinities
    if mask is not None:
        scores = np.where(mask, scores, -1e9)   # causal mask → future = -inf
    weights = softmax_rows(scores)        # each row sums to 1
    return weights @ V                    # weighted sum of value vectors

seq, d = 3, 4
Q = K = V = np.random.rand(seq, d)
print(attention(Q, K, V).shape)           # (3, 4)

3 · k-means & cosine kNN (A4, A7) common

This is the round where "I use a vector database" isn't an answer. Anyone can call a library that finds similar embeddings or clusters them; the interviewer is checking whether you know what that library does when you strip the magic away. Both of these — finding the nearest vectors, and grouping vectors into clusters — turn out to be short loops over arithmetic you already know. If you can write them by hand, you understand your vector DB; if you can't, you're trusting a black box.

Start with cosine kNN, because it's the beating heart of every RAG retrieval you'll ever build. The question "which stored documents are most like this query?" becomes "which stored vectors point in most nearly the same direction as the query vector?" The word direction is the key: two embeddings are similar when they aim the same way, regardless of how long they are. Cosine similarity measures exactly that angle — the analogy is two arrows drawn from the same origin; you care whether they're roughly parallel, not whether one is drawn longer. The tidy trick is that once you rescale every vector to length 1 ("normalize"), the messy cosine formula collapses into a plain dot product, and finding the top-k is just "score everything, sort, take the best few."

k-means answers a different question — "what natural groups exist in this pile of vectors?" — and it's one of the most satisfying algorithms to hold in your head because it's just two moves repeated until they settle. Picture scattering a handful of pins on a map (those are your centroids, the guessed centres of clusters). Move one: assign every town to its nearest pin. Move two: slide each pin to the exact middle of the towns that chose it. Repeat, and the pins drift until they stop moving — that's convergence, and now each pin marks a real cluster. The classic mistake is forgetting that k-means is not deterministic: where you drop the pins initially changes where they end up, which is why smarter initialization (k-means++) and running it several times exist. Interviewers love this one precisely because a correct answer is short but reveals whether you actually understand assignment-then-update, or just memorized a name.

Asked as: "cluster these embeddings" / "find the k nearest vectors." Tests whether you can turn the vector math from A4/A7 into a loop.

pythonimport numpy as np

def knn(corpus, query, k):
    corpus = corpus / np.linalg.norm(corpus, axis=1, keepdims=True)
    query = query / np.linalg.norm(query)
    sims = corpus @ query                 # cosine = dot on normalized vectors
    return np.argsort(-sims)[:k]          # top-k indices

def kmeans_step(X, centroids):
    # assign each point to nearest centroid, then recompute centroids
    d = ((X[:, None, :] - centroids[None, :, :]) ** 2).sum(-1)
    labels = d.argmin(axis=1)
    new = np.array([X[labels == c].mean(axis=0) for c in range(len(centroids))])
    return labels, new                   # repeat until centroids stop moving

4 · Tokenization & sampling questions (A5) common

Every bill you've ever paid to an LLM was measured in tokens, not words — and this round checks whether you know the difference. It's easy to sail through months of prompting without ever confronting what a "token" actually is, right up until you're asked to explain why your 500-word prompt cost you 700 tokens, or why a model that "supports 8,000 tokens" choked on your document. Tokenization and sampling are the two ends of the pipe — how text goes into a model and how words come out — and both hide behaviour that surprises people who only ever worked at the text level.

On the input side: a token is not a word, it's a chunk the model learned to treat as one unit — often a whole common word, but sometimes a fragment, a space-plus-word, or a single punctuation mark. The analogy that sticks is Lego: "unbelievable" might be three studs (un, believ, able) while "cat" is one, and a rare name or a URL can shatter into a surprising number of pieces. This is why token count drifts away from word count, why code and non-English text cost more, and why "fit the prompt in the window" means counting tokens, not characters — the common mistake is estimating budget by word count and blowing the limit on the input the model rejects.

On the output side is sampling, the answer to a question beginners rarely think to ask: once the model has produced probabilities for the next token, how does it actually choose? Always taking the single most likely token (greedy) sounds safe but produces flat, repetitive text that loops on itself. So we introduce controlled randomness. Temperature is the creativity dial — it reshapes the probabilities before choosing: low temperature sharpens toward the favourite (safe, deterministic), high temperature flattens them out (surprising, sometimes unhinged). Top-p (nucleus) sampling is the safety rail on that randomness: instead of ever considering a truly terrible word, it keeps only the smallest shortlist of top candidates whose probabilities add up to p (say 90%), throws away the unreliable long tail, and samples from what's left. Together they're the two knobs behind every "make it more creative" or "make it more focused" setting you've touched — and being able to implement top-p, not just name it, is what earns the check-mark.

Asked as: "why is token count ≠ word count?", "implement top-p (nucleus) sampling", "how would you truncate a prompt to fit the context window?"

pythonimport numpy as np

def top_p_sample(probs, p=0.9):
    idx = np.argsort(-probs)                 # high→low
    sorted_p = probs[idx]
    cum = np.cumsum(sorted_p)
    cutoff = np.searchsorted(cum, p) + 1    # smallest set with mass ≥ p
    keep = idx[:cutoff]
    renorm = probs[keep] / probs[keep].sum()
    return int(np.random.choice(keep, p=renorm))   # sample from the nucleus
# temperature: divide logits by T before softmax — higher T = flatter = more random

5 · System design: "Design a RAG system" (A7) the design round expert

This is the round where fluent talkers fail and clear thinkers pass. "Design a question-answering system over ten million documents" invites you to name-drop — vector databases, embeddings, rerankers — and a surprising number of candidates do exactly that: a confident tour of buzzwords with no spine holding them together. The interviewer isn't grading vocabulary. They're listening for whether you can draw a system that would actually work, name the trade-off at each box, and — the part almost everyone forgets — say how it breaks. No production system is judged by its happy path.

The one insight that organizes the whole answer is that RAG has two separate lives, and confusing them is the tell of someone who's never built one. There's an offline path you run ahead of time, when documents change: break each document into bite-sized chunks, turn each chunk into an embedding vector, and load those vectors into a searchable index. This is the slow, expensive, batch work — and it's slow precisely so the other path can be fast. Then there's the online path you run for every single question: embed the query, ask the index for the handful of closest chunks, reorder them best-first (rerank), staple them into the prompt as context, and let the model answer from those documents instead of from its fuzzy memory. The analogy is an open-book exam: offline you write and tab your notes; online you flip to the right page and answer from it. Say "offline vs online" out loud and you've already out-structured half the room.

Now the part that separates senior from mid-level: the two failure modes, because they are different bugs with different fixes. A retrieval miss means the answer-bearing chunk never got fetched — the model was set up to fail, and no amount of prompt-tweaking helps; you fix it upstream in chunking, embeddings, or top-k (it shows up as low context recall). A hallucination is the opposite: the right chunk was fetched and the model ignored it, inventing an answer anyway — you fix that downstream with reranking, a tighter grounding prompt, and forced citations (it shows up as low faithfulness). The common mistake is treating "the answer was wrong" as one problem; a strong candidate immediately asks "wrong because we didn't retrieve it, or wrong because we didn't use what we retrieved?" — because those are two entirely different repairs, and knowing which one you're looking at is the whole job.

Asked as: "design a question-answering system over 10M company documents." The interviewer wants the pipeline, the scaling choices, and the failure modes — not code.

6 · System design: "Design an LLM agent / tool-use system" (A6, Ch 4) the design round expert

The word "safely" in this question is not decoration — it's the entire exam. "Design an assistant that can call tools and take actions" sounds like a fun architecture puzzle until you remember what "take actions" means: a probabilistic text model, which sometimes confidently makes things up, is now allowed to delete files, send emails, or move money. A candidate who designs a slick agent and forgets that it's an unpredictable model wired to real buttons has failed the round no matter how elegant the rest is. This is the "would I trust your system near production?" question.

The first thing to get right is that an agent is not a single clever prompt — it's a loop. The model doesn't solve the task in one shot; it takes a step, sees what happened, and decides the next step, over and over, the way a person debugging works: try something, read the error, try the next thing. The plain shape is propose → act → observe → repeat. The model proposes an action ("call the search tool with this query"), the action runs, the result (the "observation") is fed back into the model's context, and it proposes again — until it decides the task is done. The mistake that sinks people here is forgetting the loop can run forever; a model that keeps not-quite-finishing will spin until it drains your budget, which is why a hard MAX_STEPS cap is non-negotiable, not an optimization.

Then comes the safety spine, and the mental model to reach for is a bouncer between the model and the real world. The model never gets to act directly — it only ever proposes. Every proposal passes two checkpoints. First, validate: does this tool call even have the right shape and arguments? A malformed request dies here before it can do anything (this is where the structured-output work from A6 pays off). Second, the gate: a policy that classifies the action's risk and decides allow, ask a human, or block — reading a file is fine, deleting a production database asks for a human, exfiltrating secrets is blocked outright. The senior signal is insisting these guardrails live in code and IAM permissions, not in the prompt — because a prompt is a polite request a model can ignore, while a permission boundary is a wall it cannot cross. Round it out with reliability (retries with backoff, idempotency keys so a retried action doesn't fire twice) and observability (trace every step so you can see what it did), and you've described a system, not a demo.

Asked as: "design an assistant that can call tools / take actions safely." They want the loop, the safety gate, and how you keep it reliable.

7 · "Make 1000 LLM calls efficiently" (A3) common

The naive answer to this question — "loop over them and await each one" — is technically correct and will get you a polite pass to the next candidate. Making a thousand API calls one after another works; it's also brutally slow, because each call spends almost all its time waiting on the network while your CPU sits idle. This round tests whether you understand that waiting is not working, and whether you can speed things up without swinging to the opposite failure: firing all thousand at once and getting every one rejected.

The core idea is concurrency for I/O-bound work, and the analogy is a coffee shop. A one-at-a-time loop is a single barista who takes an order, stands and watches the espresso machine until it's done, hands it over, and only then takes the next order — most of the day spent staring at a machine. Concurrency is that barista starting the next order while the first is brewing. Nothing runs faster individually; you just stop wasting the waiting. Because LLM calls are almost entirely waiting (the network round-trip dwarfs your local work), overlapping them is a massive win — this is exactly the async/await machinery from A3, and it's why "is this task I/O-bound or CPU-bound?" is the first thing to say out loud.

But unlimited overlap is its own disaster, which is where the real judgment shows. Every API enforces a rate limit, and a naive "launch all 1000 concurrently" trips it instantly — the provider starts rejecting you with HTTP 429s and you've made things worse than the slow loop. So you need a governor. A semaphore is a bag of a fixed number of permits (say 10): a task must grab a permit before it runs and returns it when done, so at most 10 are ever in flight while the other 990 politely queue. That's the "how many at once" control. The mistake that costs the check-mark is stopping there — a semaphore alone isn't enough, because you'll still occasionally get a transient 429, so you pair it with retry and exponential backoff: on a rejection, wait a moment and try again, waiting longer each time. Concurrency capping and rate-limit recovery are two different controls, and the senior answer names both plus streaming results as they finish so you're not holding a thousand responses in memory at once.

Asked as: "you need to embed a million documents / evaluate 5000 prompts — how, without hitting rate limits?" Tests concurrency + backpressure judgment.

pythonimport asyncio

async def bounded_map(coro_fn, items, limit=10):
    sem = asyncio.Semaphore(limit)         # cap concurrency → respect rate limits
    async def worker(x):
        async with sem:
            return await coro_fn(x)
    return await asyncio.gather(*[worker(x) for x in items])
# pair with token-bucket rate limiting + retry/backoff on 429s (A6)

8 · "Productionize an LLM feature" (A8) the wrap-up

This is the round where the interviewer stops asking "can you build it?" and starts asking "would I let you own it?" Anyone can demo a chatbot that works on Tuesday. The senior question is what happens on the next Tuesday — when the model provider ships a new version, when traffic triples, when a user finds the prompt injection, when the monthly bill arrives. "Productionize an LLM feature" is deliberately open-ended because they want to see whether you think in features to add or in a system that stays healthy over time. The gap between those two mindsets is the gap between mid-level and senior.

The trap is to answer with a shopping list — "add caching, add monitoring, add guardrails" — which sounds competent and misses the point. Those are ingredients; the interviewer wants the recipe. The organizing idea is a closed loop, and the analogy is the difference between building a car and running a racing team: shipping the feature is building the car, but keeping it winning means measuring every lap, catching problems the moment they appear, and feeding what you learn back into the next build. Concretely: ship behind an evaluation gate (a golden set of cases that must still pass, checked in CI, so a "small prompt tweak" can't silently regress quality) → monitor quality, cost, latency, and drift in production → when something fails, capture it as a new golden case → iterate. That loop is what turns a demo into a system that survives contact with reality.

Then layer the specifics onto that spine, and know why each one matters rather than just listing it. Cost doesn't take care of itself: prompt caching for repeated prefixes, routing easy requests to a cheaper model, and token budgets are the levers, and they compound. Latency is a product feature, not a nicety — streaming tokens so the user sees words immediately, async so slow steps overlap, a cache for repeated questions. Safety lives in code and IAM, never in the prompt (the same senior tell as the agent round) plus hard-fail safety evals that block a deploy outright. And versioning is the one people forget until it burns them: pin the exact model and prompt version, and roll out changes as shadow or canary deploys so a regression hits 1% of traffic, not 100%. The common mistake is treating the model as a fixed dependency; it isn't — it changes underneath you, so the system has to be built to notice and absorb that.

Asked as: "we shipped a chatbot — how do you make it reliable, cheap, and safe over time?" The senior-signal round.

AI-interview question → what they're testing expert

Checkpoint expert

import math

def softmax(logits):
    m = max(logits)                       # stability: shift so the max is 0
    exps = [math.exp(x - m) for x in logits]
    s = sum(exps)
    return [e / s for e in exps]

out = softmax([2.0, 1.0, 0.1])
print([round(p, 3) for p in out])         # [0.659, 0.242, 0.099]
print(round(sum(out), 6))                 # 1.0

import math

def softmax(logits):
    m = max(logits); exps = [math.exp(x-m) for x in logits]; s = sum(exps)
    return [e/s for e in exps]

def cross_entropy(logits, target):
    p = softmax(logits)[target]
    return -math.log(p)

print(round(cross_entropy([5.0, 0.0, 0.0], 0), 4))   # 0.0135  confident & correct -> low loss
print(round(cross_entropy([5.0, 0.0, 0.0], 1), 4))   # 5.0136  confident & WRONG   -> high loss

import math

def softmax(xs):
    m = max(xs); e = [math.exp(x-m) for x in xs]; s = sum(e); return [v/s for v in e]

def attention(q, keys, values):
    d = len(q)
    scores = [sum(qi*ki for qi, ki in zip(q, k)) / math.sqrt(d) for k in keys]  # scale by 1/sqrt(d)
    weights = softmax(scores)
    dim = len(values[0])
    return [sum(w*v[j] for w, v in zip(weights, values)) for j in range(dim)], weights

q      = [1.0, 0.0]
keys   = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
values = [[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]]
out, w = attention(q, keys, values)
print([round(x, 2) for x in out])         # attends most to key[0] and key[2]
print([round(x, 3) for x in w])           # attention weights sum to 1

import math

def softmax(logits, temp=1.0):
    z = [x/temp for x in logits]          # temp<1 sharpens, temp>1 flattens
    m = max(z); e = [math.exp(x-m) for x in z]; s = sum(e); return [v/s for v in e]

def top_p(probs, p=0.9):
    ranked = sorted(enumerate(probs), key=lambda t: t[1], reverse=True)
    kept, cum = [], 0.0
    for idx, pr in ranked:
        kept.append((idx, pr)); cum += pr
        if cum >= p: break                # smallest set whose mass >= p (the "nucleus")
    z = sum(pr for _, pr in kept)
    return {idx: pr/z for idx, pr in kept}  # renormalize over the nucleus

logits = [3.0, 2.0, 1.0, 0.1]
print([round(x,3) for x in softmax(logits, temp=0.5)])  # sharper
print(top_p(softmax(logits), p=0.9))                    # drops the low-mass tail

Ingestion:  chunk docs (with a situating context prefix per chunk) -> embed ->
            store vectors in a vector DB (HNSW/IVF for ANN at scale).
Retrieval:  embed query -> ANN top-k -> hybrid (dense + BM25) + RRF fusion ->
            cross-encoder rerank -> precise top-k into the prompt.
Generation: prompt = system + retrieved context + question; cite sources.
Eval:       context_recall / context_precision / faithfulness / answer_relevance
            on a curated eval set; gate changes in CI.

Failure mode 1: RETRIEVAL miss -> answer-bearing chunk never fetched
                (low context_recall). Fix chunking/embeddings/top-k, not the prompt.
Failure mode 2: HALLUCINATION -> answer not grounded in retrieved context
                (low faithfulness). Fix grounding: rerank, tighter prompt, cite.

def plan_1000_calls(latency_sensitive, shared_prefix, in_tok, out_tok):
    steps, mult = [], 1.0
    if not latency_sensitive:
        steps.append("Batches API — 50%% off, async"); mult *= 0.5
    else:
        steps.append("bounded concurrency (async) — overlap the 1000 calls")
    if shared_prefix:
        steps.append("prompt-cache the shared prefix — repeated tokens ~0.1x")
    steps.append("cheapest model that clears the quality bar (Haiku for simple)")
    # rough per-call cost on Haiku ($1/$5 per MTok) before caching/batch:
    base = in_tok/1e6*1.0 + out_tok/1e6*5.0
    return steps, round(base * 1000 * mult, 2)

steps, cost = plan_1000_calls(False, True, in_tok=800, out_tok=200)
for s in steps: print("-", s)
print(f"~est batch cost for 1000 calls: ${cost}")

Knowledge check check yourself