Big Tech AI-Engineering Patterns
The other interview loop — for AI/ML engineer, applied scientist, and LLM/GenAI engineer roles at Big Tech. Beyond rote pattern-drilling (that's D7), these rounds test whether you can implement the ML primitives from scratch, reason about production LLM systems, and design RAG/agent architectures. This part maps each A1–A8 topic to the questions actually asked, with runnable code and diagrams.
Learning objectives
- Implement the ML-from-scratch primitives interviewers ask for (softmax, attention, k-means, cosine kNN).
- Answer the LLM-systems questions: tokenization, sampling, context management, caching, rate limits.
- Design a RAG pipeline and an agent loop on a whiteboard, with the right trade-offs.
- Talk fluently about the "productionizing an LLM feature" system-design round.
1 · Implement softmax & cross-entropy (A4) very common
Asked as: "code softmax — now make it numerically stable." The stability trick (subtract the max) is the whole point; forgetting it overflows on large logits.
exp shifts values into a safe range without changing the result (it cancels in the ratio) — the detail interviewers look for.
This picture shows what softmax does: it turns a list of raw scores (called logits) into a list of probabilities that add up to 1. Read it left to right — each arrow is one step of the calculation.
- Left box
[2.0, 1.0, 0.1]— the raw scores the model produced. They can be any size, positive or negative; they are not probabilities yet. - First arrow (subtract max, then
e^(x−max)) — we shift every number down by the largest one, then raise e (≈2.718) to each. Subtracting the max is the stability trick: it keeps the numbers small enough that the computer won't overflow, and it does not change the final answer. - Second arrow (normalize) — divide each result by the total so the whole list sums to exactly 1.
- Right box
[.66, .24, .10]— the probabilities. Notice the biggest input (2.0) became the biggest probability (.66): softmax keeps the ranking, it just rescales into a 0–1 range.
In short: softmax = exponentiate, then divide by the total. The order of the numbers never changes — the largest logit always gets the largest probability.
pythonimport numpy as np
def softmax(logits):
z = logits - np.max(logits) # STABILITY: avoid exp overflow
e = np.exp(z)
return e / e.sum()
def cross_entropy(logits, label):
p = softmax(logits)
return -np.log(p[label] + 1e-12) # negative log-prob of the true class
print(softmax(np.array([2.0, 1.0, 0.1])).round(2)) # [0.66 0.24 0.1 ]
This is the diagram above written as real code, plus its partner cross-entropy — the number that measures how wrong a prediction was. Interviewers ask for softmax and expect the stability trick; forgetting it is the classic mistake.
z = logits - np.max(logits)subtracts the largest value from every logit. This is the stability step — it preventsnp.expfrom blowing up to infinity on big numbers. Because it cancels out in the division, the final probabilities are identical.e = np.exp(z)raises e to each shifted value, thene / e.sum()divides by the total so the results add to 1 — the two "exponentiate then normalize" steps from the picture.cross_entropycallssoftmax, then returns the negative log of the probability the model gave to the correct answer (p[label]). A confident-and-right model gets a probability near 1, whose log is near 0 → tiny loss. A confident-and-wrong model gets a tiny probability → huge loss.- The tiny
1e-12added inside the log is a safety guard:log(0)is undefined, so this keeps it from crashing if a probability rounds to exactly zero.
What the output means: The last line prints [0.66 0.24 0.1 ] — the three probabilities, which sum to 1. That matches the green box in the diagram.
Try this: Change the input to [100.0, 1.0, 0.1] and run it — thanks to the - np.max line it still works. Delete that line and the same input overflows to nan (not-a-number). That one line is the whole point of the question.
2 · Implement scaled dot-product attention (A4) the signature question advanced
Asked as: "implement attention" — the single most common GenAI-role question. It's three matmuls, a scale, a softmax, and a weighted sum.
QKᵀ scores every query against every key; dividing by √d keeps gradients stable; softmax turns scores into weights; multiplying by V mixes the values. This is the transformer's core.
This is the heart of every transformer. Attention lets each word look at every other word and decide how much to pay attention to it. The formula on top (softmax(QKᵀ/√d)·V) is exactly what the boxes below spell out, step by step.
- Q and Kᵀ boxes — Q (queries) is "what each word is looking for"; K (keys) is "what each word offers." The little ᵀ just means K is flipped so the shapes line up for multiplication.
- scores /√d box — multiplying Q by Kᵀ gives a grid of scores: how well each word matches each other word. Dividing by
√d(the square root of the vector size) keeps those scores from getting so large that the next step breaks. - softmax box — turns each row of scores into weights that add up to 1 (same softmax as section 1). Now every word has a set of "how much I care about each other word" percentages.
- · V box — V (values) is the actual content of each word. Multiplying the weights by V produces a weighted average: each word's output is a blend of the words it cared about most.
In short: Attention is just a weighted average of values, where the weights come from how well queries match keys. Q·Kᵀ scores the matches, √d keeps them stable, softmax turns them into percentages, and ·V mixes the content.
pythonimport numpy as np
def softmax_rows(x):
x = x - x.max(axis=-1, keepdims=True)
e = np.exp(x)
return e / e.sum(axis=-1, keepdims=True)
def attention(Q, K, V, mask=None):
d = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d) # (seq, seq) affinities
if mask is not None:
scores = np.where(mask, scores, -1e9) # causal mask → future = -inf
weights = softmax_rows(scores) # each row sums to 1
return weights @ V # weighted sum of value vectors
seq, d = 3, 4
Q = K = V = np.random.rand(seq, d)
print(attention(Q, K, V).shape) # (3, 4)
Here is the attention diagram as runnable code — "the signature question" for GenAI roles. It is only a handful of lines: a matrix multiply, a scale, a softmax, and another matrix multiply.
softmax_rowsis the section-1 softmax applied to each row separately (axis=-1means "along the last dimension"). Every row of scores becomes its own set of weights that sum to 1.- Inside
attention,Q @ K.T / np.sqrt(d)is the picture's first two boxes at once:@is matrix multiply (Q times K-flipped), and dividing bynp.sqrt(d)is the √d scaling that keeps scores stable. - The optional mask line replaces "not allowed" positions with a huge negative number so softmax drives their weight to ~0. This is how a decoder stops a word from peeking at future words it hasn't generated yet.
weights = softmax_rows(scores)turns scores into percentages, andweights @ Vis the final· Vbox — the weighted average of the value vectors.
What the output means: It prints (3, 4) — the output has the same shape as the input (3 words, each a 4-number vector). Attention transforms the vectors; it doesn't change how many there are.
Try this: The Q, K, V here are random, so the numbers are meaningless — the point is the shape. Change seq, d = 3, 4 to 5, 8 and the printed shape becomes (5, 8). In a real transformer, Q/K/V come from the words, not np.random.rand.
3 · k-means & cosine kNN (A4, A7) common
Asked as: "cluster these embeddings" / "find the k nearest vectors." Tests whether you can turn the vector math from A4/A7 into a loop.
pythonimport numpy as np
def knn(corpus, query, k):
corpus = corpus / np.linalg.norm(corpus, axis=1, keepdims=True)
query = query / np.linalg.norm(query)
sims = corpus @ query # cosine = dot on normalized vectors
return np.argsort(-sims)[:k] # top-k indices
def kmeans_step(X, centroids):
# assign each point to nearest centroid, then recompute centroids
d = ((X[:, None, :] - centroids[None, :, :]) ** 2).sum(-1)
labels = d.argmin(axis=1)
new = np.array([X[labels == c].mean(axis=0) for c in range(len(centroids))])
return labels, new # repeat until centroids stop moving
Two classic "do something with these embeddings" questions in one block. kNN finds the vectors most similar to a query; k-means groups vectors into clusters. Both are just loops over vector math.
- In
knn, dividing each vector bynp.linalg.norm(...)rescales it to length 1 ("normalizing"). Once vectors are length 1, a plain dot product is the cosine similarity — a score from -1 (opposite) to 1 (identical). corpus @ querycomputes that similarity for every stored vector at once.np.argsort(-sims)[:k]sorts from highest to lowest (the minus flips the order) and keeps the topk— the k nearest neighbours.- In
kmeans_step, the big bracketed expression measures the distance from every point to every centroid (cluster centre).d.argmin(axis=1)then labels each point with its closest centroid. - The last line recomputes each centroid as the average of the points assigned to it. Repeating this assign-then-recompute loop is the whole k-means algorithm — the centres shuffle around until they stop moving.
Try this: k-means is two steps on repeat: (1) assign each point to the nearest centre, (2) move each centre to the middle of its points. Picture dropping 3 pins on a map, then sliding each pin to the centre of its nearest towns, over and over.
4 · Tokenization & sampling questions (A5) common
Asked as: "why is token count ≠ word count?", "implement top-p (nucleus) sampling", "how would you truncate a prompt to fit the context window?"
pythonimport numpy as np
def top_p_sample(probs, p=0.9):
idx = np.argsort(-probs) # high→low
sorted_p = probs[idx]
cum = np.cumsum(sorted_p)
cutoff = np.searchsorted(cum, p) + 1 # smallest set with mass ≥ p
keep = idx[:cutoff]
renorm = probs[keep] / probs[keep].sum()
return int(np.random.choice(keep, p=renorm)) # sample from the nucleus
# temperature: divide logits by T before softmax — higher T = flatter = more random
When a model picks the next word, it doesn't always take the single most likely one — that would be repetitive. Top-p (nucleus) sampling keeps just enough of the most likely words to cover a probability mass of p (say 90%), then picks randomly from that shortlist.
np.argsort(-probs)orders the words from most to least likely (the minus sorts high→low).sorted_pis those probabilities in that order.np.cumsum(sorted_p)makes a running total: 0.5, then 0.75, then 0.9…np.searchsorted(cum, p)finds where that running total first reachesp, socutoffis the size of the shortlist (the "nucleus").keep = idx[:cutoff]is that shortlist of word indices. Dividing their probabilities by their own sum (renorm) rescales them back to add up to 1, since we threw the rest away.np.random.choice(keep, p=renorm)then picks one word at random, weighted by those probabilities. The trailing comment notes temperature: dividing logits by a number T before softmax flattens the probabilities — higher T = more surprising word choices.
What the output means: The function returns a single integer — the index of the chosen next word. Run it many times and you'll get different words, but always ones from the high-probability shortlist.
Try this: Set p=0.1 and the shortlist shrinks to basically the single top word (nearly greedy, very repetitive). Set p=1.0 and every word is eligible (more creative, more random). That dial is exactly what "top-p" controls in a real API.
5 · System design: "Design a RAG system" (A7) the design round expert
Asked as: "design a question-answering system over 10M company documents." The interviewer wants the pipeline, the scaling choices, and the failure modes — not code.
This is the whiteboard answer to "design a question-answering system over millions of documents." The trick is that RAG (retrieval-augmented generation) has two separate paths — one you run ahead of time, one you run per question. Read the top row first, then the bottom row.
- Top row (offline, run once when documents change):
docs → chunk → embed → vector DB. You break documents into small pieces (chunk), turn each piece into a vector of numbers (embed), and store those vectors in a searchable vector database. This is slow, so you do it in advance. - Bottom row (online, run for every question):
query → retrieve k → rerank → LLM + context → answer. The user's question is embedded, the DB returns thekclosest chunks (retrieve), a rerank step reorders them best-first, and those chunks are handed to the LLM as context so it answers from real documents instead of guessing. - The dashed arrow from the vector DB down to
retrieve kshows the link between the two paths: the online search reads the index the offline path built. - The
(A4),(A5),(A7)tags point at earlier lessons where each box is built from scratch — chunking (A5), embedding (A4), the vector index (A7).
In short: RAG = look it up, then answer. Offline you build a searchable index of your documents; online you fetch the few most relevant pieces and let the model write an answer grounded in them. In the interview, name the trade-off at every box.
6 · System design: "Design an LLM agent / tool-use system" (A6, Ch 4) the design round expert
Asked as: "design an assistant that can call tools / take actions safely." They want the loop, the safety gate, and how you keep it reliable.
This shows how an AI agent that can take actions (call tools) stays safe. An agent isn't one call — it's a loop that repeats until the task is done. Follow the arrows left to right, then notice the arrow that curves back.
- LLM box — the model proposes an action, e.g. "call the delete-file tool." It only suggests; it doesn't get to run anything directly.
- validate box — check the proposed action has the right shape and arguments (the
A6tag points to the validation lesson). A malformed request is rejected here before anything happens. - gate box — a safety policy decides allow, ask a human, or block. Risky or irreversible actions get stopped or require approval; safe ones pass through.
- tool exec box — the approved action actually runs, and its result (the "observation") is fed back to the LLM via the curving arrow. The caption's
MAX_STEPScap is what stops the loop from running forever.
In short: An agent is a loop with guardrails: propose → validate → gate → execute → feed the result back → repeat. The gate and the step cap are what make it safe to let a model take real actions.
7 · "Make 1000 LLM calls efficiently" (A3) common
Asked as: "you need to embed a million documents / evaluate 5000 prompts — how, without hitting rate limits?" Tests concurrency + backpressure judgment.
pythonimport asyncio
async def bounded_map(coro_fn, items, limit=10):
sem = asyncio.Semaphore(limit) # cap concurrency → respect rate limits
async def worker(x):
async with sem:
return await coro_fn(x)
return await asyncio.gather(*[worker(x) for x in items])
# pair with token-bucket rate limiting + retry/backoff on 429s (A6)
The interview question is "make 1000 LLM calls efficiently without hitting rate limits." You want many calls happening at once (fast), but not too many at once (or the API rejects you). This code runs them in parallel with a fixed ceiling.
asyncio.Semaphore(limit)is a permit counter — it hands out at mostlimitpermits (here 10) at a time. It's the "only 10 at once" rule that keeps you under the rate limit.- Each
workerdoesasync with sem:— it waits until a permit is free, takes one, runs its call withawait coro_fn(x), then releases the permit so a waiting worker can go.awaitmeans "pause here without blocking everyone else." asyncio.gather(*[worker(x) for x in items])launches a worker for every item and waits for them all to finish. The semaphore quietly ensures only 10 are actually in flight at any moment; the rest queue up.- The final comment is the senior add-on: pair this with a rate limiter and retry/backoff (wait-and-retry on HTTP 429 "too many requests") for a production-grade fan-out.
What the output means: It returns a list of all the results, in the same order as items — but the calls ran concurrently (up to 10 at a time), so 1000 calls finish far faster than doing them one by one.
Try this: Change limit=10 to limit=1 and the calls run strictly one after another (slow but gentle). Raise it to 100 and they flood out at once — which is exactly what trips a rate limit. The semaphore is the knob that balances speed against the API's limits.
8 · "Productionize an LLM feature" (A8) the wrap-up
Asked as: "we shipped a chatbot — how do you make it reliable, cheap, and safe over time?" The senior-signal round.
| Concern | What to say | Ref |
|---|---|---|
| Quality doesn't regress | golden-set evals + LLM-judge, gate deploys in CI | Ch 5, A8 |
| Cost control | prompt caching, model routing, token budgets | A8 §8 |
| Latency | streaming, async, cache, smaller model for easy steps | A3 |
| Safety | guardrails in code/IAM, hard-fail safety evals, human gate | Ch 6, Ch 8 |
| Versioning & rollback | pin model + prompt versions, shadow/canary deploy | A8 §3 |
| Monitoring | latency/cost/quality/drift dashboards + alerts | A8 §7 |
AI-interview question → what they're testing expert
| Question | They're testing | Section |
|---|---|---|
| "Implement softmax (stably)" | numerical care | §1 |
| "Implement attention" | transformer fundamentals | §2 |
| "Cluster / kNN these embeddings" | vector math → code | §3 |
| "Implement top-p sampling" | decoding knowledge | §4 |
| "Design a RAG system" | retrieval architecture | §5 |
| "Design a tool-using agent" | loop + safety design | §6 |
| "Make 1000 LLM calls efficiently" | concurrency & limits | §7 |
| "Productionize an LLM feature" | MLOps maturity | §8 |
Checkpoint expert
- Implement stable softmax, scaled dot-product attention, and cosine kNN / k-means from scratch.
- Explain tokenization and code a sampling strategy (top-p / temperature).
- Whiteboard a RAG pipeline and an agent loop, naming trade-offs and failure modes.
- Answer the "productionize / make it reliable + cheap + safe" round with the eval→monitor→iterate loop.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Softmax is the classic ML-interview warm-up, and the point being tested is the numerical-stability trick: subtracting the max before exponentiating avoids exp overflow on large logits.
Your task: Implement a numerically stable softmax(logits) and confirm its output is a valid probability distribution.
Requirements:
- Subtract the max logit before exponentiating so all exponents are ≤ 0
- Return the same result mathematically as the naive version, without overflow
- The returned values are all non-negative and sum to
1.0 - Pure stdlib —
math.expis enough, no numpy - Print the distribution and its sum to verify it totals 1
💡 Hint: Compute m = max(logits), exponentiate x - m for each, then divide by the sum of those exponentials.
Show solution
Stable softmax subtracts the max before exponentiating (pure stdlib):
import math
def softmax(logits):
m = max(logits) # stability: shift so the max is 0
exps = [math.exp(x - m) for x in logits]
s = sum(exps)
return [e / s for e in exps]
out = softmax([2.0, 1.0, 0.1])
print([round(p, 3) for p in out]) # [0.659, 0.242, 0.099]
print(round(sum(out), 6)) # 1.0Without the - m shift, large logits overflow exp; subtracting the max is mathematically identical but keeps exponents ≤ 0. The interviewer is checking you know that stability trick and that softmax yields a probability distribution (sums to 1).
Context: Cross-entropy is the standard classification and language-model training loss because it punishes confident mistakes hardest. Pairing it with softmax is the natural follow-up to the softmax warm-up.
Your task: Implement cross_entropy(logits, target_index) as the negative log of the softmax probability of the true class, and show it's low when confident-correct.
Requirements:
- Turn logits into probabilities with a stable softmax first
- Return
-log(p)of the probability assigned to the target class - Loss is near 0 when the model is confidently correct
- Loss is large when the model is confidently wrong
- Demonstrate both the correct-target and wrong-target cases numerically
💡 Hint: Reuse your stable softmax, index out the target class's probability, and take -math.log of it.
Show solution
Cross-entropy penalizes low probability on the true class (runnable):
import math
def softmax(logits):
m = max(logits); exps = [math.exp(x-m) for x in logits]; s = sum(exps)
return [e/s for e in exps]
def cross_entropy(logits, target):
p = softmax(logits)[target]
return -math.log(p)
print(round(cross_entropy([5.0, 0.0, 0.0], 0), 4)) # 0.0135 confident & correct -> low loss
print(round(cross_entropy([5.0, 0.0, 0.0], 1), 4)) # 5.0136 confident & WRONG -> high lossLoss is -log of the probability assigned to the true class: near 0 when the model is confidently correct, large when it is confidently wrong. That asymmetry — punishing confident mistakes hardest — is why cross-entropy is the standard classification/LM training loss.
Context: Scaled dot-product attention is the transformer core, and the detail interviewers probe is the 1/√d scaling — omit it and scores grow with dimension, saturating softmax and killing gradients.
Your task: Implement single-query scaled dot-product attention: score the query against keys, scale, softmax, then take the weighted sum of the values.
Requirements:
- Score by dotting the query with each key
- Divide the scores by
√dbefore the softmax - Softmax the scaled scores into weights that sum to 1
- Return the weighted sum of the value vectors (and optionally the weights)
- Show the output attends most to the keys most aligned with the query
💡 Hint: Three steps — scaled scores, softmax weights, weighted sum — each a small comprehension; d is the query/key dimension.
Show solution
score → scale → softmax → weighted sum of values (pure stdlib):
import math
def softmax(xs):
m = max(xs); e = [math.exp(x-m) for x in xs]; s = sum(e); return [v/s for v in e]
def attention(q, keys, values):
d = len(q)
scores = [sum(qi*ki for qi, ki in zip(q, k)) / math.sqrt(d) for k in keys] # scale by 1/sqrt(d)
weights = softmax(scores)
dim = len(values[0])
return [sum(w*v[j] for w, v in zip(weights, values)) for j in range(dim)], weights
q = [1.0, 0.0]
keys = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
values = [[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]]
out, w = attention(q, keys, values)
print([round(x, 2) for x in out]) # attends most to key[0] and key[2]
print([round(x, 3) for x in w]) # attention weights sum to 1The query dots each key to get relevance scores; dividing by sqrt(d) keeps scores from growing with dimension (which would saturate softmax); softmax turns them into weights that mix the values. That scaling factor is the detail interviewers probe — omit it and gradients vanish at large d.
Context: Temperature and nucleus (top-p) sampling are the decoding knobs that control how an LLM picks tokens, and interviewers want you to explain what each does. They compose: temperature reshapes the distribution, then top-p truncates its tail.
Your task: Implement temperature scaling in softmax and a top_p nucleus filter on a probability distribution, and explain each knob.
Requirements:
- Temperature < 1 sharpens toward the argmax; > 1 flattens toward uniform
- Top-p keeps the smallest set of tokens whose cumulative mass reaches
p - The kept 'nucleus' probabilities are renormalized to sum to 1
- Low-mass tail tokens outside the nucleus are dropped
- Show a sharpened distribution and a top-p filtered one on the same logits
💡 Hint: Divide logits by the temperature before softmax; for top-p, sort by probability descending and accumulate until the running mass crosses p.
Show solution
Temperature reshapes the distribution; top-p truncates its tail (runnable):
import math
def softmax(logits, temp=1.0):
z = [x/temp for x in logits] # temp<1 sharpens, temp>1 flattens
m = max(z); e = [math.exp(x-m) for x in z]; s = sum(e); return [v/s for v in e]
def top_p(probs, p=0.9):
ranked = sorted(enumerate(probs), key=lambda t: t[1], reverse=True)
kept, cum = [], 0.0
for idx, pr in ranked:
kept.append((idx, pr)); cum += pr
if cum >= p: break # smallest set whose mass >= p (the "nucleus")
z = sum(pr for _, pr in kept)
return {idx: pr/z for idx, pr in kept} # renormalize over the nucleus
logits = [3.0, 2.0, 1.0, 0.1]
print([round(x,3) for x in softmax(logits, temp=0.5)]) # sharper
print(top_p(softmax(logits), p=0.9)) # drops the low-mass tailTemperature <1 sharpens toward the argmax (more deterministic), >1 flattens (more random). Top-p keeps only the smallest set of tokens whose probability mass reaches p and renormalizes, cutting the unreliable tail. They compose: temperature reshapes, then top-p truncates.
Context: 'Design a RAG system' is a staple system-design interview, and a defensible answer names the pipeline and the two ways it breaks — a retrieval miss versus a hallucination — because they need different metrics and different fixes.
Your task: Produce the component design (ingestion → retrieval → generation → eval) and name the two failure modes you'd be graded on.
Requirements:
- Ingestion: chunk (with situating context) → embed → store in a vector DB (ANN like HNSW/IVF at scale)
- Retrieval: embed query → ANN top-k → hybrid + RRF → cross-encoder rerank → precise top-k
- Generation: system prompt + retrieved context + question, with source citations
- Eval: context recall/precision, faithfulness, answer relevance on a curated set, gated in CI
- Name failure mode 1 (retrieval miss → low context recall) and its fix
- Name failure mode 2 (hallucination → low faithfulness) and its fix
💡 Hint: The grader is listening for the distinction: a missed chunk is a recall problem you fix in retrieval, a fetched-but-ignored chunk is a faithfulness problem you fix in grounding.
Show solution
A defensible RAG design names components and failure modes (worked design):
Ingestion: chunk docs (with a situating context prefix per chunk) -> embed ->
store vectors in a vector DB (HNSW/IVF for ANN at scale).
Retrieval: embed query -> ANN top-k -> hybrid (dense + BM25) + RRF fusion ->
cross-encoder rerank -> precise top-k into the prompt.
Generation: prompt = system + retrieved context + question; cite sources.
Eval: context_recall / context_precision / faithfulness / answer_relevance
on a curated eval set; gate changes in CI.
Failure mode 1: RETRIEVAL miss -> answer-bearing chunk never fetched
(low context_recall). Fix chunking/embeddings/top-k, not the prompt.
Failure mode 2: HALLUCINATION -> answer not grounded in retrieved context
(low faithfulness). Fix grounding: rerank, tighter prompt, cite.The grader wants the pipeline (ingest → retrieve → generate → eval) and the two ways it breaks: a retrieval miss (the chunk was never fetched — a recall problem) versus a hallucination (the chunk was fetched but the model didn't ground on it — a faithfulness problem). Different metrics, different fixes.
Context: The scale question — run 1000 independent LLM calls for the least cost and wall-clock — wants you to name the four levers and show they multiply, not just 'call it in a loop'.
Your task: Write a decision function for 1000 calls that combines batching, caching, concurrency, and model choice, and quantifies the resulting cost.
Requirements:
- Use the Batches API (≈50% off) when latency doesn't matter; concurrency when it does
- Prompt-cache a shared prefix so repeated tokens cost a fraction
- Pick the cheapest model that clears the quality bar
- Compute a rough per-call cost from input/output tokens and multiply out to 1000
- Show the levers stack (multiply) rather than being either/or
💡 Hint: Model each lever as a multiplier or an added step; start from a per-call token cost, then apply the batch/cache discounts to get the fleet total.
Show solution
Combine the four levers and quantify the win (runnable):
def plan_1000_calls(latency_sensitive, shared_prefix, in_tok, out_tok):
steps, mult = [], 1.0
if not latency_sensitive:
steps.append("Batches API — 50%% off, async"); mult *= 0.5
else:
steps.append("bounded concurrency (async) — overlap the 1000 calls")
if shared_prefix:
steps.append("prompt-cache the shared prefix — repeated tokens ~0.1x")
steps.append("cheapest model that clears the quality bar (Haiku for simple)")
# rough per-call cost on Haiku ($1/$5 per MTok) before caching/batch:
base = in_tok/1e6*1.0 + out_tok/1e6*5.0
return steps, round(base * 1000 * mult, 2)
steps, cost = plan_1000_calls(False, True, in_tok=800, out_tok=200)
for s in steps: print("-", s)
print(f"~est batch cost for 1000 calls: ${cost}")The four levers stack: batch for 50% off when latency doesn't matter, cache the shared prefix (~0.1x on repeated tokens), concurrency to overlap calls when it does matter, and pick the cheapest model that clears the bar. The grader wants you to name all four and show they multiply, not just "call it in a loop."
Knowledge check check yourself
For "implement scaled dot-product attention," the lesson gives softmax(QKᵀ/√d)·V. What is the role of the √d scaling, and what does the optional mask do?
Show answer
The "make 1000 LLM calls efficiently" answer pairs a semaphore with retry/backoff on 429s. Why does the lesson insist you name both to earn the check-mark?