Context Engineering: Memory, Windowing & Retrieval

1 · Prompt vs context engineering essential

Here's a moment you'll recognise if you've shipped anything with an LLM. You spend an afternoon polishing the wording of a prompt — "be concise", "think step by step", "you are an expert" — and the answers barely move. Then, almost by accident, you paste in the one paragraph of documentation the model actually needed, and suddenly it's right every time. You didn't write a better prompt. You gave it better context. That gap — between how you ask and what you supply — is the whole subject of this lesson, and it's the lever most people reach for last when it should be first.

The reason is simple and worth saying plainly: a language model has no memory and no access to anything outside the text you hand it on this call. It cannot look up your refund policy, recall yesterday's conversation, or peek at the database. Whatever isn't in the context window on this request does not exist as far as the model is concerned. So the quality ceiling of any answer is set by what you put in front of it — phrasing only decides how well it uses what's already there.

An analogy that sticks: imagine hiring a brilliant contractor who has total amnesia between meetings. Prompt engineering is coaching them on how to work — "explain your reasoning, double-check the numbers." Context engineering is handing them the right folder of documents before they start. A world-class contractor with the wrong folder gives you a confident, well-structured, completely wrong answer. That's the most common failure in production LLM systems, and it is almost never a prompting problem — it's a context problem wearing a prompting mask.

The common mistake beginners make is treating the context as an afterthought — dumping in "everything that might help" and trusting the model to sort it out. But the window is finite, every token competes for the model's attention, and irrelevant material actively drowns the signal (you'll meet this "lost in the middle" effect in §3). Context engineering is the discipline of deciding, deliberately, what earns a place in that window and how to arrange it — retrieval, memory, windowing, compaction, caching. It is a systems problem, not a wordsmithing one, and the rest of this lesson builds the toolkit piece by piece.

Prompt engineering is how you phrase the ask. Context engineering is what information you put in the window and how you structure it — often the bigger lever. The model can only use what's in its context; curating that is the craft.

2 · The window is a budget essential

Picture the failure this section prevents. Your assistant works beautifully in testing. In production, a customer has a long chat, your retrieval pulls back a dozen documents, and the API call comes back not with an answer but with an error: the request exceeds the model's context length. Or worse — it doesn't error, it just silently truncates, and the model answers based on half the conversation it was supposed to see. Both failures come from the same blind spot: treating the context window as infinite when it is, in fact, a fixed budget you are always spending.

Think of the window as a jar that holds exactly so many marbles — 8,000, 200,000, whatever your model allows. Every marble you drop in is committed and cannot be reused. The system prompt is marbles. The conversation history is marbles. Each tool definition is marbles. And crucially, the space the model needs to write its answer is marbles too — you have to leave the jar partly empty on purpose, or there's no room for the reply. Whatever's left after all of that is your budget for retrieved documents. It is not "as much RAG as you want"; it is "exactly this many tokens, and not one more."

The reason this deserves its own section — rather than being obvious — is that the sections compete invisibly. A longer system prompt silently steals room from retrieval. A growing chat history silently steals room from both. Nobody sees the trade-off happen; they just see quality degrade or requests fail once real usage stretches the inputs. The engineer's job is to make that competition explicit: decide up front how many tokens each consumer may claim, and check the arithmetic before you send, not after the provider rejects you.

The common mistake is reserving nothing for the output. People fill the window right up to the limit with input and are baffled when the model's answer gets cut off mid-sentence — the reply had nowhere to go. Always carve out a fixed reserve_out slot first, then treat everything else as spendable. The tiny function below is nothing more than that piece of accounting: add up the committed marbles, subtract from the jar, and tell you whether you're over budget before it becomes a runtime crash. It looks trivial, and that's the point — the discipline is in running it every time, not in the maths.

The context window is finite and every token costs money + attention. It's split across the system prompt, conversation history, retrieved context (RAG), tool definitions, and room for the response. Engineering context = allocating that budget.

budget.pydef budget(window, system, history, tools, reserve_out):
    used = system + history + tools + reserve_out
    for_rag = window - used
    return {"available_for_rag": for_rag, "over_budget": for_rag < 0}

print(budget(window=200_000, system=2_000, history=8_000, tools=1_500, reserve_out=4_000))
print(budget(window=8_000, system=2_000, history=8_000, tools=1_500, reserve_out=4_000))

{'available_for_rag': 184500, 'over_budget': False}
{'available_for_rag': -7500, 'over_budget': True}

3 · Intermediate — rank & trim retrieved context intermediate

Everyone's first instinct is wrong here, and it's an understandable instinct: more information must be better. If the model needs facts to answer well, surely handing it twenty relevant-looking documents beats handing it three? So you crank up your retriever's top-k, stuff the window, and — the answers get worse. Slower, vaguer, sometimes flatly ignoring the one document that mattered. This surprises people so reliably it has a name.

The effect is called "lost in the middle." Language models attend most strongly to what's at the very start and the very end of their context, and material buried in the middle of a long stuffed window gets comparatively ignored — not deleted, just drowned out. Every irrelevant chunk you add doesn't sit politely in a corner; it competes for the model's finite attention against the chunks that actually answer the question. Think of asking a colleague a quick question while sliding forty pages of loosely-related notes across the desk. The one paragraph they needed is in there somewhere, but now they have to find it, and the noise makes them worse at the task, not better.

So the goal flips from "retrieve everything relevant" to "retrieve generously, then curate ruthlessly." You cast a wide net at the retrieval step — better to have the right chunk somewhere in your candidate pile than to miss it — and then you rank those candidates by relevance and keep only the strongest few that fit your token budget from §2. Signal in, noise out. The window is precious; every slot should earn its place by being more useful than the chunk it displaced.

The mechanism is a greedy fit: sort the candidates best-score-first, then walk down the list adding each chunk only if it still fits the budget, stopping when you run out of room. That word "greedy" is a real algorithmic idea — take the locally best option at each step — and here it's exactly right because a higher-scoring chunk is always worth more than a lower one, so grabbing the best available first can't backfire. The common mistake it guards against is the reverse: keeping chunks in retrieval order or by whichever arrived first, so a low-value 500-token chunk hogs room a high-value one needed. Rank first, then fit. The lab below is that pattern in a dozen lines, and it is the beating heart of context selection in every serious RAG system.

More context isn't better — irrelevant chunks dilute attention ("lost in the middle"). Retrieve generously, then rank and keep only the top-k that fit the budget.

rank.pydef fit_context(chunks, budget_tokens):
    """chunks: [(text, score, tokens)]. Keep highest-score chunks that fit."""
    ranked = sorted(chunks, key=lambda c: c[1], reverse=True)
    kept, used = [], 0
    for text, score, tok in ranked:
        if used + tok <= budget_tokens:
            kept.append(text); used += tok
    return kept, used

chunks = [("refund policy", 0.9, 300), ("hours", 0.4, 100),
          ("shipping", 0.7, 250), ("history trivia", 0.1, 500)]
kept, used = fit_context(chunks, budget_tokens=600)
print("kept:", kept, "| tokens:", used)

kept: ['refund policy', 'shipping'] | tokens: 550

4 · Advanced — compress & structure advanced

Sooner or later the budget from §2 stops being enough, and trimming from §3 can only cut so much before you're throwing away things you actually need. This is the wall every long-running assistant hits: a conversation that's gone thirty turns deep, or a set of documents that all genuinely matter but collectively overflow the window. You can't just drop half of it. The move now isn't to discard information — it's to compress it, to pack the same signal into fewer tokens. This is "context rot" in action: as the window fills over a long session, older material has to be squeezed down or it slowly crowds out everything else.

The clearest example is conversation history. Turn twenty-five of a chat doesn't need turn one word-for-word — but it might need to know that turn one established the customer's account number and that they're annoyed. So you keep the last few turns verbatim, because recent turns carry the live intent the model is mid-way through acting on, and you collapse everything older into a short summary line: "earlier: customer verified account, disputed a double charge, wants a refund." One line where there were fifteen. Think of it as taking meeting minutes instead of keeping the full transcript — you lose the exact words but preserve what happened, at a fraction of the space.

The same instinct applies beyond history. Deduplicate overlapping retrieved chunks — two documents that say nearly the same thing waste a slot each when one would do. And structure what remains: wrap sections in clear delimiters or headings so the model can tell the system instructions from the retrieved evidence from the user's question. A model parses a well-labelled context far more reliably than a wall of concatenated text, the same way you skim a documented report faster than an undifferentiated dump.

One placement rule ties this section to §3's "lost in the middle": put your most important context at the start or the end, where attention is strongest, and let the compressible filler live in the middle. The common mistake here is compressing too aggressively or too early — summarising a turn the model still needs in full, and watching it lose the thread. Compression is a lossy trade, so protect the recent and the critical, and only squeeze what you can genuinely afford to blur. The lab keeps the last N turns intact and summarises the rest — a crude clip here, an LLM summary in production, but the "keep-recent, summarise-the-old" shape is exactly what real agents run.

Techniques to fit more signal in fewer tokens: summarize old history, dedup overlapping chunks, structure with clear delimiters so the model parses reliably, and put the most important context where the model attends best (start/end).

compress.pydef compress_history(turns, keep_recent=2):
    """Keep recent turns verbatim; summarize the rest into one line."""
    if len(turns) <= keep_recent:
        return turns
    old, recent = turns[:-keep_recent], turns[-keep_recent:]
    summary = f"[summary of {len(old)} earlier turns: " + \
              "; ".join(t[:20] for t in old) + "]"
    return [summary] + recent

hist = ["user asked about refunds", "assistant explained policy",
        "user asked about shipping", "assistant gave times",
        "user asks about returns now"]
for line in compress_history(hist): print("-", line)

- [summary of 3 earlier turns: user asked about ref; assistant explained ; user asked about sh]
- assistant gave times
- user asks about returns now

5 · Professional — prompt caching professional

Here's where context engineering stops being about quality and starts being about the bill. Suppose you've done everything right — a tight, well-structured 10,000-token system prompt full of instructions, tool definitions, and a few examples. It's great. It's also sent on every single call, and if you serve a hundred requests you've paid to process those same 10,000 tokens a hundred times: a million tokens of pure repetition. The model re-reads the identical prefix from scratch each time as if it had never seen it. That's the waste prompt caching exists to kill.

The idea is exactly what it sounds like. The first time the provider sees your big stable prefix, it does the expensive work of processing it and saves that computed state. On later calls that begin with the same prefix, it reuses the saved state instead of recomputing — and charges you a small fraction (roughly a tenth) for the cached read. You pay full freight once to "warm" the cache, then a discount forever after. It's the difference between rebuilding your development environment from source every morning versus restoring it from a snapshot in seconds.

This is the same mechanism you met as the KV-cache in IC3, but lifted from the token level to the prompt level and exposed as a feature you control. And the control is the whole game, because caching only works on an exact, unchanged prefix. That gives you one non-negotiable layout rule: put your largest stable content first — system prompt, tool definitions, fixed few-shot examples — and put the volatile per-request material (the user's question, freshly retrieved chunks) after it. Order your context stable-to-volatile and the cache boundary lands naturally where it saves the most.

The common mistake sabotages the cache without anyone noticing: slipping something that changes — a timestamp, a request ID, a per-user greeting — into the front of the prompt. Change one token near the start and the entire prefix after it is now "different," the cache misses, and you're back to paying full price on every call while wondering where your savings went. Keep the front byte-for-byte identical across calls. The lab below models the arithmetic so you can feel the shape of the win: the payoff is small for a couple of calls and enormous for a prefix reused hundreds of times, which tells you exactly when caching is worth wiring up.

When a large context prefix repeats across calls (a long system prompt, a big RAG doc), prompt caching reuses its computed state — a big cost/latency win. The same lever as IC3, applied at the prompt level.

cache.pydef cache_savings(prefix_tokens, calls, cache_read_ratio=0.1):
    """First call pays full; later calls pay cache_read_ratio for the prefix."""
    no_cache = prefix_tokens * calls
    with_cache = prefix_tokens + prefix_tokens * cache_read_ratio * (calls - 1)
    return round(no_cache), round(with_cache), round(100*(no_cache-with_cache)/no_cache)

nc, wc, saved = cache_savings(prefix_tokens=10_000, calls=100)
print(f"no cache: {nc} tok | cached: {wc} tok | saved {saved}% on the prefix")

no cache: 1000000 tok | cached: 109000 tok | saved 89% on the prefix

6 · Tech-lead — a context strategy for the system tech-lead

Everything so far has been a tool. This section is about not leaving those tools in the hands of whoever happens to be writing each feature. The failure it prevents is subtle and organisational: five engineers each build a slightly different context-assembly routine, each guesses at how many tokens to reserve, each handles over-budget requests their own way — and the system's behaviour under load becomes a lottery. One endpoint truncates silently, another crashes, a third quietly disables caching. A tech lead's job is to make context assembly a property of the platform, decided once, rather than an ad-hoc string-concat re-invented per feature.

The shift in thinking is from "how do I build a good context for this request" to "what is the policy every request obeys." That means fixing the allocations up front: the output reserve is this many tokens, history is capped at that many, tool definitions at that many — and retrieval gets whatever remains, no more. Hard caps on the sections you control turn an unbounded, unpredictable input into a bounded, predictable one. It's the difference between a kitchen where every cook eyeballs the portions and one with a written recipe: the second is the only one that tastes the same at scale.

The move that most separates a lead's design from a beginner's is how it handles going over budget. The naive system lets the request balloon and crashes when the provider rejects it — a runtime error in a customer's face. A lead makes over-budget a graceful degradation, not a failure: apply the caps, and if there's still not enough room, trim retrieval down to what fits and compress history harder, so the request succeeds with slightly less evidence rather than dying. You reject cleanly only in the genuinely impossible case where the fixed sections plus the output reserve already exceed the window. Everything short of that should bend, not break.

Finally, a strategy you can't see is a strategy you can't defend. The lead pins the cache boundary system-wide (stable prefix first, always) so caching savings are automatic rather than accidental, and measures token cost per request as a first-class metric — because the same pipeline that keeps answers accurate is the one that keeps the monthly bill survivable, and you cannot manage what you don't watch. Context, at this altitude, is an engineered, budgeted, measured pipeline. The exercises that follow build it one stage at a time and then compose the whole thing.

A lead designs the whole context pipeline: fixed budget allocations, a retrieval+rank+trim stage, history compression, and caching of stable prefixes — then monitors token cost per request. This is what keeps a RAG/agent system both accurate and affordable at scale.

def budget(window, system, history, tools, reserve_out):
    used = system + history + tools + reserve_out
    for_rag = window - used
    return {"available_for_rag": for_rag, "over_budget": for_rag < 0}

print(budget(window=200_000, system=2_000, history=8_000, tools=1_500, reserve_out=4_000))
# {'available_for_rag': 184500, 'over_budget': False}
print(budget(window=8_000, system=2_000, history=8_000, tools=1_500, reserve_out=4_000))
# {'available_for_rag': -7500, 'over_budget': True}

def fit_context(chunks, budget_tokens):
    ranked = sorted(chunks, key=lambda c: c[1], reverse=True)   # by score desc
    kept, used = [], 0
    for text, score, tok in ranked:
        if used + tok <= budget_tokens:
            kept.append(text); used += tok
    return kept, used

chunks = [("refund policy", 0.9, 300), ("hours", 0.4, 100),
          ("shipping", 0.7, 250), ("history trivia", 0.1, 500)]
kept, used = fit_context(chunks, budget_tokens=600)
print(kept, "using", used, "tokens")   # ['refund policy', 'shipping'] using 550

def compress_history(turns, keep_recent=2):
    if len(turns) <= keep_recent:
        return turns
    old, recent = turns[:-keep_recent], turns[-keep_recent:]
    summary = f"[summary of {len(old)} earlier turns: " + \
              "; ".join(t[:20] for t in old) + "]"
    return [summary] + recent

turns = ["hi I need help with billing",
         "my card was charged twice",
         "the amount was 49 dollars",
         "can you refund one charge",
         "yes please do it today"]
out = compress_history(turns, keep_recent=2)
print(len(out), "items")   # 3: one summary + last two turns
for t in out: print(" -", t)

def cache_savings(prefix_tokens, calls, cache_read_ratio=0.1):
    no_cache = prefix_tokens * calls
    with_cache = prefix_tokens + prefix_tokens * cache_read_ratio * (calls - 1)
    saved_pct = round(100 * (no_cache - with_cache) / no_cache)
    return round(no_cache), round(with_cache), saved_pct

nc, wc, pct = cache_savings(prefix_tokens=10_000, calls=100, cache_read_ratio=0.1)
print(f"no cache: {nc:,}  with cache: {wc:,}  saved: {pct}%")
# no cache: 1,000,000  with cache: 109,000  saved: 89%

def budget(window, system, history, tools, reserve_out):
    return window - (system + history + tools + reserve_out)

def fit_context(chunks, budget_tokens):
    kept, used = [], 0
    for text, score, tok in sorted(chunks, key=lambda c: c[1], reverse=True):
        if used + tok <= budget_tokens:
            kept.append(text); used += tok
    return kept

def build_context(window, system_tok, history_turns, tool_tok, reserve, chunks):
    hist_tok = 200 if len(history_turns) > 2 else 100   # after compression
    rag_budget = budget(window, system_tok, hist_tok, tool_tok, reserve)
    kept = fit_context(chunks, rag_budget)
    return {"cacheable_prefix": ["system", "tool_defs"],   # stable -> cache first
            "rag_budget": rag_budget, "chunks_kept": kept}

chunks = [("refund", 0.9, 300), ("shipping", 0.7, 250), ("trivia", 0.1, 500)]
print(build_context(8_000, 2_000, ["a","b","c","d"], 1_500, 1_000, chunks))

POLICY = {"reserve_out": 4_000, "max_history": 2_000, "max_tools": 1_500}

def enforce(window, system, history, tools, rag_want):
    # step 1: hard caps on the controllable sections
    history = min(history, POLICY["max_history"])
    tools   = min(tools,   POLICY["max_tools"])
    fixed = system + history + tools + POLICY["reserve_out"]
    rag_budget = window - fixed
    if rag_budget < 0:
        return {"status": "REJECT: prompt+reserve exceed window", "rag": 0}
    # step 2: trim retrieval to whatever budget remains
    rag = min(rag_want, rag_budget)
    return {"status": "OK", "cached_prefix": ["system", "tool_defs"],
            "rag_tokens": rag, "trimmed": rag < rag_want}

print(enforce(8_000, 2_000, 6_000, 3_000, rag_want=5_000))  # caps history+tools, trims RAG

Knowledge check check yourself