Context Engineering: Memory, Windowing & Retrieval

The model's context window is a fixed number of tokens (think of it as a jar that holds a set number of marbles). Everything you send — the system prompt, the chat history, the tool descriptions, and the space you reserve for the model's reply — all shares that one jar. This tiny function does the bookkeeping: it adds up what's already claimed, then tells you how much room is left for retrieved documents (RAG).

1. budget(window, system, history, tools, reserve_out) takes the total window size and the token cost of each thing you plan to put in it. window is the jar; the rest are marbles you've already committed.
2. used = system + history + tools + reserve_out sums everything that is not retrieved context — including reserve_out, the room you deliberately keep free for the answer the model will write.
3. for_rag = window - used is what's left over — the budget you can spend on retrieved knowledge. If this goes negative, you've promised more than the jar holds.
4. The function returns a small dictionary: available_for_rag (the leftover room) and over_budget (True when for_rag is below zero).

What the output means: The first call uses a huge 200,000-token window, so ~184,500 tokens are free for RAG and over_budget is False. The second call uses a small 8,000-token window with the same demands, so it comes up 7,500 tokens short — for_rag is -7500 and over_budget is True. Same requests, different jar: one fits, one doesn't.

Try this: Lower the first call's window to 15_000 and see it flip to over-budget. This is exactly the calculation you run before deciding how many documents you can afford to retrieve.

When you search a knowledge base you often get back more chunks of text than will fit in your budget — and stuffing in irrelevant ones actually hurts the answer (the model gets 'lost in the middle'). The fix: sort the chunks best-first, then greedily keep only the top ones that still fit the token budget.

1. Each chunk is a triple (text, score, tokens): the content, how relevant the search thought it was (higher = better), and how many tokens it costs.
2. ranked = sorted(chunks, key=lambda c: c[1], reverse=True) sorts the list by c[1] — the score — with reverse=True so the highest scores come first. Now the most relevant chunks are at the front.
3. The loop walks the ranked chunks and, for each, checks if used + tok <= budget_tokens — 'does adding this one still fit?'. If yes, it keeps the text and adds its cost to the running used total. If not, it simply skips it.
4. It returns the list of kept texts and the total used tokens.

What the output means: With a 600-token budget it keeps 'refund policy' (score 0.9, 300 tok) and 'shipping' (score 0.7, 250 tok) for 550 tokens total. The 0.4 'hours' chunk would fit but ranks lower than shipping, and 'history trivia' (score 0.1) is both low-value and too big — both are dropped. You kept the signal and left out the noise.

Try this: Raise budget_tokens to 900 and re-run — now 'hours' sneaks in too, because there's room after the top two. This greedy 'best-first, stop when full' pattern is the core of context selection in real RAG systems.

A long conversation keeps growing, and every past turn costs tokens on every new call. Instead of sending the whole history forever, you keep the last few turns word-for-word (recent context matters most) and squash everything older into a single short summary line.

1. compress_history(turns, keep_recent=2) takes the list of conversation turns and how many recent ones to keep untouched (default 2).
2. if len(turns) <= keep_recent: — if the conversation is still short, there's nothing to compress, so it returns the turns unchanged.
3. old, recent = turns[:-keep_recent], turns[-keep_recent:] splits the list: turns[-2:] is the last two turns (kept verbatim), and turns[:-2] is everything before them (to be summarized).
4. The summary line stitches the old turns into one string, using t[:20] to keep only the first 20 characters of each — a crude but cheap way to shrink them. The function returns [summary] + recent: one summary line followed by the recent turns.

What the output means: Five turns become three lines: one [summary of 3 earlier turns: ...] line (each old turn clipped to 20 chars), then the two most recent turns in full. You've preserved the immediate context while cutting the older turns down to a fraction of their token cost.

Try this: Change keep_recent to 1 and re-run — now 4 turns get summarized and only the very last one stays verbatim. Real systems summarize with an LLM instead of clipping to 20 characters, but the keep-recent / summarize-the-rest shape is identical.

If the same big chunk of text sits at the start of many calls — a long system prompt or a large reference document — the model normally re-reads and re-processes it every single time. Prompt caching lets the provider remember that prefix after the first call, so repeat calls pay only a small fraction to reuse it. This function estimates how much you'd save.

1. cache_savings(prefix_tokens, calls, cache_read_ratio=0.1) takes the size of the repeated prefix, how many times you call it, and the discount for reads from cache (0.1 means a cached read costs 10% of a fresh one).
2. no_cache = prefix_tokens * calls — without caching you pay full price for the prefix on every call.
3. with_cache = prefix_tokens + prefix_tokens * cache_read_ratio * (calls - 1) — you pay full price once to fill the cache, then only the cheap ratio for each of the remaining calls - 1 reads.
4. It returns both totals plus the percentage saved, so you can see the win at a glance.

What the output means: For a 10,000-token prefix reused across 100 calls: no cache = 1,000,000 tokens; with cache = 109,000 tokens — an 89% saving on the prefix. The more calls that share the same prefix, the closer the saving climbs toward the full 90% discount.

Try this: Drop calls to 2 and re-run — the saving shrinks a lot, because you still pay the full first call and only save on one repeat. Caching pays off most when a stable prefix is reused many times.