Building Production-Grade LLM, RAG & Agentic Systems

1 Foundations & the mental model

Understand the tiers before you build. Most failures come from reaching for a heavier tier than the task needs.

Every LLM application is one of four tiers. Choose the simplest tier that solves the problem — complexity is a cost you pay in latency, money, and debuggability.

The four-question gate before building an agent

Only climb to Tier 3+ when all four hold:

• Complexity — is the task genuinely multi-step and hard to fully specify up front?
• Value — does the outcome justify higher cost and latency?
• Viability — is the model actually capable at this task type?
• Cost of error — can mistakes be caught and recovered (tests, review, rollback)?

2 The LLM serving layer

The single API call is the atom of everything above it. Get it robust first.

2.1 Model selection

Match the model tier to the workload, not the ambition. A blended fleet is normal: a capable model for reasoning-heavy paths, a fast/cheap model for classification and high-volume routing.

2.2 A production-grade client wrapper

Never call the raw SDK from business logic. Wrap it once with retries, timeouts, streaming, structured logging and cost accounting.

1. Resolve credentials from the environment — never hardcode keys. Use env vars or a secrets manager.
2. Set explicit timeouts & retries. SDKs retry 429/5xx/connection errors with backoff by default (2 retries) — tune per route.
3. Stream anything with large output. Non-streaming requests risk HTTP timeouts above ~16K output tokens.
4. Emit a request ID + token usage on every call for tracing and cost dashboards.
5. Handle the refusal / max_tokens stop reasons explicitly — don't assume content[0] is text.

Requires: pip install anthropic

# llm_client.py — the only place raw SDK calls live
import anthropic, logging, time
from anthropic import Anthropic

log = logging.getLogger("llm")
client = Anthropic()  # reads ANTHROPIC_API_KEY from env

def complete(messages, *, system=None, model="claude-opus-4-8",
             max_tokens=4096, effort="high", tools=None):
    t0 = time.monotonic()
    with client.messages.stream(         # stream = timeout-safe
        model=model, max_tokens=max_tokens, system=system,
        messages=messages, tools=tools or [],
        thinking={"type": "adaptive"},   # let the model decide depth
        output_config={"effort": effort},  # low|medium|high|max
    ) as stream:
        msg = stream.get_final_message()

    if msg.stop_reason == "refusal":
        log.warning("refusal req=%s cat=%s", msg._request_id,
                    msg.stop_details.category if msg.stop_details else None)
        raise RefusalError(msg)

    log.info("req=%s in=%d out=%d cache_read=%d ms=%d",
        msg._request_id, msg.usage.input_tokens, msg.usage.output_tokens,
        msg.usage.cache_read_input_tokens, (time.monotonic()-t0)*1000)
    return msg

2.3 Resilience patterns

3 Prompt engineering & context management

The prompt is code. Version it, test it, and keep its structure cache-friendly.

3.1 Structure a system prompt

1. Role & objective — who the model is and what "done" means.
2. Operating constraints — tone, length, formatting, what it must never do.
3. Tools & when to use them — be prescriptive about when to call, not just what a tool does.
4. Few-shot examples — positive examples of the desired output beat lists of prohibitions.
5. Dynamic context last — retrieved docs / user data go after the frozen prefix (see caching).

3.2 Prompt versioning & registry

Treat prompts as deployable artifacts, not string literals buried in code.

• Store prompts in a versioned file/registry with an ID and semver.
• Log the prompt version on every request so you can attribute quality regressions.
• Roll out prompt changes behind the same flags you'd use for code (canary %, A/B).

3.3 Context-window management

Context is finite and the front of it is the most expensive to change. Three levers for long-running sessions:

4 RAG pipelines — the full build

Retrieval-Augmented Generation grounds the model in your data. Most "hallucination" bugs are actually retrieval bugs.

4.1 Ingestion — build the index

1. Collect & normalize sources. PDFs, HTML, wikis, tickets. Strip boilerplate; preserve structure (headings, tables) — it carries meaning.
2. Chunk deliberately. Not fixed 512-token windows blindly. Chunk on semantic boundaries (headings, paragraphs), keep 10–20% overlap, and store a chunk's parent-section as metadata. Target 200–500 tokens for precision; add a "parent-document" fallback for context.
3. Attach rich metadata. source, title, section, url, updated_at, ACL/tenant tags. Metadata powers filtering and citations.
4. Embed. Choose an embedding model sized to your latency/quality budget; keep the same model for query and index (mismatched embedders = garbage retrieval).
5. Upsert into a vector store with the vector, the raw text, and metadata. Make ingestion idempotent (dedupe by content hash) so re-runs don't duplicate.

4.2 Retrieval — find the right context

• Hybrid search. Combine dense (vector) with sparse (BM25/keyword). Dense catches semantics; sparse catches exact terms, codes, names. Fuse with Reciprocal Rank Fusion.
• Query transformation. Rewrite conversational queries into standalone search queries; optionally expand into multiple sub-queries for multi-hop questions.
• Metadata filtering. Always scope by tenant/ACL before semantic search — never rely on the model to respect permissions.
• Re-ranking. Over-fetch (e.g. top-30) with cheap retrieval, then re-rank to top-5 with a cross-encoder or an LLM. This is the single highest-leverage quality lever.

4.3 Generation — ground the answer

class _Any:
    '''stands in for any undefined demo value; supports call/attr/index/
    iteration and basic arithmetic (as 0.7) so demo snippets run.'''
    def __call__(self, *a, **k): return _Any()
    def __getattr__(self, k): return _Any()
    def __getitem__(self, k): return _Any()
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __contains__(self, o): return True
    def __enter__(self, *a): return _Any()
    def __exit__(self, *a): return False
    def __float__(self): return 0.7
    def __int__(self): return 1
    def __lt__(self, o): return True
    def __gt__(self, o): return False
    def __le__(self, o): return True
    def __ge__(self, o): return False
    def __add__(self, o): return o
    def __radd__(self, o): return o
    def __bool__(self): return True
    def __repr__(self): return 'demo'
    def __str__(self): return 'demo'
def complete(*a, **k):  # demo stub
    return _Any()
query = _Any()
reranked_chunks = _Any()

# Assemble retrieved chunks into a grounded prompt with citations
context = "\n\n".join(
    f"[{i}] (source: {c['title']} — {c['url']})\n{c['text']}"
    for i, c in enumerate(reranked_chunks, 1)
)

system = (
    "Answer ONLY from the numbered context below. "
    "Cite sources inline as [n]. If the context does not contain "
    "the answer, say you don't know — do not use outside knowledge."
)

msg = complete(
    system=system,
    messages=[{"role":"user",
               "content": f"Context:\n{context}\n\nQuestion: {query}"}],
    effort="medium",
)

4.4 Advanced RAG patterns

4.5 What to measure

5 Agents & tool use

An agent is a loop: model proposes a tool call → your harness executes it → result goes back → repeat until done.

5.1 Designing the tool surface

The shape of your tools decides what your harness can control. A bash tool gives breadth but an opaque command string; a dedicated typed tool gives the harness a hook to gate, render, audit, and parallelize.

5.2 A manual agentic loop

Use the manual loop (over the SDK's auto tool-runner) when you need approval gates, custom logging, or conditional execution.

Uses the client / objects set up earlier in this lesson, and needs ANTHROPIC_API_KEY set. Run the earlier blocks first.

messages = [{"role":"user", "content": user_input}]
for _ in range(MAX_STEPS):        # hard cap — never an unbounded loop
    resp = client.messages.create(model="claude-opus-4-8",
             max_tokens=4096, tools=TOOLS, messages=messages)

    if resp.stop_reason == "end_turn":
        break

    messages.append({"role":"assistant", "content": resp.content})
    results = []
    for block in resp.content:
        if block.type == "tool_use":
            if is_destructive(block.name):     # human-in-the-loop gate
                if not approve(block): continue
            out = execute_tool(block.name, block.input)  # validate inside!
            results.append({"type":"tool_result",
                            "tool_use_id": block.id, "content": out})
    messages.append({"role":"user", "content": results})  # all results, one message

5.3 Long-running agents

Combine context editing (prune stale turns), compaction (summarize near the limit), and file-based memory (cross-session state). Use tool search when the tool count is large so schemas load on demand instead of bloating the prefix.

6 Structured output & function calling

If a downstream system parses the output, constrain it to a schema. Never regex free text.

6.1 Schema-constrained JSON

Uses the client / objects set up earlier in this lesson, and needs ANTHROPIC_API_KEY set. Run the earlier blocks first.

from pydantic import BaseModel

class Ticket(BaseModel):
    category: str
    priority: str        # enum in the JSON schema
    summary: str
    needs_human: bool

resp = client.messages.parse(          # validates & returns typed object
    model="claude-opus-4-8", max_tokens=1024,
    messages=[{"role":"user", "content": ticket_text}],
    output_format=Ticket,
)
ticket = resp.parsed_output            # a validated Ticket instance

6.2 When to use which

7 Evaluation & testing

You cannot improve what you cannot measure. Evals are the CI of LLM systems.

7.1 The evaluation pyramid

1. Unit-level (deterministic) — assertions on structured output: valid JSON, required fields, enum membership, no PII leak. Fast, run on every commit.
2. Reference-based — a curated dataset of inputs with known-good outputs. Score with exact-match, F1, or embedding similarity for open-ended answers.
3. LLM-as-judge — a stronger model grades quality against a rubric (groundedness, helpfulness, tone). For confidence, use a panel with diverse lenses and majority vote.
4. Human review — sample real traffic; label failures; feed them back into the reference set.

7.2 A minimal eval harness

Illustrative fragment — defines demo values / files are needed before this runs standalone.

for case in golden_set:                     # {input, expected, rubric}
    got = run_pipeline(case["input"])
    scores["schema_ok"].append(validate(got))       # deterministic
    scores["grounded"].append(judge(got, case["rubric"]))  # LLM judge
    scores["sim"].append(cosine(embed(got), embed(case["expected"])))

report(scores)   # fail the build if any metric regresses vs. baseline

7.3 Testing discipline

• Golden set in version control. Grow it from every production incident.
• Regression gate. A prompt or model change that drops a metric fails CI, like any test.
• Offline before online. Prove gains on the eval set before shipping; confirm with online A/B.
• Cost & latency are metrics too. Track them alongside quality — a 2% quality gain at 3× cost may not ship.

8 Observability

In production you debug from traces, not from reproducing a stochastic bug locally.

8.1 Trace everything

Capture a full trace per request: the rendered prompt, retrieved chunks, every tool call and result, the model's output, token usage, latency per stage, cost, and the resolved prompt/model versions.

8.2 The dashboards that matter

9 Guardrails & safety

Validate on the way in and on the way out. Both are cheap insurance against expensive incidents.

9.1 Input guardrails

• PII detection & redaction before the prompt reaches the model.
• Prompt-injection screening — treat retrieved content and tool output as untrusted; never let it silently override system instructions.
• Rate limiting & abuse detection per user/tenant.
• Topic / policy filters for out-of-scope or disallowed requests.

9.2 Output guardrails

• Schema & format validation (covered in §6) — reject malformed output before it reaches users.
• Groundedness check for RAG — flag or block claims not supported by retrieved context.
• PII / secret egress scan — don't return data the user isn't entitled to.
• Content safety classifier on the response.

9.3 Human-in-the-loop

For high-stakes actions (financial transactions, sending communications, deleting data), require explicit confirmation. Design the confidence threshold so the system escalates to a human when uncertain rather than guessing.

10 Caching, cost & scaling

Cost is an architecture decision made early, not a knob turned late.

10.1 Prompt caching — the biggest lever

Caching is a prefix match: any byte change anywhere in the prefix invalidates everything after it. Render order is tools → system → messages.

1. Freeze the prefix. Keep the large stable content (system prompt, tool defs, shared context) first and byte-identical across requests.
2. Move volatile content to the end. Timestamps, per-request IDs, the user's actual question go after the last cache breakpoint.
3. Place a breakpoint on the last stable block. Cache reads cost ~10% of base input price.
4. Verify with cache_read_input_tokens. If it's zero across identical-prefix requests, hunt the silent invalidator (a datetime.now() or unsorted JSON in the prefix).

10.2 Other cost levers

10.3 Scaling

• Respect rate limits (RPM/TPM). Use a token-bucket limiter and queue overflow rather than hammering 429s.
• Concurrency caps on fan-out — parallel requests with identical prefixes all miss the cache (nothing to read yet); warm it with one request first, then fan out.
• Async I/O for high-throughput services; stream to keep connections healthy under large outputs.

11 Security

LLM apps inherit every classic app-sec concern plus a few new ones.

12 Deployment & LLMOps

Prompts, models, retrieval indices and eval sets are all versioned artifacts in your release process.

1. CI: run the eval suite. Deterministic checks on every commit; full eval on prompt/model changes. Regression = failed build.
2. Version everything. Prompt version, model ID, embedding-model version, index snapshot. Log them per request for attribution.
3. Canary & A/B. Roll new prompts/models to a small % first; compare quality, latency, cost against control.
4. Feature-flag model swaps. A model change invalidates prompt caches and shifts behavior — roll it like any risky change, with instant rollback.
5. Migration discipline. When upgrading models, re-baseline token counts and cost, re-tune prompts, and re-run the eval set — behavior shifts even when the API is compatible.
6. Index refresh pipeline. Schedule re-ingestion; version index snapshots so you can roll back a bad rebuild.

13 Reference architectures

Three blueprints assembled from the building blocks above.

13.1 Production RAG assistant

API gateway → input guardrails (PII, injection) → query rewrite → hybrid retrieval (vector + BM25, ACL-filtered) → re-rank → assemble cited context (cache the frozen instruction prefix) → LLM with groundedness output-guard → response + citations. Traced end-to-end; nightly eval on a golden set; index refreshed on a schedule.

13.2 Tool-using agent

Gateway → guardrails → agent loop (hard step cap) with a small typed tool surface; irreversible tools gated by human-in-the-loop; context editing + compaction for long runs; every tool call traced and every input validated/sandboxed. Fall back to a cheaper model under load.

13.3 High-volume classification service

Gateway → small/fast model with a forced tool or enum schema (strict: true); aggressive prompt caching of the frozen instruction block; semantic cache in front for repeat inputs; batch API for backfills. Deterministic evals gate every deploy.

14 Production go-live checklist

Before you flip the flag to 100%.