LLM Infrastructure, Tooling & the Open-Source Stack

1 · The stack between app and model essential

Start with the 3 a.m. pager. Your LLM app is throwing errors and you're the one on call. Is it the model provider that's down? A rate limit you hit? A retrieval query returning garbage? A prompt version someone changed yesterday? A runaway bill? If in your head the whole system is one undifferentiated blob called "the AI thing," you will flail — poking at random until something works. The engineers who fix these fast have a map: they know each of those failures lives in a different layer, and they go straight to the right one. This section hands you that map.

Here's the mental model that makes it click. Think of an LLM system the way you already think of a web app. Nobody says "my web app is slow" and starts editing raw HTML — they ask "is it the load balancer, the app server, the database, or the CDN?" because they've internalised the tiers. LLM systems have grown their own tiers, and they map almost one-to-one onto problems you'll actually hit: which provider do we call and can we fail over? (gateway), can we run the model fast enough? (serving), which version of the model and prompt is live? (registry), how do we find the right documents? (vector store), is the output any good? (eval), and what happened on that one bad request? (observability). Learn the layers and you've learned where every question belongs.

The common mistake is to skip the layering entirely and wire your app directly to one provider's SDK, sprinkling retries, cost logging, and prompt strings through your business logic. It works beautifully — for exactly one provider, at low volume, until the day you need to switch models, add a fallback, or explain last month's bill. Then you discover the plumbing is welded to the app and every change touches a hundred files. The layered view isn't bureaucracy; it's the thing that lets you swap any one piece without rewriting the others. Each layer is a seam — a place designed to come apart.

Between your app and a model sits a stack: gateways, serving engines, registries, eval harnesses, vector stores, and observability. Each layer solves one problem; knowing what lives where is how you reason about (and debug) an LLM system.

2 · The layers essential

Now let's name the tiers, one row at a time. The table below is the whole lesson in miniature — six rows, each a layer with a single job and a couple of real tools that do it. Don't try to memorise the tool names; they change every quarter and half of them do overlapping jobs anyway. What's durable is the job in the middle column. Tools come and go; the six responsibilities are stable, and an engineer who knows the responsibilities can slot any new tool into the right row on sight.

A useful way to read it: each layer answers "who owns this concern?" so that no other layer has to. That single-responsibility discipline is exactly the seams idea from §1, made concrete. The gateway owns "which provider, and what if it fails" so your app never hard-codes a vendor. Serving owns "run this open model fast on our hardware" so nothing above it worries about GPUs and batching. The registry owns "which exact version of model and prompt is this" so you can answer "what changed?" after a regression. The vector store owns semantic retrieval — the search half of RAG. Eval owns "is the answer actually good," the question every other layer quietly assumes someone else is asking. And observability owns the trace/log/cost record of every request, which is what turns that 3 a.m. pager from guesswork into a lookup.

The mistake to avoid is reading this as a rigid, buy-all-six shopping list. Early on, several of these layers are just a few lines of your own code — a dict of prompts is a registry, a try/except loop is a gateway, a pytest file is an eval harness. The value of the table isn't "go install six products"; it's knowing which concern a given tool or snippet belongs to, so that when a concern outgrows your hand-rolled version you know exactly what to reach for and where it plugs in. Read the rows as roles to fill, not boxes to buy.

3 · Intermediate — the gateway pattern intermediate

Imagine your electricity came from one specific power plant. Not "the grid" — that one plant, wired straight into your house. When it goes down, you're dark. When it raises prices, you have no alternative. When a cleaner plant opens next door, you can't switch without rewiring the whole building. That's what talking directly to one model provider's SDK from all over your app feels like once the app is real. A gateway is the electrical grid: a single socket you plug into, behind which providers can be added, swapped, failed over, and metered without you ever touching your appliances.

Concretely, a gateway is one API that sits in front of many model providers and takes on the boring-but-critical cross-cutting jobs so your app doesn't have to: authentication (one place holds the keys), rate limits (one place backs off politely), retries and fallback (when provider A errors, quietly try provider B), and cost tracking (one place tallies every request). Your application code makes one kind of call — "complete this prompt" — and the gateway decides who actually answers. Why this is the highest-leverage layer: almost every operational lever you'll ever want — switch models, add a cheaper tier for easy requests, cap spend, log everything, fail over on an outage — becomes a change inside the gateway rather than a change scattered across your whole codebase. Build one seam well and it pays off at every other layer.

The classic mistake is building the gateway too fat, too early. People try to make it a full-blown intelligent router with caching, semantic dedup, prompt rewriting, and a dashboard before they've shipped anything. Resist that. The gateway earns its keep as a thin seam: a single entry point that does routing, fallback, and accounting, and stays out of the way otherwise. The other trap is the opposite — never building it, and letting each new feature reach for the provider SDK directly, so the "swap is just config" promise silently dies. The whole point is one door; the moment your app knows two providers' names, the door has a hole in it. Notice this is just the provider-agnostic client you met in DF4, promoted from a helper function to a piece of infrastructure everything routes through.

A gateway is the highest-leverage layer: one API in front of many providers, with auth, rate limits, retries, fallback, and cost tracking. Your app talks to the gateway; swapping models or providers becomes config. This is the provider-agnostic client (DF4) as infrastructure.

gateway.pyclass Gateway:
    def __init__(self):
        self.spend = 0.0
    def _call(self, provider, prompt):
        if provider == "down": raise RuntimeError("provider unavailable")
        self.spend += 0.002 * len(prompt.split())     # fake per-token cost
        return f"[{provider}] answer to: {prompt[:20]}"
    def complete(self, prompt, providers):
        for prov in providers:                          # fallback chain
            try:
                return self._call(prov, prompt)
            except RuntimeError:
                continue
        raise RuntimeError("all providers failed")

gw = Gateway()
print(gw.complete("summarize the incident report", providers=["down", "claude"]))
print("spend so far: $%.4f" % gw.spend)

[claude] answer to: summarize the incid
spend so far: $0.0080

4 · Advanced — serving open models advanced

Here's the trap that sinks first-time self-hosters. You download an open model, write the ten lines of Python that load it and call generate() in a loop, point some traffic at it — and it works. Then you turn up the load and it falls over: the GPU sits half-idle, latency balloons, and your cost-per-request is somehow worse than the managed API you were trying to escape. The naive load-and-loop approach wastes the one expensive thing you're paying for — the GPU — by feeding it one request at a time while it could be chewing through dozens. Serving models fast is a genuinely hard systems problem, and it is not the same skill as calling a model.

A serving engine — vLLM or TGI are the common ones — is the specialist that solves that problem for you. Two of its tricks are worth naming because they're where the wins come from. Continuous batching: instead of processing requests one by one, it packs many in-flight requests through the GPU together, so the hardware stays busy instead of idling between calls. And paged KV-cache (the attention cache you met in IC3): it manages the memory each in-progress generation needs the way an operating system pages memory, so you fit far more concurrent requests on the same card without running out of VRAM. The payoff is enormous — this single layer is the difference between a self-hosted model that costs a fortune per request and one that actually beats the managed price at scale. Throughput and latency are won or lost here.

The reassuring part, and the reason this layer stays clean: a good serving engine exposes an OpenAI-compatible API. That means from the outside it looks exactly like any hosted provider — same request shape, same response shape — so it slots in behind the gateway as just one more backend. Your callers never learn whether the model answering them is yours or a vendor's, which is precisely the decoupling §1 and §3 promised. The mistake to avoid here is treating serving as a bespoke special case that your app has to know about; if you find application code branching on "is this the self-hosted one?", the seam has leaked. Serving sits behind the gateway so callers don't care.

5 · Professional — build vs buy per layer professional

You don't churn your own butter to make a sandwich. You could — but the butter isn't why anyone eats your sandwich, so you buy it and spend your effort on the part that's actually yours. The same judgement governs an LLM stack, and getting it wrong is one of the most expensive mistakes a team makes. Build everything yourself and you drown in undifferentiated plumbing — running a logging pipeline that a vendor does better, for no advantage. Buy everything and you may hand your most sensitive data to a third party, or pay per-request prices that crush you at scale. The professional move is neither; it's to decide per layer, against your real constraints.

Two constraints do almost all the deciding, so learn to feel for them. The first is privacy: some layers physically touch your data — the serving engine sees every prompt, the vector store holds your documents — and if that data can't leave your walls (regulated industry, customer PII, trade secrets), those layers must be self-hosted no matter what the spreadsheet says. The second is cost at scale: at low volume a managed API is a bargain, but there's a crossover point where you're sending so much traffic that renting per-request is dramatically more expensive than running your own serving — if you have the ops team to run it. Everything that isn't pushed by one of those two forces is "undifferentiated": observability, a hosted vector DB for non-sensitive data, an eval harness. Let a vendor run those; you win nothing by doing them yourself.

The mistake that costs the most is treating this as one all-or-nothing decision — "we're a self-hosted shop" or "we're all-managed" — applied like a religion across every layer. That's how you end up self-hosting your logging (pure overhead) while sending sensitive documents to a third-party vector DB (a real risk). The discipline is to ask the same short question of each layer independently: does privacy or scale-economics force my hand here? If not, buy it. The code below turns exactly that question into a function, so you can see the rule applied layer by layer instead of argued in the abstract.

You don't build every layer. Buy managed where it's undifferentiated (observability, a hosted vector DB), build/self-host where you have a real requirement (privacy → self-served models, cost at scale → your own serving). Decide per layer against your actual constraints, not all-or-nothing.

build_buy.pydef build_or_buy(layer, sensitive_data, high_volume, team_has_ops):
    if sensitive_data and layer in ("serving", "vector store"):
        return "self-host (data stays in-house)"
    if high_volume and team_has_ops and layer == "serving":
        return "self-host (cost at scale)"
    return "buy managed (undifferentiated — don't build it)"

for layer in ["serving", "vector store", "observability"]:
    print(layer, "->", build_or_buy(layer, sensitive_data=True,
                                     high_volume=True, team_has_ops=True))

serving -> self-host (data stays in-house)
vector store -> self-host (data stays in-house)
observability -> buy managed (undifferentiated — don't build it)

6 · Tech-lead — designing the platform tech-lead

What actually separates a "platform" from "a pile of tools we happen to use"? Both might contain the same six layers. The difference is coherence — and coherence is a thing you design, not a thing that accumulates. A pile of tools is what you get when three teams each picked their own gateway, someone wired the vector store straight into their app, and the eval harness only knows about last quarter's prompts. It technically works, until you try to change one thing and discover it's load-bearing for five others. A platform is the same six layers assembled so that every seam is deliberate and any one piece can be pulled out without the rest noticing.

The tech-lead's job here is a step up from the earlier sections: not "what does each layer do" (§2) or "should I build or buy it" (§5), but how do the layers fit together into something a dozen app teams can build on for years? The organising principle is stable interfaces. App teams should code against a fixed contract — call the gateway, query the vector store, read the eval scores — while the implementations churn underneath them. That's what lets the platform swap a serving engine, re-price a provider, or migrate a vector DB without filing a hundred tickets against downstream teams. The gateway is the keystone of this: because every app already routes through it, it's the natural place to centralise auth, cost, and observability, and it's the seam that makes "swap the model" a config change org-wide rather than a migration.

The mistake that quietly kills platforms is incoherence dressed up as pragmatism — making six independent, locally-sensible tool choices with no single rule tying them together. Each decision looks fine in isolation; the sum is a system nobody can reason about, where the build/buy call was answered differently every time by whoever happened to be in the room. A coherent platform applies one rule across all six layers (the build-vs-buy rule from §5 is a fine one) and keeps the gateway thin so the expensive, self-hosted choices behind it stay swappable. The goal is that app teams code against stable interfaces while the platform evolves underneath — that's what turns a pile of tools into a platform.

LAYER = {
    "route/auth/rate-limit/fallback across providers": "gateway",
    "run an open model fast on our GPUs":              "serving",
    "version models and prompts":                      "registry",
    "semantic retrieval over documents":               "vector store",
    "measure answer quality":                          "eval",
    "trace, log, and cost every request":              "observability",
}
def which_layer(need):
    return LAYER.get(need, "unknown -- restate the need")

print(which_layer("route/auth/rate-limit/fallback across providers"))  # gateway
print(which_layer("trace, log, and cost every request"))               # observability

class Gateway:
    def __init__(self, price_per_word=0.002):
        self.spend = 0.0
        self.price = price_per_word

    def _call(self, provider, prompt):
        if provider == "down":                    # simulate an outage
            raise RuntimeError("provider unavailable")
        self.spend += self.price * len(prompt.split())
        return f"[{provider}] answer to: {prompt}"

    def complete(self, prompt, providers):
        for p in providers:                        # fallback chain, in order
            try:
                return self._call(p, prompt)
            except RuntimeError:
                continue
        raise RuntimeError("all providers failed")

gw = Gateway()
print(gw.complete("reset my password please", ["down", "primary"]))
print(f"spend so far: ${gw.spend:.4f}")   # billed only for the successful call

PRICE = {"small": 0.25, "frontier": 3.00}   # $ per 1M input tokens (illustrative)

def route(difficulty):
    return "frontier" if difficulty == "hard" else "small"

def blended_cost(requests, tokens_each=1000):
    total = 0.0
    for r in requests:
        tier = route(r)
        total += PRICE[tier] * tokens_each / 1_000_000
    return round(total, 6)

easy_heavy = ["easy"] * 90 + ["hard"] * 10
all_frontier = ["hard"] * 100
print("tiered  (90% easy):", blended_cost(easy_heavy))    # much cheaper
print("frontier everything:", blended_cost(all_frontier)) # baseline

def build_or_buy(layer, sensitive_data, high_volume, team_has_ops):
    if sensitive_data and layer in ("serving", "vector store"):
        return "self-host -- sensitive data must stay in-house"
    if high_volume and team_has_ops and layer == "serving":
        return "self-host -- volume justifies running the model"
    return "buy managed -- undifferentiated layer, let a vendor run it"

print(build_or_buy("serving", sensitive_data=True,  high_volume=False, team_has_ops=False))
print(build_or_buy("serving", sensitive_data=False, high_volume=True,  team_has_ops=True))
print(build_or_buy("eval",    sensitive_data=False, high_volume=False, team_has_ops=True))

# Launch an OpenAI-compatible server (documented vLLM CLI):
#   pip install vllm
#   python -m vllm.entrypoints.openai.api_server \
#       --model meta-llama/Llama-3.1-8B-Instruct \
#       --port 8000 --max-model-len 8192

# The gateway then treats it like any other provider -- an OpenAI-shaped URL.
# (client code needs the running server; shown for shape)
from openai import OpenAI                     # pip install openai
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed-local")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "classify: card declined twice"}],
)
print(resp.choices[0].message.content)

LAYERS = ["gateway", "serving", "registry", "vector store", "eval", "observability"]

def build_or_buy(layer, sensitive, high_vol, ops):
    if sensitive and layer in ("serving", "vector store"): return "SELF-HOST"
    if high_vol and ops and layer == "serving":            return "SELF-HOST"
    return "BUY MANAGED"

def platform(sensitive, high_vol, ops):
    plan = {L: build_or_buy(L, sensitive, high_vol, ops) for L in LAYERS}
    plan["gateway"] = "SELF-HOST (thin) -- the seam that makes swaps config"
    return plan

for layer, decision in platform(sensitive=True, high_vol=True, ops=True).items():
    print(f"{layer:14s}: {decision}")

Knowledge check check yourself