MLOps & LLMOps · Chapter O2

LLM Infrastructure, Tooling & the Open-Source Stack

Between your app and a model sits a stack: gateways, serving engines, registries, eval harnesses, vector stores, observability. This chapter maps that landscape by layer and job — so you can pick tools by the problem they solve, not by hype, and know when a hosted API means you can skip a whole layer.

⏱️ ~1.5 hours🧪 2 labs🎯 Beginner→Tech-lead

Learning objectives

Map the LLM infra stack layer by layer.
Explain what a gateway, serving engine, and registry do.
Choose components for a given requirement.
Design a coherent stack as a platform owner.

▶ Runnable companionEvery code block in this lesson is also a standalone file under code/op2-infra/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

1 · The stack between app and model essential

Between your app and a model sits a stack: gateways, serving engines, registries, eval harnesses, vector stores, and observability. Each layer solves one problem; knowing what lives where is how you reason about (and debug) an LLM system.

2 · The layers essential

Layer	Job	Examples
Gateway	route/auth/rate-limit/fallback across providers	LiteLLM, portkey
Serving	run open models fast	vLLM, TGI, Ollama
Registry	version models + prompts	MLflow, HF Hub
Vector store	semantic retrieval	pgvector, Qdrant, OpenSearch
Eval	measure quality	promptfoo, custom harness
Observability	trace/log/cost	Langfuse, OTel

3 · Intermediate — the gateway pattern intermediate

A gateway is the highest-leverage layer: one API in front of many providers, with auth, rate limits, retries, fallback, and cost tracking. Your app talks to the gateway; swapping models or providers becomes config. This is the provider-agnostic client (DF4) as infrastructure.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Python · a minimal gateway with fallback + cost tracking (runs)

gateway.pyclass Gateway:
    def __init__(self):
        self.spend = 0.0
    def _call(self, provider, prompt):
        if provider == "down": raise RuntimeError("provider unavailable")
        self.spend += 0.002 * len(prompt.split())     # fake per-token cost
        return f"[{provider}] answer to: {prompt[:20]}"
    def complete(self, prompt, providers):
        for prov in providers:                          # fallback chain
            try:
                return self._call(prov, prompt)
            except RuntimeError:
                continue
        raise RuntimeError("all providers failed")

gw = Gateway()
print(gw.complete("summarize the incident report", providers=["down", "claude"]))
print("spend so far: $%.4f" % gw.spend)

[claude] answer to: summarize the incid
spend so far: $0.0080

▶ How this works

A gateway is the one door your app knocks on instead of talking to each model provider directly. This tiny class shows the two jobs a real gateway does: try providers in order until one works (fallback) and keep a running tally of what you spent (cost tracking). Swapping providers later becomes a config change, not a code change.

__init__ starts self.spend = 0.0 — a running total of money spent, kept on the gateway object so every call adds to the same tally.
_call(provider, prompt) is a pretend model call. If the provider is "down" it raises an error (simulating an outage); otherwise it adds a fake cost based on how many words the prompt has and returns a canned answer. The leading underscore is a convention meaning "internal helper — not for outside callers".
complete(prompt, providers) is the real entry point. It loops through the providers list in order; the try/except means "attempt this provider — if it throws, continue to the next one instead of crashing". That loop is the fallback chain.
If every provider in the list fails, the loop ends and the final raise reports "all providers failed" — the gateway only gives up when it has truly run out of options.
The last three lines use the gateway: it's asked to try "down" first (which fails) then "claude" (which succeeds), then prints the running spend.

What the output means: The first provider "down" is skipped after it errors, so the answer comes from [claude]. spend so far: $0.0080 is the fake cost of the 4-word prompt (4 words × $0.002).

Try this: Change the list to providers=["down", "down"] and run it — with no working provider left, you'll see the "all providers failed" error. That is exactly what a real gateway must handle when every backend is unhealthy.

4 · Advanced — serving open models advanced

To self-host, a serving engine (vLLM, TGI) runs the model with batching, paged KV-cache (IC3), and an OpenAI-compatible API. This is where throughput/latency are won or lost — the difference between a model that costs a fortune per request and one that scales. Serving sits behind the gateway so callers don't care.

5 · Professional — build vs buy per layer professional

You don't build every layer. Buy managed where it's undifferentiated (observability, a hosted vector DB), build/self-host where you have a real requirement (privacy → self-served models, cost at scale → your own serving). Decide per layer against your actual constraints, not all-or-nothing.

Python · pick build vs buy per layer (runs)

build_buy.pydef build_or_buy(layer, sensitive_data, high_volume, team_has_ops):
    if sensitive_data and layer in ("serving", "vector store"):
        return "self-host (data stays in-house)"
    if high_volume and team_has_ops and layer == "serving":
        return "self-host (cost at scale)"
    return "buy managed (undifferentiated — don't build it)"

for layer in ["serving", "vector store", "observability"]:
    print(layer, "->", build_or_buy(layer, sensitive_data=True,
                                     high_volume=True, team_has_ops=True))

serving -> self-host (data stays in-house)
vector store -> self-host (data stays in-house)
observability -> buy managed (undifferentiated — don't build it)

▶ How this works

You do not build every layer of the stack yourself. This function encodes a simple rule of thumb: self-host (run it yourself) only when you have a real reason — sensitive data or cost at high volume — and otherwise buy managed (pay a vendor) because the layer is "undifferentiated": doing it yourself wins you nothing.

The four inputs describe your situation: which layer you're deciding on, and three yes/no facts — is the data sensitive, is traffic high_volume, does your team_has_ops (people who can run infrastructure).
First rule: if data is sensitive and the layer is serving or vector store (the layers that touch your data), self-host so the data never leaves your walls.
Second rule: even without sensitive data, if you have high volume and an ops team, self-hosting serving can be cheaper at scale than paying per request.
If neither rule fires, the function falls through to return "buy managed" — the sensible default for anything that isn't a competitive advantage.
The for loop asks the same question about three layers so you can compare the answers side by side.

What the output means: With sensitive data and high volume, serving and vector store come back self-host (they touch the data), while observability is buy managed — logging/tracing is undifferentiated, so let a vendor run it.

Try this: Flip sensitive_data=False and re-run. Now vector store switches to "buy managed" because the privacy reason is gone — only serving stays self-hosted, on the cost-at-scale rule.

6 · Tech-lead — designing the platform tech-lead

A platform owner assembles a coherent stack: a gateway as the single entry point, chosen serving/vector/eval/observability layers, and clear seams so any one can be swapped. The goal is that app teams code against stable interfaces while the platform evolves underneath — that's what turns a pile of tools into a platform.

The gateway is the keystoneIf you build one layer well, make it the gateway: it decouples every app from every provider, centralizes cost/auth/observability, and makes model swaps config. Most stack flexibility flows from that one seam.

Exercise OP2.1 — Design a stack

Context: Designing a stack for sensitive data and high volume is the canonical platform exercise: it forces a build/buy call on every layer and a clear request path through the gateway.

Your task: For a system with sensitive data and high volume, choose each layer with your build-vs-buy logic, then sketch how a request flows through the gateway to self-hosted serving.

Requirements:

Make an explicit build/buy call for every layer
Name which layers you would buy managed and why
Justify self-hosting serving and the vector store for sensitive data
Sketch the request path from gateway to self-hosted serving

💡 Hint: Let the same rule drive every layer — the coherence is the point, and the gateway is what keeps the self-hosted choices swappable.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Map a need to its stack layerBeginner

Context: The LLM platform stack has six layers, each a seam with one responsibility: gateway, serving, registry, vector store, eval, observability. Placing a change in the right layer keeps the app decoupled from the plumbing.

Your task: Write a router that, given a problem statement, names the stack layer that owns it.

Requirements:

Cover all six layers (gateway, serving, registry, vector store, eval, observability)
Route provider routing/auth/rate-limit/fallback to the gateway
Route trace/log/cost concerns to observability
Return an explicit "unknown" for an unrecognised need
Demonstrate a couple of lookups

💡 Hint: A lookup table keyed by the need is enough — the point is that a provider swap belongs in the gateway, not scattered through app code.

Show solution

The six-layer table as a router (pure stdlib):

LAYER = {
    "route/auth/rate-limit/fallback across providers": "gateway",
    "run an open model fast on our GPUs":              "serving",
    "version models and prompts":                      "registry",
    "semantic retrieval over documents":               "vector store",
    "measure answer quality":                          "eval",
    "trace, log, and cost every request":              "observability",
}
def which_layer(need):
    return LAYER.get(need, "unknown -- restate the need")

print(which_layer("route/auth/rate-limit/fallback across providers"))  # gateway
print(which_layer("trace, log, and cost every request"))               # observability

Each layer is a seam with a single responsibility. Placing a change in the right layer — a provider swap belongs in the gateway, not the app — keeps the app decoupled from the plumbing underneath it.

Exercise 2 · A gateway with provider fallback + cost trackingIntermediate

Context: The gateway is the keystone layer: one API in front of many providers, with fallback and a running cost total. When a provider dies, traffic slides to the next without touching app code.

Your task: Implement a minimal gateway that tries providers in order, accumulates a per-token cost, and returns the first successful response.

Requirements:

Expose one complete(prompt, providers) method the app calls
Try providers in order, falling back on failure to the next
Simulate an outage so the fallback path actually runs
Accumulate spend only for the successful call
Raise cleanly if every provider fails
Run it end to end offline with fake providers

💡 Hint: Wrap each provider call in a try/except and continue on failure — fallback, retries, and cost accounting all belong inside this one class.

Show solution

The lesson's gateway, self-contained and runnable (fake providers, no network):

class Gateway:
    def __init__(self, price_per_word=0.002):
        self.spend = 0.0
        self.price = price_per_word

    def _call(self, provider, prompt):
        if provider == "down":                    # simulate an outage
            raise RuntimeError("provider unavailable")
        self.spend += self.price * len(prompt.split())
        return f"[{provider}] answer to: {prompt}"

    def complete(self, prompt, providers):
        for p in providers:                        # fallback chain, in order
            try:
                return self._call(p, prompt)
            except RuntimeError:
                continue
        raise RuntimeError("all providers failed")

gw = Gateway()
print(gw.complete("reset my password please", ["down", "primary"]))
print(f"spend so far: ${gw.spend:.4f}")   # billed only for the successful call

The gateway turns provider choice into config: the app calls one method, and fallback, retries, and cost accounting live in one place. When a provider dies, traffic slides to the next without touching app code.

Exercise 3 · Route by model tier to control costAdvanced

Context: Not every request needs the frontier model. Tier routing — cheap model for easy requests, frontier for hard ones — is the highest-leverage cost control the gateway enables, and it is invisible to the app.

Your task: Extend the gateway idea with tier routing and show the blended cost falling as more of the traffic is easy.

Requirements:

Route easy requests to a small/cheap tier and hard ones to the frontier tier
Use realistic per-tier pricing (illustrative is fine)
Compute the blended cost over a mixed request stream
Compare a mostly-easy stream against sending everything to the frontier model
Show the blended cost dropping as the easy fraction rises

💡 Hint: The savings come from the 90% easy tail going to the cheap tier — the frontier model only handles the hard minority.

Show solution

Tier routing modeled offline — the cost lever behind 'route easy to cheap':

PRICE = {"small": 0.25, "frontier": 3.00}   # $ per 1M input tokens (illustrative)

def route(difficulty):
    return "frontier" if difficulty == "hard" else "small"

def blended_cost(requests, tokens_each=1000):
    total = 0.0
    for r in requests:
        tier = route(r)
        total += PRICE[tier] * tokens_each / 1_000_000
    return round(total, 6)

easy_heavy = ["easy"] * 90 + ["hard"] * 10
all_frontier = ["hard"] * 100
print("tiered  (90% easy):", blended_cost(easy_heavy))    # much cheaper
print("frontier everything:", blended_cost(all_frontier)) # baseline

Sending the 90% of easy requests to a small model collapses the bill while the frontier model handles only the hard tail. Tier routing is the highest-leverage cost control the gateway enables — and it is invisible to the app.

Exercise 4 · Build-vs-buy decision per layerExpert

Context: Self-hosting is only worth it with a real reason. The lesson's rule: self-host when data is sensitive and the layer is serving or vector store, or when volume is high and you have the ops for serving; otherwise buy managed.

Your task: Encode the build-vs-buy decision as a function over layer, data sensitivity, volume, and whether the team has ops.

Requirements:

Self-host serving or vector store when data is sensitive
Self-host serving when volume is high and the team has ops
Default everything else to buying managed
Return a decision with a short justification
Demonstrate the three branches with different inputs

💡 Hint: Default to buying for layers that are not your differentiator — self-host only when privacy or scale-economics force it.

Show solution

The build-vs-buy decision as the lesson states it (pure logic):

def build_or_buy(layer, sensitive_data, high_volume, team_has_ops):
    if sensitive_data and layer in ("serving", "vector store"):
        return "self-host -- sensitive data must stay in-house"
    if high_volume and team_has_ops and layer == "serving":
        return "self-host -- volume justifies running the model"
    return "buy managed -- undifferentiated layer, let a vendor run it"

print(build_or_buy("serving", sensitive_data=True,  high_volume=False, team_has_ops=False))
print(build_or_buy("serving", sensitive_data=False, high_volume=True,  team_has_ops=True))
print(build_or_buy("eval",    sensitive_data=False, high_volume=False, team_has_ops=True))

Default to buying managed for layers that are not your differentiator; self-host only when privacy or scale-economics force it. The rule keeps ops effort on the layers that actually earn it instead of on running plumbing a vendor does better.

Exercise 5 · vLLM serving config as infrastructure (needs creds/GPU)Professional

Context: When you self-host, the serving engine is config, not code. Because vLLM exposes an OpenAI-compatible API, the self-hosted model plugs into the same gateway as any managed provider — the serving choice becomes a base-URL.

Your task: Show a correct, minimal way to stand up an OpenAI-compatible endpoint with vLLM behind the gateway, labelled as needing a GPU host.

Requirements:

Show the documented vLLM server launch command with a real open model
Point an OpenAI-compatible client at the local base_url
Make clear the app treats it exactly like any other provider
Label the block as needing a GPU host + model weights (not runnable inline)
Emphasise the app never learns whether the model is yours or a vendor's

💡 Hint: The whole trick is the OpenAI-shaped URL — the gateway sees one more provider, not a self-hosted special case.

Show solution

Serving an open model — needs a GPU host + the model weights (config-as-infra, not runnable here):

# Launch an OpenAI-compatible server (documented vLLM CLI):
#   pip install vllm
#   python -m vllm.entrypoints.openai.api_server \
#       --model meta-llama/Llama-3.1-8B-Instruct \
#       --port 8000 --max-model-len 8192

# The gateway then treats it like any other provider -- an OpenAI-shaped URL.
# (client code needs the running server; shown for shape)
from openai import OpenAI                     # pip install openai
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed-local")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "classify: card declined twice"}],
)
print(resp.choices[0].message.content)

Because vLLM exposes an OpenAI-compatible API, the self-hosted model plugs into the same gateway as any managed provider — the serving choice becomes a base-URL in config. The app never learns whether the model is yours or a vendor's.

Exercise 6 · Design a coherent platform for sensitive + high volumeIndustry scenario

Context: As platform lead you design the stack for a team with sensitive data AND high volume. A platform is coherent when every layer's build/buy call follows one rule, not six ad-hoc opinions.

Your task: Compose the per-layer build/buy decisions into one platform blueprint, keep the gateway as the swappable seam, and print it with justification.

Requirements:

Apply the single build-vs-buy rule across all six layers
Self-host serving and vector store to keep sensitive data in-house
Buy the undifferentiated layers
Own a thin gateway so every other choice stays config-swappable
Print the resulting per-layer blueprint

💡 Hint: Reuse the build-vs-buy rule from the Expert rung across every layer, then override the gateway to a thin self-hosted seam.

Show solution

Compose the per-layer decisions into one blueprint (pure logic):

LAYERS = ["gateway", "serving", "registry", "vector store", "eval", "observability"]

def build_or_buy(layer, sensitive, high_vol, ops):
    if sensitive and layer in ("serving", "vector store"): return "SELF-HOST"
    if high_vol and ops and layer == "serving":            return "SELF-HOST"
    return "BUY MANAGED"

def platform(sensitive, high_vol, ops):
    plan = {L: build_or_buy(L, sensitive, high_vol, ops) for L in LAYERS}
    plan["gateway"] = "SELF-HOST (thin) -- the seam that makes swaps config"
    return plan

for layer, decision in platform(sensitive=True, high_vol=True, ops=True).items():
    print(f"{layer:14s}: {decision}")

The blueprint self-hosts serving and the vector store to keep sensitive data in-house, buys the undifferentiated layers, and owns a thin gateway so every other choice stays swappable. A platform is coherent when each layer's build/buy call follows one rule, not six ad-hoc opinions.

✓ Checkpoint — you can move on when you can…

Map the LLM infra stack layers.
Explain the gateway/serving/registry roles.
Decide build vs buy per layer.
Design a coherent, swappable platform.

Knowledge check check yourself

✓ Knowledge check

Why does the lesson call the gateway the highest-leverage / keystone layer of the LLM stack?

Show answer

One API sits in front of many providers with auth, rate limits, retries, fallback, and cost tracking, so it decouples every app from every provider, centralizes cost/auth/observability, and makes model or provider swaps a config change rather than a code change.

✓ Knowledge check

What build-vs-buy rule does the lesson apply per layer, and which layers does it push toward self-hosting when data is sensitive?

Show answer

Buy managed for undifferentiated layers (e.g. observability, a hosted vector DB); build/self-host only where you have a real requirement. Sensitive data pushes the layers that touch data — serving and the vector store — toward self-hosting; high volume plus an ops team also justifies self-hosting serving for cost at scale.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · The stack between app and model essential

2 · The layers essential

3 · Intermediate — the gateway pattern intermediate

4 · Advanced — serving open models advanced

5 · Professional — build vs buy per layer professional

6 · Tech-lead — designing the platform tech-lead

Exercise OP2.1 — Design a stack

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in MLOps & LLMOps