Observability & SLOs

1 · The four signals — what each one answers

Picture the 3 a.m. page. Your phone buzzes: "checkout is slow." You SSH into a box and… now what? If all you have is a login shell and a hunch, you're a detective who arrived after the crime with no camera footage, no witnesses, and no fingerprints. You'll spend an hour guessing. The engineers who fix it in five minutes aren't smarter — they instrumented the system beforehand so the evidence was already being recorded. That is the entire promise of observability: turn a black box you poke at into a system that tells you what it's doing.

Here's the mental model that makes the rest of this lesson click. Think of your service as a patient in a hospital. Metrics are the vital-signs monitor — heart rate, blood pressure, updated every few seconds, cheap to keep running, and the thing that sets off the alarm. Logs are the detailed chart notes: "14:02 — patient reported chest pain" — a rich, per-event record you read after the alarm to learn exactly what happened. Traces are the surgeon following the problem through the body, timing each step to find where the delay actually is. And events are the hospital's own admin log — "moved to room 4, scheduled for surgery" — Kubernetes narrating what it decided to do (schedule, evict, OOM-kill). Four instruments, four questions, one patient.

The common beginner mistake is to reach for the wrong instrument. People try to build a dashboard out of logs (expensive and slow — like taking someone's pulse by reading their diary), or they try to debug a single failing request from metrics alone (impossible — a heart-rate average can't tell you what one patient said). Metrics answer "is something wrong, and roughly how bad?" — they're aggregated, so they can't point at the one request that broke. Logs and traces answer "what and where exactly?" for a specific request, but they're too voluminous and costly to alarm on continuously. The skill isn't collecting more of everything; it's knowing which signal answers which question, and — the part most setups miss — wiring them so you can jump between them. A metric spikes → you pivot to the trace for a slow request → the trace hands you a shared ID → you land on the exact log lines. That hand-off is what separates a pile of dashboards from real observability.

Observability is not one thing. Four kinds of telemetry answer four different questions, and a mature setup collects all four and correlates them. Metrics tell you that something is wrong; traces and logs tell you where and why.

2 · Prometheus: scrape, store, query

Most monitoring tools you've met push their data somewhere — your app calls a "send this metric" API and hopes it lands. Prometheus flips that around, and the flip is the single most important thing to understand about it. Instead of every app shouting its numbers into a central collector, Prometheus walks up to each app and reads the numbers off it, on its own schedule. Imagine a meter reader who strolls down the street once a minute and jots down the number showing on each house's electricity meter. The houses don't phone anyone; they just keep a running total on a dial, and the reader records it. That is pull-based scraping, and it's why Prometheus scales calmly: the central server controls the pace, a dead app simply shows up as "target down" instead of silently stopping its pushes, and there's no queue to overflow when things get busy.

The "dial" each app exposes is a plain HTTP endpoint at /metrics that prints current values as text — you can literally curl it and read it. The counterpart to grasp: because Prometheus only records a snapshot each scrape, most of the numbers you expose are counters that only ever go up (total requests since start, total errors since start). A raw counter is nearly useless on its own — "4,912,338 requests total" tells you nothing about now. The magic happens in the query language, PromQL, where rate() looks at two snapshots and computes the slope between them: "requests are climbing by 42 per second." Nearly every useful query is a rate() over a counter, and forgetting the rate() — graphing a bare counter and wondering why it's a straight line marching to the moon — is the rookie mistake everyone makes once.

Where Kubernetes comes in. On a cluster, pods come and go constantly, so hand-editing a static list of scrape targets is hopeless — the list is stale the moment you save it. The Prometheus Operator (bundled in the kube-prometheus-stack Helm chart) solves this by letting you declare intent as a Kubernetes object instead of config: a ServiceMonitor says "scrape any Service with these labels, on this port, at this path," and the Operator keeps Prometheus's target list in sync as pods churn. You describe the rule; the Operator does the bookkeeping. Prometheus itself is, at heart, a pull-based time-series database — it periodically scrapes an HTTP /metrics endpoint on each target, stores the samples, and answers PromQL queries. In Kubernetes the Prometheus Operator lets you declare scrape targets as CRDs — a ServiceMonitor — instead of editing raw config.

install.sh# Verify chart name/version and values against the current kube-prometheus-stack docs.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kps prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace
# This bundles Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics.

Now point Prometheus at your app. Your app must expose /metrics (most languages have a Prometheus client). A ServiceMonitor selects the Service by label and tells Prometheus which port/path to scrape:

servicemonitor.yamlapiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-api
  namespace: monitoring
  labels:
    release: kps          # must match the Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: llm-api        # selects the Service to scrape
  namespaceSelector:
    matchNames: [default]
  endpoints:
    - port: http          # the named Service port
      path: /metrics
      interval: 15s

PromQL basics. Counters only go up, so you almost always wrap them in rate() over a window. Three queries you will write constantly:

queries.promql# 1) Requests per second, per route, averaged over 5m:
sum by (route) (rate(http_requests_total[5m]))

# 2) Error ratio (5xx as a fraction of all requests) — an SLI:
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))

# 3) p95 latency from a histogram (buckets recorded by the client lib):
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

route         value
/chat         42.7
/embed        11.3
# error ratio -> 0.004  (0.4%)
# p95 -> 0.82  (820 ms)

3 · kube-state-metrics, node metrics & Grafana

Here's a failure that fools people for hours. Your app is down — users are getting errors — but every graph you open is green. CPU is calm, memory is fine, the nodes are barely breathing. So what's wrong? The trap is that you're only watching how hard the machines are working, and completely missing what Kubernetes thinks the state of the world is. Your Deployment wants 5 replicas; only 1 is actually ready; the other 4 are stuck Pending because no node has room. The surviving pod is under-loaded (hence low CPU) yet drowning in traffic it can't serve. No amount of staring at CPU graphs reveals this, because CPU is the wrong instrument for the question.

That's why there are two completely different families of metric, and confusing them is one of the most common gaps in a monitoring setup. Think of a delivery company. node-exporter and cAdvisor are the fuel gauges and engine-temperature sensors on each truck — resource usage: how much CPU, memory, and disk the nodes and containers are actually burning. kube-state-metrics is the dispatch office whiteboard — object state read straight from the Kubernetes API server: how many trucks you ordered out (desired replicas) versus how many are actually on the road (ready replicas), which ones are stuck in the depot (Pending), and which keep breaking down and returning (restart counts / CrashLoopBackOff). One family tells you the engines are hot; the other tells you the gap between what you asked for and what you got — and that gap is where most outages live.

You genuinely need both, and the reason is that Kubernetes is a declarative system: you tell it your desired state and it continuously tries to make reality match. When reality can't match — image won't pull, no room to schedule, liveness probe failing — the trucks-ordered-vs-on-road gap is your earliest and clearest signal, long before it shows up as user pain. Once you've collected all these numbers, though, a raw pile of time series helps no one at 3 a.m. That's the job of Grafana: it turns PromQL into dashboards you can actually read. The temptation — and the mistake — is to build a wall of 200 panels that looks impressive and tells you nothing. A useful dashboard is focused: it leads with the handful of signals that map to user pain and relegates everything else to drill-down. The next sections give you two naming conventions (RED and USE) so you build the focused kind on purpose.

Grafana turns PromQL into dashboards. The stack ships useful defaults; the skill is building a focused service dashboard around your SLIs (rate, errors, latency — the 'RED' method) rather than 200 panels no one reads. Provision dashboards as code (a ConfigMap the Grafana sidecar picks up) so they live in Git.

4 · Logs with Loki, traces with OpenTelemetry

The old way of storing logs bankrupts you slowly. Traditional log systems index every word of every line so you can search on anything — which sounds great until you're paying to build a full-text search engine over terabytes of "GET /health 200" that nobody will ever search for. Teams routinely spend more on their log stack than on the app it watches. Loki is the deliberate cheapskate's answer, and its one clever idea is worth internalizing: don't index the log text at all — index only a few labels (app, namespace, pod), the same way Prometheus labels a metric. Picture a filing cabinet. Full-text logging photocopies and cross-references every sentence in every document; Loki just writes a label on each folder — "llm-api / prod" — and drops the papers in unsorted. To find something you first grab the right folder by its label (instant, indexed), then skim the pages inside (a quick scan). You give up "search everything everywhere" and get a log store cheap enough to actually keep. You query it with LogQL, which is deliberately shaped to look like PromQL, and because Loki reuses the same labels as your metrics, you can click from a spiking error graph straight to the matching logs — no separate mental map.

But logs and metrics share a blind spot in a world of many small services: neither can tell you where a slow request spent its time. Say a chat request takes 4 seconds. Was it the retriever? The reranker? The LLM call? The database? Metrics say "p95 is 4s" (unhelpfully aggregate); logs say "request started… request finished" in each service separately, with no thread connecting them. This is what distributed tracing fixes, and the analogy is a parcel's tracking history. Every request gets a trace ID — one tracking number stamped on it at the front door and carried through every service it touches. Each service records a span: a timestamped "I received it here, I handed it off there," with attributes attached. Assemble the spans by their shared trace ID and you get a waterfall — a Gantt chart of the request's life — where the longest bar is your culprit, obvious at a glance.

OpenTelemetry (OTel) is the piece that makes this portable rather than a vendor lock-in trap. The mistake teams made for years was instrumenting their code against one tracing vendor's SDK, then rewriting everything when they switched. OTel is the vendor-neutral standard: you instrument once with the OTel SDK, ship spans to the OTel Collector, and the Collector fans them out to whatever backend you choose (Tempo, Jaeger) — swap the backend, keep your code. For a RAG or agent app this is genuinely revelatory: you finally see retrieve → rerank → LLM-call as separate spans and discover, concretely, that the LLM call is 90% of your latency and the retrieval you spent a week optimizing was never the problem. Now the plain-language recap. Loki is 'Prometheus for logs': it indexes only labels (app, namespace, pod) — not the full log text — which makes it cheap to run; you query with LogQL, which looks like PromQL, and an agent (Promtail / Grafana Alloy) ships pod stdout to Loki using the same labels as your metrics. Distributed tracing assigns each request a trace ID; each service records spans that nest into a waterfall; OpenTelemetry is the vendor-neutral way to emit them and export via the Collector to any backend.

logs.logql# Label filter (indexed, fast) then a line filter (scanned):
{app="llm-api", namespace="default"} |= "error" | json | status >= 500

# Rate of error lines per pod (LogQL metric query):
sum by (pod) (rate({app="llm-api"} |= "error" [5m]))

Distributed tracing answers the question metrics can't: for this slow request, which service or call ate the time? A request is assigned a trace ID; each service records spans (start/end + attributes) that nest into a waterfall. OpenTelemetry (OTel) is the vendor-neutral standard — instrument once with the OTel SDK, export via the OTel Collector to any backend (Tempo, Jaeger). For a RAG/agent app this is gold: you see retrieve → rerank → LLM-call as separate spans and learn the LLM call is 90% of latency.

5 · SLIs, SLOs, error budgets & burn rate

"How reliable should our service be?" has a wrong answer that everyone gives: "100%." It sounds responsible and it's a trap. Chasing 100% is infinitely expensive — every extra nine of reliability costs multiples more than the last — and it's also pointless, because the user's home wifi, their phone, and the public internet between you and them already fail more often than your service ever will. Aiming for perfect uptime means you can never ship a change, because every deploy carries risk. The uncomfortable truth SRE culture is built on: the right reliability target is not 100%, and choosing it is a business decision, not a virtue.

The vocabulary here is precise, so anchor each term to the previous one — they stack. An SLI (Service Level Indicator) is just a measurement: a ratio of good events to total, like "fraction of requests served under 500 ms without a 5xx." It's a thermometer reading, no judgment attached. An SLO (Service Level Objective) adds the judgment — a target for that SLI over a window: "99.9% of requests, measured over 28 days." (An SLA, which you'll hear next to these, is the same target with money attached — a contractual refund if you miss it; the SLO is the internal goal you set tighter than the SLA so you never actually breach the contract.) Then comes the idea that changes how teams behave: subtract the SLO from 100% and you get the error budget — 1 − SLO. At 99.9%, that budget is 0.1% of requests you are allowed to fail before you've broken your promise.

Reframing failure as a budget you spend rather than a sin you avoid is the whole point, and the mistake is treating the budget as leftover slack rather than a tool. Think of it like a monthly data cap. If you've barely used any, you can stream freely — ship risky features fast, run bold experiments, because you have reliability to spare. If you've nearly blown through it, you throttle yourself — freeze risky changes, spend the sprint on hardening — because one more incident breaks the promise to users. That's what makes the error budget a data-driven brake on shipping: it turns "should we deploy on Friday?" from a gut argument into arithmetic. And because a budget can be spent slowly or catastrophically fast, the last term you need is burn rate — how quickly you're draining the budget compared to the "even" pace that would use it up exactly at the end of the window. A burn rate of 1× means you'll finish the budget right on schedule; 14.4× means a two-day budget is gone in a couple of hours, which is the difference between "file a ticket" and "wake someone up." Now the crisp definitions to keep. An SLI is a measured ratio of good events to total; an SLO is a target for that SLI over a window; the gap between 100% and the SLO is your error budget (at 99.9%, you may 'spend' 0.1% of requests failing); this reframes reliability from 'never fail' to 'fail within budget' and gives you a data-driven brake on shipping.

The error budget and burn rate are just arithmetic — here it is as offline-runnable Python (no cluster needed), the same math Prometheus alert rules encode:

error_budget.pySLO = 0.999                      # 99.9% success objective
WINDOW_DAYS = 28
budget = 1 - SLO                 # 0.001 -> 0.1% of requests may fail

def burn_rate(bad, total):
    """How many times faster than the 'even' pace we are spending budget."""
    if total == 0:
        return 0.0
    error_ratio = bad / total
    return error_ratio / budget  # 1.0 = exactly on pace to exhaust budget over the window

def budget_remaining(bad_28d, total_28d):
    spent = (bad_28d / total_28d) / budget if total_28d else 0.0
    return max(0.0, 1 - spent)   # fraction of the error budget still available

# A fast-burn page: 2% errors over the last 5m is 20x the budget pace.
print(round(burn_rate(bad=20, total=1000), 1))          # -> 20.0  (page now)
print(round(budget_remaining(bad_28d=1200, total_28d=2_000_000) * 100, 1))  # -> 40.0% left

20.0
40.0

6 · Alerting that pages on symptoms, not causes

There is a specific way that good monitoring destroys a team, and it's worth naming out loud. You wire up alerts on everything — CPU, memory, disk, restarts, queue depth — because more visibility feels safer. Then one bad deploy trips sixty of them at once, the pager goes off twelve times in an hour, and by month three your on-call engineers have muted the channel out of pure survival. The night the real outage arrives, the page that matters is buried under forty that don't, and nobody's looking. This is alert fatigue, and it kills more incident responses than any technical failure. It's the boy who cried wolf, encoded in YAML — and the cure is not more alerts or better alerts, it's far fewer, far more meaningful ones.

The governing principle is a single distinction: alert on symptoms, not causes. A symptom is something a user feels — requests failing, pages loading slowly, the SLO error budget burning down. A cause is a mechanical detail underneath — a node at 80% CPU, a pod that restarted, a disk 70% full. The reason to page on symptoms is subtle but decisive: causes are simultaneously too noisy and too quiet. Too noisy, because a node can happily run at 85% CPU for weeks while every user request succeeds — paging on that wakes someone for a non-problem. Too quiet, because your service can be completely broken (a dependency timing out at trivial CPU) while every resource gauge stays green. Symptoms track the thing you actually promised users, so they fire when it matters and stay silent when it doesn't. Causes still belong on dashboards — you consult them after a symptom page, to find the "why" — they just don't earn the right to wake a human.

Even paging only on symptoms, one incident can still produce a flood, which is where the two tools of this section come in. Prometheus records the alert conditions as rules — the "if this, then alert" logic — but it deliberately doesn't decide who gets told or how. That's the job of Alertmanager, the switchboard sitting between raw firing alerts and human beings. It does four things worth knowing by name: grouping (thirty pods failing the same way become one notification, not thirty), routing (a severity=page alert goes to PagerDuty and the on-call phone; a severity=ticket alert goes to a work queue), silencing (mute expected noise during a planned maintenance window), and inhibiting — the cleverest one — where a firing root-cause alert suppresses the hundred downstream alerts it obviously caused ("the whole region is down" mutes "service A can't reach service B"). The philosophy that keeps a rotation sane, stated plainly: alert on the symptoms your users feel — SLO burn, high error ratio, high latency — and never on every cause; one node at 80% CPU is not a page.

prometheusrule.yamlapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-api-slo
  labels: { release: kps }
spec:
  groups:
    - name: slo-burn
      rules:
        - alert: FastErrorBudgetBurn
          # 5xx ratio > 14.4x the 0.1% budget, confirmed over 1h and 5m windows.
          expr: |
            (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4*0.001)
            and
            (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4*0.001)
          for: 2m
          labels: { severity: page }
          annotations:
            summary: "Fast error-budget burn on llm-api"

alertmanager.yamlroute:
  receiver: default-ticket
  group_by: [alertname, namespace]
  routes:
    - matchers: [ 'severity = page' ]
      receiver: pagerduty-oncall
inhibit_rules:
  - source_matchers: [ 'severity = page' ]
    target_matchers: [ 'severity = ticket' ]
    equal: [namespace]        # if we're already paging, don't also file tickets for the same ns
receivers:
  - name: default-ticket
  - name: pagerduty-oncall
    # pagerduty_configs / webhook_configs go here — verify fields vs current docs

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-api
  labels: { release: kps }   # matches Prometheus serviceMonitorSelector
spec:
  selector: { matchLabels: { app: llm-api } }   # matches the Service
  endpoints: [ { port: http, path: /metrics, interval: 15s } ]

# Rate per route
sum by (route) (rate(http_requests_total[5m]))

# Error ratio (an SLI)
sum(rate(http_requests_total{status=~"5.."}[5m]))
 /
sum(rate(http_requests_total[5m]))

# p95 latency (an SLI)
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# Fast burn -> PAGE: 14.4x budget, confirmed on 1h AND 5m windows (~2 days to exhaust)
err_ratio_1h > 14.4*0.001 and err_ratio_5m > 14.4*0.001

# Slow burn -> TICKET: ~3x budget on 6h AND 30m windows (a slow leak)
err_ratio_6h > 3*0.001 and err_ratio_30m > 3*0.001

route: { group_by: [alertname, namespace], routes:
  [ { matchers: ['severity=page'], receiver: pagerduty } ] }
inhibit_rules:
  - source_matchers: ['severity=page']
    target_matchers: ['severity=ticket']
    equal: [namespace]