Observability & SLOs

1 · The four signals — what each one answers

Observability is not one thing. Four kinds of telemetry answer four different questions, and a mature setup collects all four and correlates them. Metrics tell you that something is wrong; traces and logs tell you where and why.

2 · Prometheus: scrape, store, query

Prometheus is a pull-based time-series database: it periodically scrapes an HTTP /metrics endpoint on each target, stores samples, and answers PromQL queries. In Kubernetes the Prometheus Operator (shipped in the kube-prometheus-stack Helm chart) lets you declare scrape targets as CRDs — a ServiceMonitor — instead of editing raw config.

install.sh# Verify chart name/version and values against the current kube-prometheus-stack docs.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kps prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace
# This bundles Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics.

Now point Prometheus at your app. Your app must expose /metrics (most languages have a Prometheus client). A ServiceMonitor selects the Service by label and tells Prometheus which port/path to scrape:

servicemonitor.yamlapiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-api
  namespace: monitoring
  labels:
    release: kps          # must match the Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: llm-api        # selects the Service to scrape
  namespaceSelector:
    matchNames: [default]
  endpoints:
    - port: http          # the named Service port
      path: /metrics
      interval: 15s

PromQL basics. Counters only go up, so you almost always wrap them in rate() over a window. Three queries you will write constantly:

queries.promql# 1) Requests per second, per route, averaged over 5m:
sum by (route) (rate(http_requests_total[5m]))

# 2) Error ratio (5xx as a fraction of all requests) — an SLI:
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))

# 3) p95 latency from a histogram (buckets recorded by the client lib):
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

route         value
/chat         42.7
/embed        11.3
# error ratio -> 0.004  (0.4%)
# p95 -> 0.82  (820 ms)

3 · kube-state-metrics, node metrics & Grafana

Two different metric sources are easy to confuse. node-exporter / cAdvisor report resource usage — CPU, memory, disk of nodes and containers. kube-state-metrics reports object state from the API server — how many replicas a Deployment wants vs has ready, whether a Pod is Pending, how many restarts a container has. You need both: usage tells you the machine is hot; state tells you the desired vs actual gap.

Grafana turns PromQL into dashboards. The stack ships useful defaults; the skill is building a focused service dashboard around your SLIs (rate, errors, latency — the 'RED' method) rather than 200 panels no one reads. Provision dashboards as code (a ConfigMap the Grafana sidecar picks up) so they live in Git.

4 · Logs with Loki, traces with OpenTelemetry

Loki is 'Prometheus for logs': it indexes only labels (app, namespace, pod) — not the full log text — which makes it cheap to run. You query with LogQL, which looks like PromQL. An agent (Promtail / Grafana Alloy) ships pod stdout to Loki, and because it uses the same labels as your metrics, you can jump from a spiking error graph straight to the matching logs in Grafana.

logs.logql# Label filter (indexed, fast) then a line filter (scanned):
{app="llm-api", namespace="default"} |= "error" | json | status >= 500

# Rate of error lines per pod (LogQL metric query):
sum by (pod) (rate({app="llm-api"} |= "error" [5m]))

Distributed tracing answers the question metrics can't: for this slow request, which service or call ate the time? A request is assigned a trace ID; each service records spans (start/end + attributes) that nest into a waterfall. OpenTelemetry (OTel) is the vendor-neutral standard — instrument once with the OTel SDK, export via the OTel Collector to any backend (Tempo, Jaeger). For a RAG/agent app this is gold: you see retrieve → rerank → LLM-call as separate spans and learn the LLM call is 90% of latency.

5 · SLIs, SLOs, error budgets & burn rate

Now the part that decides what you page on. An SLI (indicator) is a measured ratio of good events to total — e.g. the fraction of requests served under 500 ms without a 5xx. An SLO (objective) is a target for that SLI over a window — e.g. 99.9% over 28 days. The gap between 100% and the SLO is your error budget: at 99.9%, you may 'spend' 0.1% of requests failing before you've broken your promise. This reframes reliability from 'never fail' to 'fail within budget' — and gives you a data-driven brake on shipping.

The error budget and burn rate are just arithmetic — here it is as offline-runnable Python (no cluster needed), the same math Prometheus alert rules encode:

error_budget.pySLO = 0.999                      # 99.9% success objective
WINDOW_DAYS = 28
budget = 1 - SLO                 # 0.001 -> 0.1% of requests may fail

def burn_rate(bad, total):
    """How many times faster than the 'even' pace we are spending budget."""
    if total == 0:
        return 0.0
    error_ratio = bad / total
    return error_ratio / budget  # 1.0 = exactly on pace to exhaust budget over the window

def budget_remaining(bad_28d, total_28d):
    spent = (bad_28d / total_28d) / budget if total_28d else 0.0
    return max(0.0, 1 - spent)   # fraction of the error budget still available

# A fast-burn page: 2% errors over the last 5m is 20x the budget pace.
print(round(burn_rate(bad=20, total=1000), 1))          # -> 20.0  (page now)
print(round(budget_remaining(bad_28d=1200, total_28d=2_000_000) * 100, 1))  # -> 40.0% left

20.0
40.0

6 · Alerting that pages on symptoms, not causes

Prometheus records alert conditions as rules; Alertmanager handles them — grouping related alerts into one notification, routing by severity/team, silencing during maintenance, and inhibiting (suppress the 100 downstream alerts when the one root-cause alert fires). The philosophy that keeps on-call sane: alert on symptoms your users feel (SLO burn, high error ratio, high latency) — not on every cause (one node at 80% CPU is not a page).

prometheusrule.yamlapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-api-slo
  labels: { release: kps }
spec:
  groups:
    - name: slo-burn
      rules:
        - alert: FastErrorBudgetBurn
          # 5xx ratio > 14.4x the 0.1% budget, confirmed over 1h and 5m windows.
          expr: |
            (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4*0.001)
            and
            (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4*0.001)
          for: 2m
          labels: { severity: page }
          annotations:
            summary: "Fast error-budget burn on llm-api"

alertmanager.yamlroute:
  receiver: default-ticket
  group_by: [alertname, namespace]
  routes:
    - matchers: [ 'severity = page' ]
      receiver: pagerduty-oncall
inhibit_rules:
  - source_matchers: [ 'severity = page' ]
    target_matchers: [ 'severity = ticket' ]
    equal: [namespace]        # if we're already paging, don't also file tickets for the same ns
receivers:
  - name: default-ticket
  - name: pagerduty-oncall
    # pagerduty_configs / webhook_configs go here — verify fields vs current docs

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-api
  labels: { release: kps }   # matches Prometheus serviceMonitorSelector
spec:
  selector: { matchLabels: { app: llm-api } }   # matches the Service
  endpoints: [ { port: http, path: /metrics, interval: 15s } ]

# Rate per route
sum by (route) (rate(http_requests_total[5m]))

# Error ratio (an SLI)
sum(rate(http_requests_total{status=~"5.."}[5m]))
 /
sum(rate(http_requests_total[5m]))

# p95 latency (an SLI)
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# Fast burn -> PAGE: 14.4x budget, confirmed on 1h AND 5m windows (~2 days to exhaust)
err_ratio_1h > 14.4*0.001 and err_ratio_5m > 14.4*0.001

# Slow burn -> TICKET: ~3x budget on 6h AND 30m windows (a slow leak)
err_ratio_6h > 3*0.001 and err_ratio_30m > 3*0.001

route: { group_by: [alertname, namespace], routes:
  [ { matchers: ['severity=page'], receiver: pagerduty } ] }
inhibit_rules:
  - source_matchers: ['severity=page']
    target_matchers: ['severity=ticket']
    equal: [namespace]