Observability & cost

You can't operate what you can't see advanced

Start with the story every team eventually lives. You ship an AI feature on Friday. It works in the demo, so you move on. The following Wednesday, finance forwards you a bill that is 8× what you projected, a customer complains the assistant "went crazy" last Tuesday at 3pm, and your CTO asks a simple question you cannot answer: which caller spent that money, and what exactly did the model say? You have no logs, no alarm history, no cost breakdown. The system was running the whole time — you just had no window into it. That blind week is what this chapter exists to prevent.

The mental model to hold is a car dashboard. An engine can be running perfectly and still be seconds from disaster if the temperature gauge is buried, the fuel light is disconnected, and there's no odometer. Observability is the dashboard for your AI system: gauges you can read at a glance (latency, error rate), a warning light that goes off before you're stranded (alarms, budgets), and a trip log of where you've been (the invocation log). None of it makes the engine faster — it makes the engine operable, which is a different and equally essential property.

The precise idea is this: observability is the ability to answer questions about your system's behaviour from the outputs it emits — without shipping new code to investigate. For a production AI system those outputs cluster into four signals worth memorising, because a healthy dashboard shows all four: latency (are calls fast enough?), errors (are calls failing or being throttled?), cost (are we spending what we expected?), and quality (are the answers still good?). Miss any one and you have a blind spot exactly where an incident will hide — cost runaways sneak past a latency-only dashboard; a model quietly degrading in quality never trips an error alarm.

The common mistake is treating observability as something you "add later," after the feature works. But you can't reconstruct history you never recorded — the invocation log only captures calls made after you enable it, an alarm can only page on a metric it was already watching, and a budget only alerts once you've set the threshold. Everything in this chapter is a decision made before the incident, so that when the incident comes you're reading gauges instead of guessing. The rest of the chapter walks the four signals in AWS primitives: logging (the audit trail), metrics + alarms + budgets (the gauges and warning lights), and governance + a kill switch (who's allowed to drive, and how you stop the car).

This is the AWS-specific version of your O4 chapter. A live AI system needs the four signals — latency, errors, cost, quality — plus an audit trail of every model call. AWS gives you the primitives; you wire them.

Model-invocation logging advanced

Picture the aftermath of that "the assistant went crazy" complaint. A customer swears the bot gave dangerous advice at 3pm Tuesday. You open your application logs and find… that a Bedrock call happened, returned HTTP 200, and took 900ms. That's it. You have the envelope — a call occurred — but not the letter inside it: what prompt went in, what text came back. You cannot reproduce it, can't confirm or deny the complaint, can't tell your compliance team what was actually said. This is the gap model-invocation logging closes.

Think of it like the flight recorder on an aircraft. The plane flies fine without a black box — but the moment something goes wrong, the difference between "we know exactly what happened" and "we're guessing" is whether the recorder was running. Bedrock model-invocation logging is that recorder for your model calls: a single account-and-region setting that tells Bedrock to write down every invocation — the input prompt, the output completion, and the token counts — to CloudWatch Logs, an S3 bucket, or both. Crucially, it's a platform-level switch. You flip it once and every subsequent Bedrock call is captured automatically; you don't wrap each call site in your own logging code, and you can't accidentally forget it on one path.

Two properties are worth burning into memory because they trip people up. First, logging is not retroactive — it only records calls made after you enable it. There is no way to recover the prompt from last Tuesday if the recorder wasn't on, which is exactly why "enable it later" is a trap; the audit trail you'll wish you had is the one you turned on before you needed it. Second, the textDataDeliveryEnabled flag is the difference between recording metadata (a call happened, N tokens) and recording the actual text of prompts and responses. Metadata is enough for cost accounting; the full text is what you need for debugging and compliance — and it's also what makes the logs sensitive.

That sensitivity is the governance twist, and it's why enabling logging is a decision rather than a free win. Those log records now contain raw prompt and completion text — which may include customer PII, secrets a user pasted in, or proprietary data. You have just created a second, highly sensitive copy of your most confidential traffic sitting in a bucket. So enabling logging comes bundled with obligations: lock the destination bucket and log group down with least-privilege IAM (only the people and roles who genuinely need it can read), and set a retention policy so you're not hoarding sensitive text forever. The lab below flips the switch; the warning box after it is the half of the lesson most tutorials skip.

enable_logging.pyimport boto3
bedrock = boto3.client("bedrock", region_name="us-east-1")

# Log every Bedrock invocation (prompt, response, token counts) to CloudWatch + S3.
bedrock.put_model_invocation_logging_configuration(
    loggingConfig={
        "cloudWatchConfig": {
            "logGroupName": "/bedrock/invocations",
            "roleArn": "arn:aws:iam::123456789012:role/bedrock-logging",
        },
        "s3Config": {"bucketName": "bedrock-logs"},
        "textDataDeliveryEnabled": True,
    }
)
print("invocation logging enabled")

Metrics, alarms & cost expert

Logging tells you what happened; it doesn't wake you up. A perfect audit trail is useless if nobody reads it until Wednesday. The classic failure here is the team that has all the data and still learns about the outage from an angry customer, or learns about the cost blowout from the monthly invoice. Logs are passive — they sit there. What you need next is something active: numbers that update continuously and machinery that shouts when a number goes wrong. That's the job of metrics and alarms.

First separate the three words people blur together. A metric is a number tracked over time — a line on a graph, like "average latency per minute." Bedrock publishes several to CloudWatch on its own, without you instrumenting anything: InvocationLatency (how long calls take), Invocations (how many), and input/output token counts. An alarm is a rule watching one metric that flips between states (OK ↔ ALARM) and can trigger an action when it crosses a line. A budget (AWS Budgets) is the same idea aimed specifically at spend. The mental split: metric = the gauge, alarm = the warning light wired to the gauge, budget = the fuel light.

The design decision that separates a useful alarm from a useless one is how long a problem must persist before it counts. This is what Period and EvaluationPeriods encode, and it's worth understanding rather than copying. Period is the size of each measurement window (300 seconds = a 5-minute bucket); EvaluationPeriods is how many consecutive windows must breach the threshold before the alarm fires. Set both together and you're saying "only page me if latency stays high for two 5-minute windows in a row" — because a single slow window is usually a harmless blip, but sustained slowness is a real incident. Get this wrong in the tight direction and you'll teach your on-call to ignore the pager (alarm fatigue); too loose and you'll sleep through the real thing.

Then there's cost, which is subtler because AWS has no "Bedrock spend right now" metric that updates by the second. You build the cost view yourself from what you do have: the token-count metrics (or the token counts in your invocation logs) multiplied by the model's published per-token price — and remember input and output tokens are priced differently, so you cost each side at its own rate. That gives you a near-real-time spend estimate. AWS Budgets is the safety net underneath it: you set a dollar (or usage) threshold, and when actual or forecast spend crosses it, Budgets sends an alert. The whole point is expressed in one line — runaway spend should page a human, not surface as a surprise on next month's invoice. The lab wires exactly this latency alarm; the same pattern extends to error rate and to your derived cost metric.

Bedrock publishes CloudWatch metrics (InvocationLatency, Invocations, input/output token counts). Alarm on latency and error spikes; build a cost view from token metrics × price. Set AWS Budgets alerts so runaway spend pages you, not the invoice.

alarm.pyimport boto3
cw = boto3.client("cloudwatch", region_name="us-east-1")

cw.put_metric_alarm(
    AlarmName="bedrock-latency-high",
    Namespace="AWS/Bedrock",
    MetricName="InvocationLatency",
    Statistic="Average",
    Period=300, EvaluationPeriods=2,
    Threshold=5000.0,                       # ms
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789012:oncall"],
)
print("alarm set")

Governance & the incident runbook expert

Here's the moment governance stops being an abstract word. It's 2am. A budget alert has just fired — spend is 10× normal and climbing. You're half awake, and the only question that matters is: how do I make it stop, right now? If your honest answer is "um, I'd have to find who has AWS access, figure out which service is calling, maybe redeploy something…", you don't have governance — you have hope. Governance is what turns that 2am scramble into a checklist you can execute in ninety seconds. It's the least glamorous part of the chapter and the part that saves you.

Strip away the jargon and governance answers five plain questions about your AI system, each mapping to a concrete AWS mechanism. Who is allowed to call the models? — least-privilege IAM, meaning each role gets exactly the permissions it needs and no more, so a compromised component can't do arbitrary damage. Where does the data physically live? — your region choice (and VPC endpoints for private routing), which is how you satisfy data-residency rules like "EU customer data stays in the EU." What was actually asked and answered? — the invocation log from the previous section, serving as the audit trail. What is the model allowed to say? — Guardrails (from W6) enforcing content policy. And how do we stop everything instantly? — the kill switch. Notice these aren't five products to buy; they're five questions to have an answer for, and a gap in any one is a compliance risk sitting quietly until an auditor or an incident finds it. The table below is that mapping, worth reading as a readiness checklist.

The idea that earns its own emphasis is the kill switch. In any incident — a cost runaway, a prompt-injection attack, a model behaving badly — your first move is not to diagnose, it's to stop the bleeding, then investigate from a safe state. A kill switch is simply the fastest concrete action that halts all model calls, and the discipline is deciding what it is before you need it. On AWS it might be disabling the agent alias (so callers hit a dead endpoint), revoking the IAM role the caller assumes (so calls fail authorization), or flipping a feature flag your application checks. The test of a good kill switch is brutally practical: could a groggy on-call engineer execute it correctly at 2am from their phone? A switch that requires a code change and a deploy is not a kill switch — it's a delay.

This is where the whole chapter clicks together into one loop. Observability detects the problem (the budget alarm pages you). Logging attributes it (you read the invocation log and see one caller is responsible for 95% of the spend). And the runbook's kill switch stops it (you disable that specific caller's alias — surgical, rather than taking the whole service down). Detect → attribute → stop. A runbook is just that loop written down in advance so nobody has to improvise it while the money burns. The exercises below have you build each piece and then assemble exactly this incident response.

Governance is least-privilege IAM per service, data-residency via region choice, guardrails (W6) as policy, and the invocation log as the audit trail. The incident runbook needs a kill switch: the fastest way to stop all model calls (disable the agent alias, revoke the role, or flip a feature flag) when something goes wrong.

import boto3

bedrock = boto3.client("bedrock", region_name="us-east-1")
bedrock.put_model_invocation_logging_configuration(
    loggingConfig={
        "cloudWatchConfig": {"logGroupName": "/bedrock/invocations",
                             "roleArn": "arn:aws:iam::123:role/bedrock-logs"},
        "s3Config": {"bucketName": "bedrock-logs"},
        "textDataDeliveryEnabled": True,
    },
)
print("logging enabled")

import boto3

cw = boto3.client("cloudwatch", region_name="us-east-1")
cw.put_metric_alarm(
    AlarmName="bedrock-latency-high",
    Namespace="AWS/Bedrock", MetricName="InvocationLatency",
    Statistic="Average", Period=300, EvaluationPeriods=2,
    Threshold=5000.0, ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123:oncall"],
)
print("alarm created")

def batch_cost(records, in_per_k, out_per_k):
    total = 0.0
    for r in records:
        total += r["in"]/1000*in_per_k + r["out"]/1000*out_per_k
    return round(total, 6)

logs = [{"in": 1200, "out": 300},
        {"in": 800,  "out": 500},
        {"in": 1500, "out": 200}]
print(batch_cost(logs, 0.003, 0.015))   # 0.0261

def health(latency_ms, error_rate, cost_per_hr, quality):
    checks = {
        "latency": "red" if latency_ms > 5000 else "yellow" if latency_ms > 3000 else "green",
        "errors":  "red" if error_rate > 0.05 else "yellow" if error_rate > 0.01 else "green",
        "cost":    "red" if cost_per_hr > 50 else "yellow" if cost_per_hr > 25 else "green",
        "quality": "red" if quality < 0.7 else "yellow" if quality < 0.85 else "green",
    }
    worst = "red" if "red" in checks.values() else \
            "yellow" if "yellow" in checks.values() else "green"
    return {"status": worst, "signals": checks}

print(health(2000, 0.002, 12, 0.9))     # green
print(health(6000, 0.002, 12, 0.9))     # red (latency)

REQUIRED = {"least_privilege_iam", "region_vpc", "invocation_logging",
            "guardrails", "kill_switch"}

def governance_gaps(controls):
    have = {k for k, v in controls.items() if v}
    return sorted(REQUIRED - have)

controls = {"least_privilege_iam": True, "region_vpc": True,
            "invocation_logging": True, "guardrails": False,
            "kill_switch": False}
gaps = governance_gaps(controls)
print(gaps)   # ['guardrails', 'kill_switch']
print("NOT ready" if gaps else "ready")

def incident(spend_now, budget, top_caller_share):
    over = spend_now / budget
    if over < 1.0:
        return {"action": "monitor", "over_budget_x": round(over, 2)}
    action = "disable_agent_alias" if top_caller_share > 0.8 else "revoke_role"
    return {"action": action,               # the kill switch
            "over_budget_x": round(over, 2),
            "then": "flip feature flag off, page on-call, review logs"}

print(incident(spend_now=1000, budget=100, top_caller_share=0.95))
# over_budget_x 10.0 -> disable_agent_alias (one caller dominates), then flag off

Knowledge check check yourself