FinOps for AI at scale
An AI feature with a great demo and unknown unit economics is a liability, not a product. This lesson is FinOps for LLM features: the discipline of knowing the cost per successful request, attributing spend to teams and features, enforcing token budgets and quotas, capturing the ROI of caching and batching, running showback / chargeback, and forecasting spend before finance is surprised. The methods are durable; every price is illustrative and must be verified against current vendor pricing.
Learning objectives
- Compute the unit economics of an LLM feature: cost per request and cost per successful request.
- Attribute spend with tagging so you know which team, feature, and customer drives cost.
- Set and enforce token budgets and quotas that protect margin without breaking the product.
- Quantify the ROI of caching and batching instead of assuming they help.
- Run showback / chargeback to change behavior, and forecast spend to avoid surprises.
1 · Unit economics — the only number that matters
The foundational FinOps metric for an AI feature is cost per successful request. Not cost per token (an input), not cost per API call (ignores retries and failures) — cost per request that actually delivered value. The formula is deliberately blunt:
cost_per_success = (input_tokens·in_rate + output_tokens·out_rate + retries + tool_calls) / success_rate. The ÷ success_rate is the part teams forget: if 20% of requests fail or get retried, your true cost per useful answer is ~25% higher than the naive per-call price. A feature can be profitable per call and unprofitable per success.Output tokens usually dominate because output rates are typically higher than input rates and generation is where length balloons. This single fact drives most cost wins: cap and shape output, and you move the biggest lever. Tie the unit cost to the revenue or value per request and you have the feature's margin — the number an exec actually needs.
2 · Cost attribution — tag everything
You cannot manage what you cannot attribute. Every request should carry metadata that lets you slice spend after the fact: team, feature, environment (prod/dev), customer/tenant (or a hashed id), and model. Without tagging, a spend spike is a mystery; with it, it's a query.
| Tag | Answers the question | FinOps action it enables |
|---|---|---|
| team | Who owns this spend? | Showback / chargeback |
| feature | Which product line drives cost? | Kill or optimize low-ROI features |
| environment | Are we burning prod money in dev? | Cap non-prod aggressively |
| tenant | Which customers are expensive? | Price tiers, quota per plan |
| model | Is a cheaper model viable here? | Route by task difficulty |
3 · Budgets, quotas, and model routing
Attribution tells you where money goes; budgets and quotas stop it going somewhere bad. Three complementary controls: a per-tenant/per-feature token budget (soft alert, hard cap), a rate quota (requests per minute) to stop runaway loops, and model routing that sends easy requests to a cheaper model and reserves the frontier model for hard ones. The flow below shows a request passing these gates.
4 · Caching & batching — ROI, not folklore
Two of the biggest cost levers are caching (don't pay twice for the same work) and batching (trade latency for a lower rate). Both are widely available — Anthropic publicly documents prompt caching and a batch/Message Batches path — but neither is free ROI; you must measure.
| Lever | How it saves | When ROI is real | When it isn't |
|---|---|---|---|
| Exact-response cache | Skip the call for repeated identical requests. | High repeat rate (FAQs, popular queries). | Long-tail unique requests — near-zero hit rate. |
| Prompt caching | Reuse a large stable prefix (system prompt, docs) across calls at a reduced rate. | Big shared context reused often. | Tiny or constantly-changing prompts. |
| Batching | Submit many requests together for a lower rate. | Latency-tolerant offline jobs (evals, backfills, enrichment). | Interactive, latency-sensitive requests. |
saving = hit_rate × cost_of_avoided_work − cache_overhead, and verify current caching/batch discounts in the provider's docs before modeling them.5 · Showback, chargeback, and forecasting
Showback shows each team what its AI usage costs (visibility, no bill). Chargeback actually bills it to their budget (accountability, real incentive). Showback changes awareness; chargeback changes behavior — but chargeback needs trustworthy attribution first, or you'll spend more time arguing about the numbers than saving money. Forecasting projects spend from usage trends and launch plans so finance sees the curve before it bends.
Σ(feature: projected_requests × cost_per_success), plus a line for each planned launch. That ties the finance number directly to product plans, so when someone asks "why is next quarter's AI bill up 40%?" the answer is "the new feature X at its projected volume," not a shrug.✓ Checkpoint — you can move on when you can…
- Compute cost per successful request and explain why dividing by success_rate matters.
- List the five tags every request should carry and one action each enables.
- Choose between soft alert, hard cap, degrade, and queue for a tenant that exceeds budget.
- Decide whether caching or batching is worth it for a given workload, using hit rate / latency tolerance.
- Explain the difference between showback and chargeback and what must be true before chargeback works.
A feature's dashboard shows $0.006 per API call and leadership is happy. You look closer and find a 15% retry rate and a 30% of answers that users immediately rephrase (a soft failure). What's the real unit cost story, and what do you report?
Show answer
An engineer proposes adding prompt caching everywhere to cut costs. Under what conditions will this actually save money, and when could it add cost or complexity for little gain?
Show answer
saving = reuse × cached_prefix_cost_delta − overhead, and confirm the current caching mechanics and discount in the provider's docs. "Cache everywhere" is folklore; "cache the big reused prefix" is FinOps.🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Unit economics is the first FinOps reflex; everything else builds on it.
Your task: Given usage numbers for a feature, compute cost per call and cost per successful request.
Requirements:
- Input: 800 input tokens, 400 output tokens per call (verify rates against current pricing)
- Assume in-rate $3 / M tokens, out-rate $15 / M tokens (illustrative)
- Success rate is 80% (20% fail or are retried)
- Report both cost-per-call and cost-per-success and explain the gap
💡 Hint: cost_per_call = in·in_rate + out·out_rate; then ÷ success_rate.
Show solution
Per call (illustrative rates — verify current pricing): input = 800/1e6 × $3 = $0.0024; output = 400/1e6 × $15 = $0.0060; per call = $0.0084.
Per successful request: $0.0084 / 0.80 = $0.0105 — about 25% higher than the per-call figure.
The gap explained: one in five calls doesn't deliver a useful answer but still costs money (a failure or a retry), so the cost of each useful answer absorbs those wasted calls. Notice output is 71% of per-call cost — capping/shaping output is the biggest single lever here.
Context: You can't manage what you can't attribute; tagging is the enabling infrastructure.
Your task: Design the metadata schema attached to every request and show one spend query it unlocks.
Requirements:
- List the required tags and their types
- Show how a hashed tenant id preserves privacy while enabling per-customer cost
- Write one aggregation (pseudo-SQL) that finds the most expensive feature
- State what you'd do first if 60% of spend came back untagged
💡 Hint: Tags travel with the usage record so cost can be grouped by any dimension later.
Show solution
Schema (per request usage record):
team(string),feature(string),env(prod|staging|dev),tenant_hash(sha256 of tenant id, truncated),model(string)input_tokens,output_tokens,retries,success(bool),cost_usd(computed)
Privacy: hashing the tenant id lets you compute per-customer cost and spot expensive tenants without storing raw customer identifiers in the FinOps store.
Query — most expensive feature: SELECT feature, SUM(cost_usd) c FROM usage WHERE env='prod' GROUP BY feature ORDER BY c DESC LIMIT 5;
If 60% is untagged: stop optimizing and fix attribution first — instrument the untagged call sites, because any optimization on 40% visibility is guesswork.
Context: Attribution finds the spend; budgets and routing keep it in bounds.
Your task: Design the enforcement for a per-tenant token budget with a soft alert and hard cap, plus difficulty-based model routing.
Requirements:
- Define soft-alert (e.g. 80%) and hard-cap (100%) behavior
- Decide what happens at the hard cap and justify it with product
- Define the routing rule that sends easy vs hard requests to different models
- Note how routing interacts with the budget
💡 Hint: Cheap model for easy requests both saves money and slows budget burn.
Show solution
Budget states. Track cumulative tokens per tenant per billing period. At 80%: soft alert to the tenant + account owner (no behavior change). At 100%: hard cap.
Hard-cap behavior (decided with product): degrade, don't fail — route remaining requests to the cheaper model and add a clear notice, rather than returning errors. Refusing outright looks like an outage; degrading preserves the product and margin. Enterprise plans may instead queue or allow overage-billing per contract.
Routing rule. A cheap classifier (or heuristic: length, presence of tool need, task type) labels each request easy|hard. Easy → cheap model; hard → frontier model. Fallback to frontier only if the cheap model's confidence/quality check fails.
Interaction: routing reduces average cost per request, which slows budget burn, so tenants hit the cap later — routing and budgeting reinforce each other rather than competing.
Context: Caching is assumed to help; a lead proves it with a number before shipping complexity.
Your task: Model the ROI of two caching strategies for a workload and recommend one.
Requirements:
- Workload: 1M req/month, of which 12% are exact repeats and 100% share a large system prompt
- Strategy A: exact-response cache. Strategy B: prompt caching on the shared prefix
- Estimate savings for each and net out cache overhead
- Recommend, and state what measurement would change the call
💡 Hint: saving = affected_fraction × avoided_cost − overhead; compare against doing nothing.
Show solution
Strategy A — exact-response cache. Only the 12% exact repeats benefit; they skip the call entirely. Saving ≈ 12% of inference spend, minus modest cache infra. Real and worth it at this hit rate.
Strategy B — prompt caching on the shared prefix. All 100% of requests share the large system prompt, so the cached prefix is billed at a reduced rate on nearly every call. If the prefix is, say, 60% of input tokens and caching cuts that portion's rate substantially, the saving applies broadly — typically a larger absolute win than A here, because it touches every request, not 12%. (Verify the current prompt-caching discount and minimums in the provider docs.)
Recommendation: do both — they stack (A avoids repeated work; B cheapens the rest). If forced to pick one, B, because it affects 100% of traffic. What would change it: if the system prompt actually varies per request (prefix not stable), B's saving evaporates and A becomes the only real lever — so measure prefix stability and exact-repeat rate first.
Context: Finance hates surprises; a bottom-up forecast ties the AI bill to product plans.
Your task: Build a next-quarter LLM spend forecast for a representative product with three features plus a planned launch.
Requirements:
- Forecast per feature = projected_requests × cost_per_success
- Add a line for a new feature launching mid-quarter
- Show the total and the month-over-month curve
- State the top assumption and how you'd track it against actuals
💡 Hint: Forecast = Σ over features; make each line traceable to a product number.
Show solution
Bottom-up forecast (illustrative).
| Feature | Req/mo | $/success | Monthly |
|---|---|---|---|
| Support assistant | 200k | $0.011 | $2,200 |
| Doc search | 500k | $0.004 | $2,000 |
| Summaries (batch) | 300k | $0.002 | $600 |
| New: sales-email drafter (launches month 2) | 0 → 150k | $0.009 | $0 / $1,350 / $1,350 |
Curve: Month 1 ≈ $4,800; Month 2 ≈ $6,150; Month 3 ≈ $6,150 — the step is the launch, not mystery growth. Quarter total ≈ $17,100.
Top assumption: the drafter's 150k projected volume and $0.009/success. Track actuals weekly against this line; if adoption runs hot, the forecast flexes on a known variable, not a surprise. All rates verified against current pricing at forecast time.
Context: Representative scenario: the monthly LLM invoice tripled with no obvious feature launch. The CFO wants an explanation and a plan by Friday.
Your task: Run the FinOps investigation and deliver a diagnosis plus a remediation plan.
Requirements:
- State the first thing you check and why
- List the likely culprits a tripled bill usually hides
- Give the immediate stop-the-bleeding actions and the durable fixes
- Propose the controls that prevent a recurrence
- Frame the CFO-facing summary
💡 Hint: Attribution first — you can't fix what you can't see; then triage by biggest line item.
Show solution
First check: attribution. Slice the spend by feature/team/env/model/tenant. A 3× jump almost always shows up as one or two lines, not uniform growth. If spend is largely untagged, that's finding #1 and the first fix.
Likely culprits: (1) a retry storm — a failing dependency causing loops that each pay for tokens; (2) a prompt/context bloat change that ballooned input tokens on every call; (3) traffic hitting the frontier model where routing should have sent it to a cheaper one; (4) dev/staging pointed at prod-priced endpoints; (5) a single abusive or runaway tenant.
Stop the bleeding (this week): add rate quotas to cap runaway loops, cap non-prod spend, and put a hard budget on the top offending tenant/feature (degrade, don't fail). Durable fixes: fix the retry logic with backoff + a circuit breaker; trim the bloated prompt; turn on difficulty routing; add caching if reuse is real.
Prevent recurrence: spend anomaly alerts on the tagged data, budgets/quotas per tenant, a cost gate in CI that flags prompt changes that raise average tokens, and a monthly showback so teams see their own line.
CFO summary: "The increase is X% from [specific cause], not broad growth. We've capped it this week and the durable fix lands in N weeks; going forward, budgets, routing, and anomaly alerts make a silent 3× impossible."