FinOps for AI at scale

1 · Unit economics — the only number that matters

Start with a scene you've watched play out. A team ships an AI feature. The demo is dazzling, adoption climbs, everyone celebrates — and then, two months later, the cloud invoice arrives and nobody can say whether the feature makes money or loses it. They know the total bill. They cannot tell you what a single answer costs, or whether that cost is smaller than the value it creates. That gap is not a rounding error; it is the difference between a product and a very expensive science project.

Here is the mental model that fixes it. Think of your LLM feature like a factory that makes one thing: a successful answer. A factory owner doesn't brag about the price of raw steel or the number of times a machine cycled — they care about the all-in cost of one finished, sellable part, including the scrap. FinOps for AI is the same discipline pointed at tokens: the only number that lets you reason about margin is the fully-loaded cost of one answer a user actually kept.

That's why the foundational metric is cost per successful request — and why the two obvious alternatives quietly lie to you. Cost per token is an input, like the price of steel: it tells you nothing about how many tokens a real answer burns or how often you throw the part away. Cost per API call is closer but still deceptive, because it counts every retry and every failed attempt as if it delivered value. The most common mistake in AI FinOps is exactly this: a dashboard proudly reports a low cost-per-call, leadership relaxes, and nobody notices that a fifth of those calls were retries or answers the user immediately rephrased. You were measuring the machine cycling, not the parts you sold.

So the formula deliberately divides by the fraction of calls that succeeded — because the wasted calls don't vanish from the bill, they get absorbed into the cost of the answers that worked. Pair that unit cost with the value (revenue, deflected support ticket, hour saved) each successful answer creates, and you finally have the one figure an executive can act on: the feature's margin. Everything else in this lesson — tagging, budgets, caching, forecasting — exists to move or protect this single number.

The foundational FinOps metric for an AI feature is cost per successful request. Not cost per token (an input), not cost per API call (ignores retries and failures) — cost per request that actually delivered value. The formula is deliberately blunt:

Output tokens usually dominate because output rates are typically higher than input rates and generation is where length balloons. This single fact drives most cost wins: cap and shape output, and you move the biggest lever. Tie the unit cost to the revenue or value per request and you have the feature's margin — the number an exec actually needs.

2 · Cost attribution — tag everything

Picture the meeting nobody enjoys. The AI bill jumped 40% this month and the VP wants to know why. You open the provider console and find… one number. A single line item: "model usage." No breakdown by team, feature, or customer. You can see that spend happened; you have no idea where. So the honest answer is a shrug, and the next hour is spent guessing. That shrug is the single most common failure in AI cost management — not a rate that's too high, but spend you cannot explain.

The analogy that makes it click: imagine running a shared office and getting one combined electricity bill for the whole building with no sub-meters. You know the total went up. Was it the server room? The new team on the third floor? Someone leaving the AC on all weekend? Without meters on each circuit, you can only argue, never fix. Tagging is installing the sub-meters. Every request carries a little label — which team, which feature, which environment, which customer, which model — so that when the bill arrives, a mystery becomes a query.

Why these particular tags, and not a hundred others? Because each one answers a question that leads directly to an action, and that's the test a tag must pass — a label you'll never slice by is just noise. Team and feature tell you who owns the spend and which product line drives it, so you can bill it back or kill a low-ROI feature. Environment catches the classic leak of burning production-priced money in dev and staging. Tenant (hashed, so you keep the cost signal without storing raw customer identifiers) reveals which customers are expensive enough to need a price tier or a quota. Model shows where you're paying frontier rates for work a cheaper model could do.

The mistake to avoid is treating tagging as a nice-to-have you'll add "once things settle." Attribution is not an optimization — it is the precondition for every optimization. If your first tagging pass comes back and 60% of spend has no label, that untagged blob is your finding: you fix attribution before you touch anything else, because tuning a system you can only see 40% of is guessing with extra steps.

You cannot manage what you cannot attribute. Every request should carry metadata that lets you slice spend after the fact: team, feature, environment (prod/dev), customer/tenant (or a hashed id), and model. Without tagging, a spend spike is a mystery; with it, it's a query.

3 · Budgets, quotas, and model routing

Here's the failure this section prevents. A dependency starts timing out at 2 a.m. Your agent, dutifully, retries. The retry also fails, so it retries again — a tight loop, each pass paying full price for tokens, thousands of times an hour, while everyone sleeps. By morning a single misbehaving code path has spent a month's budget. Nobody chose to spend that money; there was simply nothing standing between the code and the credit card. Attribution would tell you afterward where the money went. Budgets, quotas, and routing are what stop it going there in the first place.

Think of it like the controls on household spending. A token budget is your monthly spending limit — a soft alert when you're getting close, a hard cap when you've hit it — scoped per tenant or per feature so one heavy user can't drain the shared pool. A rate quota (requests per minute) is the circuit breaker: it doesn't care about the monthly total, it just refuses to let any single source fire absurdly fast, which is exactly what kills a runaway retry loop before it does real damage. And model routing is buying the right tool for each job instead of the most expensive tool for every job.

Routing deserves a moment because it's the most leverage for the least pain. Most requests are easy — a short classification, a lookup, a reformat — and a cheaper, faster model handles them perfectly well. A minority are genuinely hard and need the frontier model's reasoning. If you send everything to the frontier model "to be safe," you're paying premium rates on traffic that never needed them. A cheap classifier or a simple heuristic (length, whether a tool is required, task type) labels each request easy or hard and routes accordingly, falling back to the frontier model only when the cheap one's quality check fails. The quiet bonus: because routing lowers the average cost per request, it also slows how fast tenants burn their budget — the two controls reinforce each other.

The mistake to avoid is imagining the hard cap as a simple on/off switch an engineer flips. What happens when a paying customer hits their cap? Silence and errors look exactly like an outage, and the support cost of that surprise can dwarf whatever you saved. The cap is a product decision — degrade to a cheaper model, queue the work, or refuse with a clear message — made with product and honoring the contract, not a value quietly hard-coded in a config file.

Attribution tells you where money goes; budgets and quotas stop it going somewhere bad. Three complementary controls: a per-tenant/per-feature token budget (soft alert, hard cap), a rate quota (requests per minute) to stop runaway loops, and model routing that sends easy requests to a cheaper model and reserves the frontier model for hard ones. The flow below shows a request passing these gates.

4 · Caching & batching — ROI, not folklore

You've probably heard the confident advice. Someone in a review says "just add caching, it'll cut costs," and everyone nods because it sounds obviously true. Three weeks later the cache is live, the bill hasn't moved, and there's a new layer of infrastructure to maintain. The advice wasn't wrong in general — it was applied without checking whether this workload had the property that makes caching pay. Caching and batching are two of the largest cost levers in AI, and both are widely available, but neither delivers savings by being switched on. They deliver savings only when the workload's numbers cooperate — which is why this section is called ROI, not folklore.

Start with the intuition behind each. Caching is "don't pay twice for the same work." There are two flavors. An exact-response cache is a photocopier: the identical question comes in, you hand back the answer you already generated, and you skip the model entirely. That only helps to the degree that questions actually repeat — great for a popular FAQ, useless for a stream of unique long-tail prompts. Prompt caching is subtler and often more powerful: when many different requests all begin with the same large stable prefix — a big system prompt, a shared set of documents — you can have the provider bill that reused prefix at a reduced rate instead of full price on every call. It's like a restaurant prepping a base sauce once each morning rather than from scratch per plate: the shared part gets cheaper, the per-order part stays custom.

Batching is a different trade entirely — you give up latency to get a lower rate. Submit many requests together as an offline job and the provider can schedule them efficiently and charge less. That's a bargain for latency-tolerant work like nightly evals, backfills, or bulk enrichment, and a non-starter for anything a user is waiting on in real time. Anthropic, for instance, publicly documents both prompt caching and a Message Batches path; the mechanics and the exact discounts move, so confirm the current numbers in the provider's docs before you put them in a model.

The discipline — and the mistake to avoid — is celebrating before you measure. A cache with a 5% hit rate saves about 5% and adds complexity you'll maintain forever; that's often a net loss. Prompt caching pays off only when the reused prefix is genuinely large and genuinely reused before it expires — if the "stable" prefix actually changes every request, the saving evaporates. So you compute it honestly: saving = hit_rate × cost_of_avoided_work − overhead, compared against doing nothing. "Cache everywhere" is folklore; "cache the big reused prefix, because the hit rate says so" is FinOps.

Two of the biggest cost levers are caching (don't pay twice for the same work) and batching (trade latency for a lower rate). Both are widely available — Anthropic publicly documents prompt caching and a batch/Message Batches path — but neither is free ROI; you must measure.

5 · Showback, chargeback, and forecasting

Consider two ways to change how a team spends. In the first, at the end of each month you send every team a report: "here's what your AI usage cost." They can read it or ignore it; nothing lands on their budget. In the second, that same cost is actually deducted from the team's budget — real money, out of a pool they had other plans for. The first is showback; the second is chargeback. The distinction sounds like accounting trivia, but it's the whole psychology of cost control: visibility makes people aware; a bill on their own budget makes them act. A team that merely sees a number tends to keep doing what it was doing; a team whose budget shrinks when it's wasteful suddenly cares about that bloated prompt.

There's a sequencing trap here, though. Chargeback is more powerful precisely because it's real money — which means the moment the numbers look wrong, the conversation stops being about saving money and becomes an argument about your accounting. If your attribution is shaky (remember the untagged-spend problem from §2), turning on chargeback too early buys you months of disputes instead of savings. So the rule is: showback first to build trust in the numbers, chargeback only once the attribution is solid enough that nobody can credibly contest their line.

Forecasting is the third piece, and it exists to kill the surprise. Finance can absorb a rising AI bill; what it cannot forgive is a rising bill nobody warned them about. The naive forecast — "last month times some growth factor" — is a shrug dressed up as a projection, and it falls apart the moment someone asks why. The credible forecast is built bottom-up from the unit economics you established in §1: for each feature, projected_requests × cost_per_success, summed across the portfolio, plus an explicit line for each planned launch. Now every dollar in the projection traces back to a product number.

That traceability is the entire payoff, and it's why forecasting closes the loop this lesson opened. When an exec asks "why is next quarter's AI bill up 40%?", the bottom-up forecast lets you answer "the new sales-drafter feature at its projected 150k requests a month" — a specific, defensible, product-linked cause — instead of the shrug you started with in §2. Unit economics, attribution, budgets, and caching all feed this final number; getting it right is what turns FinOps from a cleanup exercise into a planning tool.

Showback shows each team what its AI usage costs (visibility, no bill). Chargeback actually bills it to their budget (accountability, real incentive). Showback changes awareness; chargeback changes behavior — but chargeback needs trustworthy attribution first, or you'll spend more time arguing about the numbers than saving money. Forecasting projects spend from usage trends and launch plans so finance sees the curve before it bends.