Throughput economics

1 · The only cost metric that matters: tokens/sec/GPU advanced

Start with the meeting you've either sat in or will. A team ships a self-hosted model, the demo screams, everyone's thrilled — then the first cloud bill lands and it's five times what the spreadsheet promised. Nobody lied; the model didn't change, the GPU price didn't change. What broke the estimate was that the spreadsheet quoted a single glossy number ("$0.20/1M tokens!") without asking the one question that actually decides the bill: how many tokens does this GPU really produce per second, and how much of the hour is it actually producing them? This whole chapter exists to make that question reflexive.

Here's the analogy to hold onto: a GPU is a taxi with the meter always running. You pay by the hour the car is on the road, not by the passenger. A taxi that carries one rider per trip and a taxi that carries four cost the same per hour — but the four-rider taxi's cost per passenger is a quarter as much. Your GPU is identical: it bills a fixed dollars-per-hour whether it emits 10 tokens/sec or 10,000, so the only way to cheapen each token is to cram more tokens into every paid second. That "tokens carried per paid hour" is exactly what tokens/sec/GPU measures.

Now the precise definition: tokens/sec/GPU (also called goodput when you count only useful output) is the aggregate output tokens one accelerator generates per second, summed across every request it's serving at once. It is the pivot on which the entire cost model turns, because $/token = GPU-hour price ÷ tokens produced that hour — the price is fixed, so tokens/sec/GPU is the only variable you control. That reframing is the mental unlock of this chapter: every optimization you'll ever read about — quantization (IC2), the KV-cache (IC3), batching (IC4), tensor/pipeline parallelism (MS4) — is worth exactly the tokens/sec/GPU it buys and not a cent more. Quote a speedup in "20% faster" and you can't compare it to anything; quote it in tokens/sec/GPU and it converts straight to dollars.

The common mistake is confusing tokens/sec/GPU with raw single-stream speed — how fast one user's response streams. Those pull in opposite directions: batching many users together makes each individual response a little slower but drives aggregate tokens/sec/GPU (and thus cost) dramatically better. A card that feels "slow" to one user can be the cheapest per token in the fleet. Serving economics cares about the aggregate, not the single stream — keep the two straight and half the confusing benchmarks you'll read suddenly make sense.

Everything in serving economics reduces to one number: tokens/sec/GPU (also written tok/s/GPU or the aggregate goodput). A GPU costs a fixed number of dollars per hour whether it emits 10 tokens/sec or 10,000. So $/token is just the GPU-hour price divided by how many tokens that GPU produces in an hour. Every other lever in this track — quantization (MS/IC2), the KV-cache (IC3), batching (IC4), tensor/pipeline parallel (MS4) — is worth exactly the tokens/sec/GPU it buys you. Learn to quote everything in this unit.

Order-of-magnitude anchors you should carry in your head (2025-era, single accelerator): an H100 rents for roughly $2–4/GPU-hour on-demand from the big clouds and specialist providers; an A100 is cheaper (~$1–2). A well-batched 7–8B model on one H100 does thousands of output tokens/sec aggregate; a 70B model sharded across GPUs does far fewer per GPU. These are ballparks to reason with — always re-measure your own stack (IC1).

2 · From GPU-hour to $/1M tokens advanced

You have tokens/sec/GPU. Finance has a budget in dollars. Section 2 is the bridge between them — and it's a bridge people cross badly all the time. The classic failure: someone divides the GPU-hour price by the datasheet's peak throughput, gets a beautiful number, and quotes it as gospel. Six months later the real bill is triple that, and nobody can explain why because the arithmetic was "correct." The arithmetic was correct; the inputs were fantasies. This section teaches you to do the conversion honestly, which mostly means being honest about one sneaky input.

Think of it like converting a car's fuel economy into a cost-per-mile. You don't just take the sticker MPG and the fuel price — you take the MPG you actually get in your real driving (city traffic, hills, the AC on), and only that number turns into a cost you can trust. Serving is the same: you take your measured tokens/sec (not the datasheet peak), the GPU-hour price you actually pay, and — the input everyone forgets — the fraction of paid hours the GPU is genuinely working. Multiply those honestly and you get a $/1M-tokens figure you can defend.

The conversion itself is grade-school arithmetic, which is exactly why it lulls people into trusting it. One GPU-hour buys tokens_per_sec × 3600 tokens, but only if the GPU is busy every one of those 3600 seconds. So you multiply by utilization — the fraction of paid time actually spent generating billable tokens — to get the tokens you truly bought. Divide the hourly price by that, scale to a million, and you have $/1M tokens. Written out: $/1M = GPU-hour price ÷ (tokens/sec × 3600 × utilization) × 1,000,000. The three boxes in the diagram below — price, throughput, utilization — are the only inputs, and the last one is the one that quietly wrecks estimates.

Why bother, when the API just posts a price? Because that's the whole point of the conversion: it puts your self-hosted cost in the same unit as the API's published $/1M so you can lay them side by side. If your honest number sits below the API's price at your real volume, self-hosting is worth investigating; if it sits above, the API is simply cheaper and you can stop the analysis right here and save yourself the ops burden. The mistake to avoid is comparing an API's real, utilization-inclusive price against your own 100%-utilization fantasy — an apples-to-fantasy comparison that has launched a thousand regretted GPU clusters.

Here is the whole cost model on one line. Three inputs — the GPU-hour price, the tokens/sec/GPU it sustains, and your utilization (the fraction of paid time the GPU is actually generating billable tokens) — collapse into a single $/1M tokens, which you then hold up against the API's published price.

The arithmetic: one GPU-hour buys tokens_per_sec × 3600 tokens at full utilization. Divide the hourly price by that token count and scale to a million. Then divide by utilization, because tokens you didn't serve still cost you the full hour. If your measured cost sits below the API's $/1M price at your real volume, self-hosting is in the running; if it sits above, the API is simply cheaper and you should stop here.

3 · The $/1M-tokens calculator advanced

The best way to feel how brutal utilization is on your wallet is to hold everything else fixed and watch one number move. That's what this tiny calculator is for. It's tempting to skip past a five-line function as too trivial to matter — but the triviality is the lesson. If the formula were complicated, you'd blame the math when the bill surprised you. It isn't. The formula is one division; the surprises all live in the inputs you feed it.

Picture the same physical GPU on a bench, running the identical model at an identical 2,500 tokens/sec, at an identical $3/hour. Nothing about the hardware or the workload changes. The only thing we vary is how many of the paid hours it's actually busy. Run it flat-out and the tokens are cheap; let it idle two-thirds of the day and each token you did serve has to carry the cost of the two you didn't. The calculator makes that carry-cost visible: at 100% utilization the H100 lands near $0.33/1M tokens; at 50% it doubles; at 20% it more than quintuples — all with the sticker price frozen.

Why does halving utilization exactly double the cost? Because utilization sits in the denominator: $/1M ∝ 1 ÷ utilization. Cut the busy fraction in half and you've halved the tokens that same fixed hourly bill buys, so the cost per token doubles. This inverse relationship is the single most important intuition in the chapter, and it's why a "cheap on paper" self-host quote so often turns into an expensive reality — the paper assumed 100% and reality delivered 30%. The guard clause in the code (rejecting a utilization outside 0–1 or a non-positive token rate) exists for the same honesty reason: a typo shouldn't be able to hand you a comforting garbage number.

Read the code as three moves: convert the hourly price into tokens-per-hour at your real utilization, divide to get dollars-per-token, then scale by a million to reach the industry-standard $/1M quote everyone speaks in. The loop deliberately changes only utilization so you can attribute the entire cost swing to that one lever — the discipline you want when you later argue a budget.

This is the model above as code. It is deliberately tiny — the point of throughput economics is that the formula is trivial and the inputs (measured goodput, honest utilization) are where all the judgement lives.

cost_per_mtok.pydef cost_per_million_tokens(gpu_hourly, tokens_per_sec, utilization):
    """$/1M output tokens for a self-hosted GPU.
    gpu_hourly     : $ per GPU-hour (e.g. 3.00 for an H100 on-demand)
    tokens_per_sec : measured aggregate output tokens/sec on that GPU
    utilization    : fraction of paid time actually generating (0<u<=1)
    """
    if tokens_per_sec <= 0 or not (0 < utilization <= 1):
        raise ValueError("need tokens_per_sec>0 and 0<utilization<=1")
    tokens_per_hour = tokens_per_sec * 3600 * utilization
    dollars_per_token = gpu_hourly / tokens_per_hour
    return dollars_per_token * 1_000_000

for u in (1.0, 0.5, 0.2):
    c = cost_per_million_tokens(gpu_hourly=3.00, tokens_per_sec=2500, utilization=u)
    print(f"util {u:>4.0%}  ->  ${c:6.2f} / 1M tokens")

util 100%  ->  $  0.33 / 1M tokens
util  50%  ->  $  0.67 / 1M tokens
util  20%  ->  $  1.67 / 1M tokens

At full utilization an H100 doing 2,500 tok/s costs about $0.33 per million output tokens — cheaper than most frontier APIs. But halve utilization and the cost doubles; drop to 20% and you're at $1.67. The GPU-hour price never changed. Utilization is the entire story, which is why it gets its own section.

4 · Batching: the throughput→cost curve and its knee expert

Here's the seductive trap that catches good engineers. You've internalized "utilization is everything" and "cram more tokens into each paid second," so batching looks like free money — bigger batch, more tokens/sec, lower $/token. Someone plots the first few points, sees cost falling fast, and reasons: if batch 4 is cheaper than batch 1, batch 128 must be cheaper still, so max the batch and cut the bill. Then p99 latency detonates, users start timing out, and the "savings" past a certain point turn out to have been pennies bought with seconds. This section is about finding the exact point where batching stops being a gift.

The analogy is a shared shuttle bus versus a private car. One passenger in a car pays for the whole trip; put four in the shuttle and they split the cost of the same fuel — that's batching amortizing the model's weight-read across many requests, and it's genuinely powerful. The reason it works: token generation (decode) is memory-bound, meaning each token spends most of its time re-reading the model's weights from GPU memory, not doing math. Batch several requests and they all ride on one weight-read, so aggregate tokens/sec climbs steeply and $/token falls in lockstep. So far, so good.

But a shuttle only seats so many, and a fuller shuttle takes longer as it waits for everyone to board — and that's where the free lunch ends. Two hard ceilings bite as the batch grows. First, the KV-cache runs out of GPU memory: every concurrent sequence needs its own cache (IC3), and once the card is full you physically cannot add another rider. Second, per-token latency (TPOT) rises as more requests compete for the same compute, so each user waits longer. The result is a curve that climbs steeply, then bends over and flattens — the knee. Before the knee, each doubling of the batch is a big cost win; after it, throughput barely moves while latency keeps climbing.

The mistake, then, is optimizing for the ceiling instead of the knee. Past the knee you're paying real, SLO-threatening latency to shave fractions of a cent off each million tokens — a terrible trade for an interactive service. The right rule is batch to the knee, not to the ceiling: find where the marginal savings go flat, park your operating point just below it, and when you need more capacity, add GPUs (scale out) rather than stuffing the batch deeper. The lab below prints the marginal saving per doubling precisely so you can see the knee arrive rather than guess at it.

Decode is memory-bound: every token re-reads the model's weights (IC1/IC3). Batching amortizes that one weight-read across many concurrent requests (IC4), so aggregate tokens/sec/GPU climbs steeply with batch size — and $/token falls in lockstep. But it does not climb forever. Two ceilings bite: the KV-cache runs out of GPU memory (you can't hold more sequences), and per-request latency (TPOT) rises as the batch competes for compute. The knee is where throughput flattens; pushing past it buys almost no cost savings while blowing your latency SLO.

batch_curve.pydef throughput_at_batch(batch, per_stream_toks=55, knee=32, ceiling=3200):
    """Toy model of aggregate tokens/sec vs batch size.
    Near-linear early (weight-read amortized), saturating past the knee
    where KV-cache/compute cap the GPU. Illustrative shape, not a spec."""
    ideal = per_stream_toks * batch                 # if it scaled forever
    saturating = ceiling * (batch / (batch + knee))  # diminishing returns
    return round(min(ideal, saturating))

def cost_per_mtok(gpu_hourly, tps):
    return gpu_hourly / (tps * 3600) * 1_000_000

print("batch  tok/s   $/1M   marginal $/1M saved")
prev = None
for b in (1, 4, 8, 16, 32, 64, 128):
    tps = throughput_at_batch(b)
    c = cost_per_mtok(3.00, tps)
    delta = "" if prev is None else f"{prev - c:+.3f}"
    print(f"{b:>4}  {tps:>5}  ${c:6.3f}   {delta}")
    prev = c

batch  tok/s   $/1M   marginal $/1M saved
   1     55  $15.152   
   4    220  $ 3.788   +11.364
   8    440  $ 1.894   +1.894
  16    880  $ 0.947   +0.947
  32   1600  $ 0.521   +0.426
  64   2133  $ 0.391   +0.130
 128   2560  $ 0.326   +0.065

Read the last column. Going from batch 1→4 saves $11/1M tokens; 32→64 saves 13¢; 64→128 saves 6¢ — and every step up the batch adds queueing and TPOT latency. The economically interesting region is the early, steep part; beyond the knee (~32 here) you are trading real latency for pennies. Batch to the knee, not to the ceiling.

5 · Utilization — the hidden cost expert

Two teams, same model, same GPU, same $3/hour — and one team's tokens cost five times the other's. That's not a benchmark error or a magic kernel; it's the most common and most preventable cost blowout in serving, and the culprit has been lurking in every formula so far: utilization. It's earned its own section because it's the input people omit — not because they can't compute it, but because it doesn't feel like a cost. An idle GPU isn't doing anything wrong; it's just sitting there. But sitting there is precisely the expense.

The mental model is a gym membership. You pay the same monthly fee whether you go every day or twice a year — the fee is for access, not for workouts. Your "cost per workout" is entirely determined by how often you show up. A reserved or on-demand GPU is identical: it bills for wall-clock time, every hour of it, whether it's generating tokens or staring at an empty queue at 3 a.m. The GPU-hour price is the membership fee; utilization is how often you actually show up; and your true $/token is the fee spread across only the tokens you served.

This is why the same hardware yields wildly different costs. Real traffic is diurnal and bursty — busy during working hours, dead overnight, spiky within the day — so a GPU sized to handle the peak sits mostly idle the rest of the time. Left un-managed, diurnal patterns alone can cap realistic average utilization around 30–40%, which (by the inverse law from section 3) roughly triples your cost versus the 100% fantasy. The levers to claw it back are the practical payoff of this section: autoscale down at night, batch offline work into the troughs so the GPU is never truly idle, or rent by the token (an API) and let someone else solve the packing problem.

And that last point reveals why APIs can undercut a naive self-host quote. A frontier API is effectively a giant shared GPU pool aggregating thousands of customers' bursty traffic into one smooth, near-100%-utilized load — the pooled efficiency is baked into its per-token price. When you self-host, you inherit the utilization problem yourself, and your single-tenant traffic is far lumpier than a pool's. The mistake to never make: trusting a self-host cost that assumed 100% utilization. Measure your real busy fraction first; the purchasing-mode table below then shows how commitment level (on-demand, reserved, spot) interacts with that number to set your effective price.

A reserved or on-demand GPU bills for wall-clock time, not for tokens. Every second it sits idle — overnight, between traffic bursts, waiting on a slow upstream, or over-provisioned "just in case" — is billed at 100% and produces zero tokens. This is why two teams running the same model on the same GPU can have a 5× difference in $/token: one runs it hot, the other runs it at 20% average utilization. Diurnal traffic (busy 9–5, dead at night) alone can cap your realistic average utilization near 30–40% unless you autoscale, batch offline work into the troughs, or rent by the token.

The purchasing mode is a second utilization lever. Reserved capacity is cheap per hour only if you keep it busy — an idle reserved GPU is the worst of both worlds. Spot is the cheapest raw price but can vanish mid-request, so it fits batch/offline (AP1-style) work, not your latency-SLO online path. A common pattern: reserved for the predictable baseline, on-demand to absorb peaks, spot for overnight batch.

6 · Self-host vs API: the break-even volume expert

"Self-hosting is cheaper" and "the API is cheaper" are both true — the honest question is at what volume? Engineers argue this in absolutes and talk past each other for hours, because each is picturing a different traffic level. A weekend side-project and a company serving a billion tokens a month live on opposite sides of a line, and the entire debate collapses once you find that line. This section turns a religious war into a single number you can compute.

The cleanest way to see it is two phone plans. One is pay-as-you-go: no monthly fee, you pay per minute — that's the API, a straight line through the origin whose slope is the per-token price. The other is a fixed monthly plan: a chunky bill every month whether you call or not, plus almost nothing per minute — that's self-hosting, a line with a high y-intercept (the GPUs you rent 24/7) and a gentle slope (your tiny marginal per-token cost). Two lines with different intercepts and slopes cross at exactly one point. Below that crossover the pay-as-you-go plan is cheaper; above it the fixed plan wins. That crossover is your break-even volume.

The arithmetic follows straight from the picture. Self-host beats the API by (api_price − self_host_marginal) on every million tokens — call that the per-million saving. To justify self-hosting you must save enough to pay off the fixed GPU bill, so the break-even is simply fixed_monthly_cost ÷ per-million-saving. Feed in a reserved GPU's monthly cost and a realistic marginal rate and the crossover pops out in millions of tokens per month. Below it, don't self-host; above it, self-hosting can win.

Now the two mistakes that make this number lie. First, that gentle self-host slope assumes the utilization your section-3 marginal cost was computed at — leave the GPU 30% utilized and your real marginal cost triples, tilting the line up and shoving break-even far to the right. Break-even and utilization are the same conversation wearing different clothes. Second, the fixed cost in the formula is not just GPUs: a self-hosted stack needs a human to patch, autoscale, and answer pages at 2 a.m. Fold even a fraction of an engineer's salary into the fixed cost and the break-even jumps by hundreds of millions of tokens. The calculator below computes the naive crossover; the "try this" then makes you add the salary and watch it move — because that's the version finance actually lives with.

Self-hosting converts a per-token cost (the API bill, which scales with usage) into a fixed monthly cost (GPUs you rent whether busy or not) plus a small marginal per-token cost. The API is a flat line through the origin; self-host is a line with a big y-intercept and a gentle slope. They cross at one volume — your break-even. Below it the API wins on cost; above it self-host can win, if you actually reach the utilization your fixed cost assumed. This is the GPU-serving version of the general build-vs-buy from LM5.

breakeven.pydef breakeven_volume(api_price_per_mtok, gpu_monthly_fixed, self_host_per_mtok):
    """Monthly volume (in millions of tokens) where self-host total cost
    equals the API bill.  api = api_price * V ;  self = fixed + shp * V.
    Returns the crossover V in M-tokens/month, or None if self-host never wins."""
    slope_gap = api_price_per_mtok - self_host_per_mtok
    if slope_gap <= 0:
        return None  # API's per-token price already <= our marginal cost
    return gpu_monthly_fixed / slope_gap

def monthly(api, fixed, shp, V):
    return api * V, fixed + shp * V   # (api_bill, self_host_bill) at V M-tokens

# One reserved H100 ~ $1.80/hr committed -> ~$1,314/month fixed.
# Self-host marginal cost ~ $0.33/1M (from section 3). API list ~ $3.00/1M.
be = breakeven_volume(api_price_per_mtok=3.00, gpu_monthly_fixed=1314,
                      self_host_per_mtok=0.33)
print(f"break-even at ~{be:,.0f} M tokens/month")
for V in (100, be, 1000):
    api_bill, sh_bill = monthly(3.00, 1314, 0.33, V)
    winner = "API" if api_bill < sh_bill else "self-host"
    print(f"  V={V:>7,.0f}M  API=${api_bill:>8,.0f}  self=${sh_bill:>8,.0f}  -> {winner}")

break-even at ~492 M tokens/month
  V=    100M  API=$     300  self=$   1,347  -> API
  V=    492M  API=$   1,476  self=$   1,476  -> self-host
  V=  1,000M  API=$   3,000  self=$   1,644  -> self-host

Below ~492M tokens/month the single reserved GPU's fixed cost isn't amortized and the API is cheaper; above it self-host pulls ahead, and at 1B tokens/month it's nearly 2× cheaper. But note the trap: that $1,314 fixed cost assumes the GPU is busy enough to serve 492M tokens. If diurnal traffic leaves it 30% utilized, your effective marginal cost triples and the break-even volume moves far to the right. Break-even and utilization are the same conversation.

7 · Capacity planning under a latency SLO professional

This is where the demo number goes to die. Every prior section handed you a clean cost — $0.33/1M at full tilt — and here we watch that number balloon to five times bigger the moment it meets real traffic and a real latency promise. If you've ever presented a serving budget that finance later called "off by 5×," this is the section that explains the gap and, more importantly, closes it before the meeting instead of after.

Two forces you've already met now collide. From section 4 you know you can't just max the batch, because p99 latency is a hard promise to users, not a knob — push the batch past the SLO-safe point and you've broken the product to save pennies. So the throughput you're actually allowed to use per GPU is the SLO-safe rate, deliberately below the raw ceiling. From section 5 you know you must size the fleet to the peak (you can't drop requests at the busy hour), yet the fleet then sits mostly idle at the average hour — so you buy for the peak but run at the average.

Think of it like staffing a restaurant for the Friday-night rush. You hire enough servers that no table waits too long at peak (that's the SLO, and it caps how many tables each server can cover). But those same servers are on the clock — and paid — through the dead Tuesday lunch. Your labor cost per meal isn't set by the busy night; it's set by all the quiet hours you're still paying for. A GPU fleet is identical: sized to peak, billed 24×7, and its true $/token driven by the ratio of average load to peak load.

So the planner does three honest things in order: divide peak demand by the SLO-safe per-GPU throughput (plus a headroom margin) to get the fleet size; compute the real average utilization from the average-to-peak ratio, which is low precisely because you sized to peak; then feed that utilization — not 100% — into the section-3 cost formula. The result is a $/token that already includes the idle Tuesday hours. The mistake this kills is quoting the peak-throughput cost as if the GPU ran hot all day; the planner instead surfaces the ugly-but-true number, and the levers to shrink it (autoscale down at night, drain batch work into the troughs, serve peaks on on-demand/spot rather than reserving for them) are exactly the ones you can defend to a finance team.

You cannot simply max out the batch to minimize cost, because p99 latency is a hard constraint (IC4). Capacity planning is: given a peak request rate and a per-GPU throughput that is only valid while you stay under the SLO, how many GPUs do you need — and how much does that fleet cost per million tokens at the utilization the peak/average ratio implies? The planner below sizes the fleet to peak, then reports the honest cost at the resulting average utilization.

capacity.pyimport math

def plan_capacity(peak_tokens_per_sec, per_gpu_tps_at_slo, avg_to_peak_ratio,
                  gpu_hourly, headroom=1.2):
    """Size a GPU fleet to a peak load while respecting a latency SLO.
    per_gpu_tps_at_slo : tokens/sec/GPU you can sustain *without* breaching p99
                         (well below the throughput ceiling from batch_curve).
    avg_to_peak_ratio  : average load / peak load (drives real utilization).
    headroom           : over-provision factor for safety (1.2 = 20% slack).
    """
    needed = peak_tokens_per_sec / per_gpu_tps_at_slo * headroom
    gpus = math.ceil(needed)
    # Average utilization = (avg load) / (provisioned capacity)
    provisioned_tps = gpus * per_gpu_tps_at_slo
    avg_tps = peak_tokens_per_sec * avg_to_peak_ratio
    utilization = avg_tps / provisioned_tps
    monthly_fixed = gpus * gpu_hourly * 24 * 30
    cost_per_mtok = gpu_hourly / (per_gpu_tps_at_slo * 3600 * utilization) * 1_000_000
    return {"gpus": gpus, "avg_util": round(utilization, 3),
            "monthly_$": round(monthly_fixed), "$_per_mtok": round(cost_per_mtok, 2)}

plan = plan_capacity(peak_tokens_per_sec=9000, per_gpu_tps_at_slo=1500,
                     avg_to_peak_ratio=0.4, gpu_hourly=3.00)
print(plan)

{'gpus': 8, 'avg_util': 0.3, 'monthly_$': 17280, '$_per_mtok': 1.85}

Sizing to a 9,000 tok/s peak at a SLO-safe 1,500 tok/s/GPU (well below the batch ceiling) needs 8 GPUs with 20% headroom. But because average load is only 40% of peak and we hold 20% headroom, the fleet averages just 30% utilization — so the real cost is $1.85/1M tokens, more than 5× the $0.33 the full-utilization spreadsheet promised. That gap between the demo number and the SLO-and-diurnal number is exactly what separates a capacity plan that survives contact with production from one that doesn't. To close it: autoscale down at night, drain batch work into the troughs, or serve the peak on on-demand/spot instead of reserving for it.

8 · The tech-lead's serving-cost model tech-lead

Everything so far has been a piece; a lead's job is to hold all the pieces at once and hand finance a single number they can trust. The failure mode here isn't a wrong formula — you've mastered the formulas — it's presenting a $/token that quietly assumed the best case, then owning the fallout when reality delivers the worst. A tech-lead is the person in the room who states the assumption out loud before anyone commits budget to it. That habit, more than any equation, is what this section is really teaching.

Think of it as the difference between a mechanic and a foreman. Each earlier section taught you a tool: measure tokens/sec/GPU, convert to $/1M, find the batching knee, expose utilization, compute break-even, plan capacity under the SLO. A foreman doesn't just own each tool — they own the sequence and the caveats: measure real throughput and real utilization (never the datasheet); quote every optimization in $/1M so batching, quantization, and parallelism compare apples-to-apples; compute the API break-even with ops cost folded in; size to the p99 SLO rather than the throughput ceiling; and pick the reserved/on-demand/spot mix that matches the traffic shape. String those together and the output is a defensible build-vs-buy recommendation, not a spreadsheet cell.

The one non-negotiable running through all of it is stating the utilization assumption out loud, because — as every section has hammered — that is the number that silently breaks naive estimates. A cost of "$0.33/1M" is a lie by omission; "$0.33/1M at 100% utilization, which we'll actually hit ~35% of, so budget ~$0.95" is a number a business can plan around. The habit is cheap and the credibility it buys is enormous.

One practical honesty note the code that follows makes concrete: in the real world you don't hard-code the GPU-hour price, you pull it live from the cloud's pricing API so your model tracks the market. That snippet is genuine infrastructure code — it needs cloud credentials and is explicitly not run by this lesson's offline labs — but it's the shape of how a production cost model stays honest as prices drift. Treat it as the seam where the teaching model meets the real billing system.

A lead owns the whole model end to end: measure real tokens/sec/GPU and real utilization (never trust the datasheet), quote everything in $/1M tokens so batching, quantization, and parallelism can be compared apples-to-apples, compute the API break-even honestly with ops cost folded in, size capacity to the p99 SLO rather than the throughput ceiling, and pick the reserved/on-demand/spot mix that matches the traffic shape. The output is a defensible build-vs-buy recommendation and a $/token you can put in front of finance — with the utilization assumption stated out loud, because that is the number that quietly breaks every naive estimate.

For live pricing you would pull GPU-hour rates from the cloud pricing API rather than hard-coding them. The snippet below sketches that; it is real infrastructure code and is not part of the offline labs above.

live_pricing.py# ▶ needs cloud creds — real infra, NOT run by this lesson's offline labs.
# Requires: pip install boto3, configured AWS credentials, pricing:GetProducts.
import boto3, json

def ondemand_gpu_price(instance_type="p5.48xlarge", region="US East (N. Virginia)"):
    pricing = boto3.client("pricing", region_name="us-east-1")  # Pricing API lives in us-east-1
    resp = pricing.get_products(
        ServiceCode="AmazonEC2",
        Filters=[
            {"Type": "TERM_MATCH", "Field": "instanceType", "Value": instance_type},
            {"Type": "TERM_MATCH", "Field": "location", "Value": region},
            {"Type": "TERM_MATCH", "Field": "tenancy", "Value": "Shared"},
            {"Type": "TERM_MATCH", "Field": "operatingSystem", "Value": "Linux"},
            {"Type": "TERM_MATCH", "Field": "preInstalledSw", "Value": "NA"},
        ],
        MaxResults=1)
    product = json.loads(resp["PriceList"][0])
    terms = product["terms"]["OnDemand"]
    dim = next(iter(next(iter(terms.values()))["priceDimensions"].values()))
    hourly = float(dim["pricePerUnit"]["USD"])
    gpus_per_instance = 8  # p5.48xlarge = 8x H100
    return hourly / gpus_per_instance  # -> feed into cost_per_million_tokens()

# print(round(ondemand_gpu_price(), 2))  # ~ per-GPU-hour, then run the offline model above

def tps_per_gpu(total_tps, gpus):
    return total_tps / gpus

configs = [("A: 1 GPU", 1200, 1), ("B: 4 GPU", 4000, 4), ("C: 8 GPU", 9600, 8)]
ranked = sorted(configs, key=lambda c: tps_per_gpu(c[1], c[2]), reverse=True)
for name, tps, g in ranked:
    print(f"{name:>10}: {tps_per_gpu(tps, g):7.1f} tok/s/GPU")
# C (1200) = A (1200) > B (1000): more total speed can still be worse per GPU

def cost_per_million(gpu_hour, tokens_per_sec):
    per_sec = gpu_hour / 3600.0
    per_token = per_sec / tokens_per_sec
    return per_token * 1_000_000

for name, hr, tps in [("A100 @ 900 tok/s", 2.0, 900),
                      ("H100 @ 2500 tok/s", 4.0, 2500)]:
    c = cost_per_million(hr, tps)
    print(f"{name:>18}: ${c:5.2f}/1M   vs API $3.00 -> "
          f"{'cheaper' if c < 3 else 'pricier'} to self-host")

def throughput_at_batch(b, peak=4000.0, half=8.0):
    # saturating curve: approaches `peak` tok/s, half-saturation at batch=half
    return peak * b / (b + half)

def cost_per_million(gpu_hour, tps):
    return (gpu_hour / 3600.0 / tps) * 1_000_000

print("batch   tok/s    $/1M   marginal $/1M saved")
prev = None
for b in (1, 4, 8, 16, 32, 64, 128):
    tps = throughput_at_batch(b)
    c = cost_per_million(4.0, tps)
    delta = "" if prev is None else f"{prev - c:6.3f}"
    print(f"{b:>5} {tps:8.0f} {c:7.3f}   {delta}")
    prev = c
# savings shrink each doubling -- the knee is where marginal savings stop mattering

def effective_cost(raw_cost_per_m, utilization):
    return raw_cost_per_m / utilization

raw = 1.78   # $/1M at full tilt
for u in (1.0, 0.7, 0.4, 0.2):
    print(f"utilization {u:>4.0%}: true cost = ${effective_cost(raw, u):5.2f}/1M")
# 40% util -> 2.5x the sticker cost -- idle GPUs are the silent budget killer

def break_even_tokens(gpus, gpu_hour, api_per_million):
    # self-host bills a fixed monthly GPU cost regardless of volume;
    # the API bills per token. Break-even is where API cost = fixed cost.
    fixed_month = gpus * gpu_hour * 24 * 30
    api_per_tok = api_per_million / 1_000_000
    return fixed_month / api_per_tok, fixed_month

be, fixed = break_even_tokens(gpus=2, gpu_hour=4.0, api_per_million=3.0)
print(f"fixed self-host: ${fixed:,.0f}/mo")
print(f"break-even: ~{be/1e6:,.1f}M tokens/mo (API cheaper below, self-host above)")

import math

def throughput_at_batch(b, peak=4000.0, half=8.0):
    return peak * b / (b + half)

def latency_ms(b, base=40.0, per_item=6.0):
    return base + per_item * b            # simple linear batch-latency model

def plan(tokens_per_day, p95_ms=800.0, gpu_hour=4.0):
    # largest batch under the SLO
    b = 1
    while latency_ms(b + 1) <= p95_ms:
        b += 1
    tps = throughput_at_batch(b)
    day_capacity = tps * 86400
    gpus = math.ceil(tokens_per_day / day_capacity)
    cost_m = (gpu_hour / 3600.0 / tps) * 1_000_000
    return (f"SLO-capped batch={b}, {tps:.0f} tok/s/GPU, "
            f"gpus={gpus}, ${cost_m:.2f}/1M, ${gpus*gpu_hour*24*30:,.0f}/mo")

print(plan(tokens_per_day=5_000_000_000))
# real p95 & tok/s MUST be measured on the target GPU under load -- needs GPU