Throughput economics

1 · The only cost metric that matters: tokens/sec/GPU advanced

Everything in serving economics reduces to one number: tokens/sec/GPU (also written tok/s/GPU or the aggregate goodput). A GPU costs a fixed number of dollars per hour whether it emits 10 tokens/sec or 10,000. So $/token is just the GPU-hour price divided by how many tokens that GPU produces in an hour. Every other lever in this track — quantization (MS/IC2), the KV-cache (IC3), batching (IC4), tensor/pipeline parallel (MS4) — is worth exactly the tokens/sec/GPU it buys you. Learn to quote everything in this unit.

Order-of-magnitude anchors you should carry in your head (2025-era, single accelerator): an H100 rents for roughly $2–4/GPU-hour on-demand from the big clouds and specialist providers; an A100 is cheaper (~$1–2). A well-batched 7–8B model on one H100 does thousands of output tokens/sec aggregate; a 70B model sharded across GPUs does far fewer per GPU. These are ballparks to reason with — always re-measure your own stack (IC1).

2 · From GPU-hour to $/1M tokens advanced

Here is the whole cost model on one line. Three inputs — the GPU-hour price, the tokens/sec/GPU it sustains, and your utilization (the fraction of paid time the GPU is actually generating billable tokens) — collapse into a single $/1M tokens, which you then hold up against the API's published price.

The arithmetic: one GPU-hour buys tokens_per_sec × 3600 tokens at full utilization. Divide the hourly price by that token count and scale to a million. Then divide by utilization, because tokens you didn't serve still cost you the full hour. If your measured cost sits below the API's $/1M price at your real volume, self-hosting is in the running; if it sits above, the API is simply cheaper and you should stop here.

3 · The $/1M-tokens calculator advanced

This is the model above as code. It is deliberately tiny — the point of throughput economics is that the formula is trivial and the inputs (measured goodput, honest utilization) are where all the judgement lives.

cost_per_mtok.pydef cost_per_million_tokens(gpu_hourly, tokens_per_sec, utilization):
    """$/1M output tokens for a self-hosted GPU.
    gpu_hourly     : $ per GPU-hour (e.g. 3.00 for an H100 on-demand)
    tokens_per_sec : measured aggregate output tokens/sec on that GPU
    utilization    : fraction of paid time actually generating (0<u<=1)
    """
    if tokens_per_sec <= 0 or not (0 < utilization <= 1):
        raise ValueError("need tokens_per_sec>0 and 0<utilization<=1")
    tokens_per_hour = tokens_per_sec * 3600 * utilization
    dollars_per_token = gpu_hourly / tokens_per_hour
    return dollars_per_token * 1_000_000

for u in (1.0, 0.5, 0.2):
    c = cost_per_million_tokens(gpu_hourly=3.00, tokens_per_sec=2500, utilization=u)
    print(f"util {u:>4.0%}  ->  ${c:6.2f} / 1M tokens")

util 100%  ->  $  0.33 / 1M tokens
util  50%  ->  $  0.67 / 1M tokens
util  20%  ->  $  1.67 / 1M tokens

At full utilization an H100 doing 2,500 tok/s costs about $0.33 per million output tokens — cheaper than most frontier APIs. But halve utilization and the cost doubles; drop to 20% and you're at $1.67. The GPU-hour price never changed. Utilization is the entire story, which is why it gets its own section.

4 · Batching: the throughput→cost curve and its knee expert

Decode is memory-bound: every token re-reads the model's weights (IC1/IC3). Batching amortizes that one weight-read across many concurrent requests (IC4), so aggregate tokens/sec/GPU climbs steeply with batch size — and $/token falls in lockstep. But it does not climb forever. Two ceilings bite: the KV-cache runs out of GPU memory (you can't hold more sequences), and per-request latency (TPOT) rises as the batch competes for compute. The knee is where throughput flattens; pushing past it buys almost no cost savings while blowing your latency SLO.

batch_curve.pydef throughput_at_batch(batch, per_stream_toks=55, knee=32, ceiling=3200):
    """Toy model of aggregate tokens/sec vs batch size.
    Near-linear early (weight-read amortized), saturating past the knee
    where KV-cache/compute cap the GPU. Illustrative shape, not a spec."""
    ideal = per_stream_toks * batch                 # if it scaled forever
    saturating = ceiling * (batch / (batch + knee))  # diminishing returns
    return round(min(ideal, saturating))

def cost_per_mtok(gpu_hourly, tps):
    return gpu_hourly / (tps * 3600) * 1_000_000

print("batch  tok/s   $/1M   marginal $/1M saved")
prev = None
for b in (1, 4, 8, 16, 32, 64, 128):
    tps = throughput_at_batch(b)
    c = cost_per_mtok(3.00, tps)
    delta = "" if prev is None else f"{prev - c:+.3f}"
    print(f"{b:>4}  {tps:>5}  ${c:6.3f}   {delta}")
    prev = c

batch  tok/s   $/1M   marginal $/1M saved
   1     55  $15.152   
   4    220  $ 3.788   +11.364
   8    440  $ 1.894   +1.894
  16    880  $ 0.947   +0.947
  32   1600  $ 0.521   +0.426
  64   2133  $ 0.391   +0.130
 128   2560  $ 0.326   +0.065

Read the last column. Going from batch 1→4 saves $11/1M tokens; 32→64 saves 13¢; 64→128 saves 6¢ — and every step up the batch adds queueing and TPOT latency. The economically interesting region is the early, steep part; beyond the knee (~32 here) you are trading real latency for pennies. Batch to the knee, not to the ceiling.

5 · Utilization — the hidden cost expert

A reserved or on-demand GPU bills for wall-clock time, not for tokens. Every second it sits idle — overnight, between traffic bursts, waiting on a slow upstream, or over-provisioned "just in case" — is billed at 100% and produces zero tokens. This is why two teams running the same model on the same GPU can have a 5× difference in $/token: one runs it hot, the other runs it at 20% average utilization. Diurnal traffic (busy 9–5, dead at night) alone can cap your realistic average utilization near 30–40% unless you autoscale, batch offline work into the troughs, or rent by the token.

The purchasing mode is a second utilization lever. Reserved capacity is cheap per hour only if you keep it busy — an idle reserved GPU is the worst of both worlds. Spot is the cheapest raw price but can vanish mid-request, so it fits batch/offline (AP1-style) work, not your latency-SLO online path. A common pattern: reserved for the predictable baseline, on-demand to absorb peaks, spot for overnight batch.

6 · Self-host vs API: the break-even volume expert

Self-hosting converts a per-token cost (the API bill, which scales with usage) into a fixed monthly cost (GPUs you rent whether busy or not) plus a small marginal per-token cost. The API is a flat line through the origin; self-host is a line with a big y-intercept and a gentle slope. They cross at one volume — your break-even. Below it the API wins on cost; above it self-host can win, if you actually reach the utilization your fixed cost assumed. This is the GPU-serving version of the general build-vs-buy from LM5.

breakeven.pydef breakeven_volume(api_price_per_mtok, gpu_monthly_fixed, self_host_per_mtok):
    """Monthly volume (in millions of tokens) where self-host total cost
    equals the API bill.  api = api_price * V ;  self = fixed + shp * V.
    Returns the crossover V in M-tokens/month, or None if self-host never wins."""
    slope_gap = api_price_per_mtok - self_host_per_mtok
    if slope_gap <= 0:
        return None  # API's per-token price already <= our marginal cost
    return gpu_monthly_fixed / slope_gap

def monthly(api, fixed, shp, V):
    return api * V, fixed + shp * V   # (api_bill, self_host_bill) at V M-tokens

# One reserved H100 ~ $1.80/hr committed -> ~$1,314/month fixed.
# Self-host marginal cost ~ $0.33/1M (from section 3). API list ~ $3.00/1M.
be = breakeven_volume(api_price_per_mtok=3.00, gpu_monthly_fixed=1314,
                      self_host_per_mtok=0.33)
print(f"break-even at ~{be:,.0f} M tokens/month")
for V in (100, be, 1000):
    api_bill, sh_bill = monthly(3.00, 1314, 0.33, V)
    winner = "API" if api_bill < sh_bill else "self-host"
    print(f"  V={V:>7,.0f}M  API=${api_bill:>8,.0f}  self=${sh_bill:>8,.0f}  -> {winner}")

break-even at ~492 M tokens/month
  V=    100M  API=$     300  self=$   1,347  -> API
  V=    492M  API=$   1,476  self=$   1,476  -> self-host
  V=  1,000M  API=$   3,000  self=$   1,644  -> self-host

Below ~492M tokens/month the single reserved GPU's fixed cost isn't amortized and the API is cheaper; above it self-host pulls ahead, and at 1B tokens/month it's nearly 2× cheaper. But note the trap: that $1,314 fixed cost assumes the GPU is busy enough to serve 492M tokens. If diurnal traffic leaves it 30% utilized, your effective marginal cost triples and the break-even volume moves far to the right. Break-even and utilization are the same conversation.

7 · Capacity planning under a latency SLO professional

You cannot simply max out the batch to minimize cost, because p99 latency is a hard constraint (IC4). Capacity planning is: given a peak request rate and a per-GPU throughput that is only valid while you stay under the SLO, how many GPUs do you need — and how much does that fleet cost per million tokens at the utilization the peak/average ratio implies? The planner below sizes the fleet to peak, then reports the honest cost at the resulting average utilization.

capacity.pyimport math

def plan_capacity(peak_tokens_per_sec, per_gpu_tps_at_slo, avg_to_peak_ratio,
                  gpu_hourly, headroom=1.2):
    """Size a GPU fleet to a peak load while respecting a latency SLO.
    per_gpu_tps_at_slo : tokens/sec/GPU you can sustain *without* breaching p99
                         (well below the throughput ceiling from batch_curve).
    avg_to_peak_ratio  : average load / peak load (drives real utilization).
    headroom           : over-provision factor for safety (1.2 = 20% slack).
    """
    needed = peak_tokens_per_sec / per_gpu_tps_at_slo * headroom
    gpus = math.ceil(needed)
    # Average utilization = (avg load) / (provisioned capacity)
    provisioned_tps = gpus * per_gpu_tps_at_slo
    avg_tps = peak_tokens_per_sec * avg_to_peak_ratio
    utilization = avg_tps / provisioned_tps
    monthly_fixed = gpus * gpu_hourly * 24 * 30
    cost_per_mtok = gpu_hourly / (per_gpu_tps_at_slo * 3600 * utilization) * 1_000_000
    return {"gpus": gpus, "avg_util": round(utilization, 3),
            "monthly_$": round(monthly_fixed), "$_per_mtok": round(cost_per_mtok, 2)}

plan = plan_capacity(peak_tokens_per_sec=9000, per_gpu_tps_at_slo=1500,
                     avg_to_peak_ratio=0.4, gpu_hourly=3.00)
print(plan)

{'gpus': 8, 'avg_util': 0.3, 'monthly_$': 17280, '$_per_mtok': 1.85}

Sizing to a 9,000 tok/s peak at a SLO-safe 1,500 tok/s/GPU (well below the batch ceiling) needs 8 GPUs with 20% headroom. But because average load is only 40% of peak and we hold 20% headroom, the fleet averages just 30% utilization — so the real cost is $1.85/1M tokens, more than 5× the $0.33 the full-utilization spreadsheet promised. That gap between the demo number and the SLO-and-diurnal number is exactly what separates a capacity plan that survives contact with production from one that doesn't. To close it: autoscale down at night, drain batch work into the troughs, or serve the peak on on-demand/spot instead of reserving for it.

8 · The tech-lead's serving-cost model tech-lead

A lead owns the whole model end to end: measure real tokens/sec/GPU and real utilization (never trust the datasheet), quote everything in $/1M tokens so batching, quantization, and parallelism can be compared apples-to-apples, compute the API break-even honestly with ops cost folded in, size capacity to the p99 SLO rather than the throughput ceiling, and pick the reserved/on-demand/spot mix that matches the traffic shape. The output is a defensible build-vs-buy recommendation and a $/token you can put in front of finance — with the utilization assumption stated out loud, because that is the number that quietly breaks every naive estimate.

For live pricing you would pull GPU-hour rates from the cloud pricing API rather than hard-coding them. The snippet below sketches that; it is real infrastructure code and is not part of the offline labs above.

live_pricing.py# ▶ needs cloud creds — real infra, NOT run by this lesson's offline labs.
# Requires: pip install boto3, configured AWS credentials, pricing:GetProducts.
import boto3, json

def ondemand_gpu_price(instance_type="p5.48xlarge", region="US East (N. Virginia)"):
    pricing = boto3.client("pricing", region_name="us-east-1")  # Pricing API lives in us-east-1
    resp = pricing.get_products(
        ServiceCode="AmazonEC2",
        Filters=[
            {"Type": "TERM_MATCH", "Field": "instanceType", "Value": instance_type},
            {"Type": "TERM_MATCH", "Field": "location", "Value": region},
            {"Type": "TERM_MATCH", "Field": "tenancy", "Value": "Shared"},
            {"Type": "TERM_MATCH", "Field": "operatingSystem", "Value": "Linux"},
            {"Type": "TERM_MATCH", "Field": "preInstalledSw", "Value": "NA"},
        ],
        MaxResults=1)
    product = json.loads(resp["PriceList"][0])
    terms = product["terms"]["OnDemand"]
    dim = next(iter(next(iter(terms.values()))["priceDimensions"].values()))
    hourly = float(dim["pricePerUnit"]["USD"])
    gpus_per_instance = 8  # p5.48xlarge = 8x H100
    return hourly / gpus_per_instance  # -> feed into cost_per_million_tokens()

# print(round(ondemand_gpu_price(), 2))  # ~ per-GPU-hour, then run the offline model above

def tps_per_gpu(total_tps, gpus):
    return total_tps / gpus

configs = [("A: 1 GPU", 1200, 1), ("B: 4 GPU", 4000, 4), ("C: 8 GPU", 9600, 8)]
ranked = sorted(configs, key=lambda c: tps_per_gpu(c[1], c[2]), reverse=True)
for name, tps, g in ranked:
    print(f"{name:>10}: {tps_per_gpu(tps, g):7.1f} tok/s/GPU")
# C (1200) = A (1200) > B (1000): more total speed can still be worse per GPU

def cost_per_million(gpu_hour, tokens_per_sec):
    per_sec = gpu_hour / 3600.0
    per_token = per_sec / tokens_per_sec
    return per_token * 1_000_000

for name, hr, tps in [("A100 @ 900 tok/s", 2.0, 900),
                      ("H100 @ 2500 tok/s", 4.0, 2500)]:
    c = cost_per_million(hr, tps)
    print(f"{name:>18}: ${c:5.2f}/1M   vs API $3.00 -> "
          f"{'cheaper' if c < 3 else 'pricier'} to self-host")

def throughput_at_batch(b, peak=4000.0, half=8.0):
    # saturating curve: approaches `peak` tok/s, half-saturation at batch=half
    return peak * b / (b + half)

def cost_per_million(gpu_hour, tps):
    return (gpu_hour / 3600.0 / tps) * 1_000_000

print("batch   tok/s    $/1M   marginal $/1M saved")
prev = None
for b in (1, 4, 8, 16, 32, 64, 128):
    tps = throughput_at_batch(b)
    c = cost_per_million(4.0, tps)
    delta = "" if prev is None else f"{prev - c:6.3f}"
    print(f"{b:>5} {tps:8.0f} {c:7.3f}   {delta}")
    prev = c
# savings shrink each doubling -- the knee is where marginal savings stop mattering

def effective_cost(raw_cost_per_m, utilization):
    return raw_cost_per_m / utilization

raw = 1.78   # $/1M at full tilt
for u in (1.0, 0.7, 0.4, 0.2):
    print(f"utilization {u:>4.0%}: true cost = ${effective_cost(raw, u):5.2f}/1M")
# 40% util -> 2.5x the sticker cost -- idle GPUs are the silent budget killer

def break_even_tokens(gpus, gpu_hour, api_per_million):
    # self-host bills a fixed monthly GPU cost regardless of volume;
    # the API bills per token. Break-even is where API cost = fixed cost.
    fixed_month = gpus * gpu_hour * 24 * 30
    api_per_tok = api_per_million / 1_000_000
    return fixed_month / api_per_tok, fixed_month

be, fixed = break_even_tokens(gpus=2, gpu_hour=4.0, api_per_million=3.0)
print(f"fixed self-host: ${fixed:,.0f}/mo")
print(f"break-even: ~{be/1e6:,.1f}M tokens/mo (API cheaper below, self-host above)")

import math

def throughput_at_batch(b, peak=4000.0, half=8.0):
    return peak * b / (b + half)

def latency_ms(b, base=40.0, per_item=6.0):
    return base + per_item * b            # simple linear batch-latency model

def plan(tokens_per_day, p95_ms=800.0, gpu_hour=4.0):
    # largest batch under the SLO
    b = 1
    while latency_ms(b + 1) <= p95_ms:
        b += 1
    tps = throughput_at_batch(b)
    day_capacity = tps * 86400
    gpus = math.ceil(tokens_per_day / day_capacity)
    cost_m = (gpu_hour / 3600.0 / tps) * 1_000_000
    return (f"SLO-capped batch={b}, {tps:.0f} tok/s/GPU, "
            f"gpus={gpus}, ${cost_m:.2f}/1M, ${gpus*gpu_hour*24*30:,.0f}/mo")

print(plan(tokens_per_day=5_000_000_000))
# real p95 & tok/s MUST be measured on the target GPU under load -- needs GPU