API vs self-host economics
Self-hosting looks cheaper until you count GPUs and the engineer who runs them. The right answer is a break-even calculation plus the non-cost factors: quality, ops, and data residency.
Learning objectives
- Build an honest cost comparison of hosted API vs self-hosting.
- Find the break-even volume where self-hosting wins.
- Weigh the non-cost factors: quality, latency, ops burden, data residency.
- Make a defensible build-vs-buy recommendation.
The honest comparison advanced
Self-hosting looks cheaper (no per-token fee) but carries real costs: GPU rental/purchase, ops, and the quality gap vs frontier models. The right answer is a calculation, not a preference. This is the AWS cost-model lesson generalized.
This diagram lays out the build-vs-buy decision as a simple money comparison. Read it left to right: your usage feeds two competing cost lines, and where they cross is your answer.
- The first box (Volume + needs) is your actual usage — how many tokens per month you expect to run through the model.
- The second box (Hosted $/token × volume) is the cost of a paid API: you pay per token, so this cost grows as you use more.
- The third box (vs GPU + ops fixed) is the cost of running it yourself: mostly fixed — a GPU plus the people to operate it — and it barely changes whether you run a little or a lot.
- The last box (Break-even) is where those two costs are equal. Below that volume the hosted API is cheaper; above it, self-hosting can win.
In short: A growing per-token bill versus a flat self-host bill. Find where they cross — that volume is your break-even, and the single biggest thing people forget to include is the engineer's time.
Break-even math advanced
Compute your break-even
- Hosted cost/month = tokens/month × $/token (from the provider's pricing).
- Self-host cost/month = GPU (rental or amortized) + electricity + a fraction of an engineer.
- The engineer time is the cost people forget — patching, scaling, incidents don't self-manage.
- Break-even = the monthly volume where the two lines cross. Below it, hosted wins; above, self-host can.
| Factor | Hosted API | Self-host |
|---|---|---|
| Marginal cost | per token | ≈ zero (fixed capacity) |
| Fixed cost | ≈ zero | GPU + ops engineer |
| Best model quality | frontier (Claude, …) | open weights (a step behind) |
| Data residency | provider's infra | fully yours |
| Ops burden | none | you own uptime + scaling |
The non-cost factors expert
Even past break-even, hosted can still win on quality (frontier models) and zero ops. Self-host wins decisively when data cannot leave your infra or you need offline operation — sometimes cost isn't even the deciding factor.
Exercise LM5.1 — Make the call
Context: The deliverable is a decision you can defend — and naming the single factor that actually decided it is the discipline.
Your task: For a realistic workload (pick a tokens/month number), compute the hosted cost and a self-host cost including a fraction of an engineer, find break-even, then state your recommendation and the ONE factor that decided it.
Requirements:
- Pick a concrete monthly token volume
- Compute hosted cost and self-host cost (with the engineer fraction)
- Find the break-even volume
- State a recommendation and the single deciding factor (cost, quality, or residency)
💡 Hint: Often the deciding factor isn't cost at all — residency or quality can override the break-even math.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Hosted inference is a pure variable cost: cost/month = tokens/month × $/token. No volume, no bill — but it scales linearly forever.
Your task: Write the hosted monthly-cost calculator and run it for a couple of volumes.
Requirements:
- Cost = tokens/month / 1000 × a blended $/1k-token price
- Run it at a low and a high monthly volume
- Print the monthly cost for each
- Note that the bill scales linearly with usage
💡 Hint: A one-line formula; the lesson is that hosted cost never stops growing with volume.
Show solution
The per-token bill grows with usage. Runnable:
def hosted_monthly(tokens_per_month, usd_per_1k_tokens):
return tokens_per_month / 1000 * usd_per_1k_tokens
for toks in [5_000_000, 200_000_000]:
cost = hosted_monthly(toks, usd_per_1k_tokens=0.90) # blended in+out price
print(f"{toks:>12,} tok/mo -> ${cost:,.2f}/mo")Hosted is a pure variable cost: no volume, no bill — but it scales linearly and never stops.
Context: Self-hosting is mostly a fixed cost — GPU + electricity + a fraction of an engineer — and the engineer time is the line item teams forget.
Your task: Write the self-host monthly-cost calculator, including the engineer fraction.
Requirements:
- Sum GPU (rental/amortized), electricity, and a fraction of an engineer's salary
- Compute power from GPU wattage, count, hours, and a kWh price
- The engineer fraction is a share of an annual salary per month
- Return the total and a breakdown of the three parts
- Show that people cost often dominates at low volume
💡 Hint: The GPU is visible; patching/scaling/incidents (the engineer fraction) are the hidden cost that frequently dominates when volume is low.
Show solution
Self-host is mostly fixed cost — including people. Runnable:
def selfhost_monthly(gpu_hourly, num_gpus, kwh_price, gpu_watts,
engineer_salary, engineer_fraction):
gpu = gpu_hourly * num_gpus * 24 * 30
power = (gpu_watts/1000) * num_gpus * 24 * 30 * kwh_price
people = engineer_salary/12 * engineer_fraction # the forgotten cost
return gpu + power + people, {"gpu": gpu, "power": round(power), "people": round(people)}
total, parts = selfhost_monthly(
gpu_hourly=1.20, num_gpus=2, kwh_price=0.12, gpu_watts=400,
engineer_salary=180_000, engineer_fraction=0.25)
print(f"${total:,.0f}/mo", parts)The GPU is visible; patching, scaling, and incidents (the engineer fraction) are the line item teams forget — and it often dominates at low volume.
Context: Break-even is the monthly volume where hosted cost equals the fixed self-host cost — a number, not an opinion. Below it hosted wins; above it self-host can.
Your task: Write a solver that returns the crossover token volume.
Requirements:
- Set hosted(volume) equal to the fixed self-host monthly cost and solve for volume
- tokens = fixed / price × 1000
- Return the break-even volume
- Show, for a couple of volumes, which side of break-even they fall on
💡 Hint: Invert the hosted formula for the volume that matches the fixed cost; it's a crossover, so label volumes above and below it.
Show solution
Solve hosted(v) = fixed self-host cost. Runnable:
def breakeven_tokens(usd_per_1k, selfhost_fixed_monthly):
# tokens/1000 * price = fixed -> tokens = fixed/price * 1000
return selfhost_fixed_monthly / usd_per_1k * 1000
be = breakeven_tokens(usd_per_1k=0.90, selfhost_fixed_monthly=8_000)
print(f"break-even ~ {be:,.0f} tokens/month")
for v in [5_000_000, 20_000_000]:
verdict = "self-host can win" if v > be else "hosted wins"
print(f"{v:>12,} tok/mo -> {verdict}")The crossover is a volume, not an opinion: below it the flat self-host bill is wasted; above it the per-token bill overtakes it.
Context: A self-hosted GPU you use 20% of the time still costs 100%. Folding utilization into break-even is what exposes the classic build-vs-buy mistake.
Your task: Redo break-even accounting for utilization — idle capacity raises the effective self-host cost per useful token.
Requirements:
- Divide the fixed self-host cost by utilization to get the effective fixed cost
- Recompute break-even from that effective cost
- Lower utilization pushes break-even higher (hosted stays cheaper longer)
- Show break-even at 100%, 50%, and 20% utilization
💡 Hint: You pay for the whole GPU but only 'utilization' of it does work, so divide the fixed cost by utilization before solving for break-even.
Show solution
Idle GPUs wreck the self-host case. Runnable:
def effective_breakeven(usd_per_1k, fixed_monthly, utilization):
# you pay for the whole GPU but only 'utilization' of it does work
effective_fixed = fixed_monthly / max(utilization, 0.01)
return effective_fixed / usd_per_1k * 1000
for u in [1.0, 0.5, 0.2]:
be = effective_breakeven(0.90, 8_000, u)
print(f"utilization {int(u*100):>3}% -> break-even ~ {be:,.0f} tok/mo")
# lower utilization pushes break-even much higher -> hosted stays cheaper longerSelf-hosting only wins if you keep the GPU busy. At 20% utilization the break-even volume balloons — a half-idle cluster is the classic build-vs-buy mistake.
Context: Cost isn't everything — quality, latency, ops burden, and data residency matter, and sometimes dominate. A weighted scorer makes those tradeoffs explicit.
Your task: Write a weighted scorer that combines the cost verdict with the non-cost factors into a recommendation.
Requirements:
- Score each option (hosted, self-host) on cost, quality, ops, and residency (0–1)
- Apply explicit weights that sum to 1
- Recommend the higher weighted score
- Show that changing weights can flip the recommendation
💡 Hint: A dot product of scores and weights per option; making the weights explicit is the point — frontier quality or strict residency can outweigh raw cost.
Show solution
Combine money with the factors money doesn't capture. Runnable:
def score(option, weights):
return sum(option[k] * weights[k] for k in weights)
WEIGHTS = {"cost": 0.4, "quality": 0.25, "ops": 0.15, "residency": 0.2}
# scores 0-1, higher = better for that option
hosted = {"cost": 0.5, "quality": 1.0, "ops": 1.0, "residency": 0.2}
selfhost = {"cost": 0.9, "quality": 0.7, "ops": 0.3, "residency": 1.0}
sh, ss = score(hosted, WEIGHTS), score(selfhost, WEIGHTS)
print(f"hosted={sh:.2f} self-host={ss:.2f}")
print("recommend:", "self-host" if ss > sh else "hosted")A pure-cost answer ignores that frontier quality, zero-ops, or strict data-residency can dominate the decision. Make the weights explicit.
Context: The whole chapter as one defensible memo: compute both costs at the real volume, find break-even, factor utilization and residency, and emit a decision with the numbers attached.
Your task: Produce the full build-vs-buy recommendation function.
Requirements:
- Compute hosted cost at the given volume
- Compute the effective self-host fixed cost from utilization
- Find break-even and compare the volume to it
- Let a residency requirement force self-host regardless of cost
- Return a decision plus the numbers that justify it
💡 Hint: It's the earlier pieces composed — hosted cost, utilization-adjusted break-even, and a residency override — with the reasons attached so finance and engineering both accept it.
Show solution
The whole chapter as one defensible function. Runnable:
def recommend(tokens_per_month, usd_per_1k, selfhost_fixed, utilization,
residency_required):
hosted = tokens_per_month/1000 * usd_per_1k
eff_fixed = selfhost_fixed / max(utilization, 0.01)
be = eff_fixed / usd_per_1k * 1000
decision = "SELF-HOST" if (tokens_per_month > be or residency_required) else "HOSTED"
reasons = [
f"hosted ~${hosted:,.0f}/mo at this volume",
f"self-host effective fixed ~${eff_fixed:,.0f}/mo (util {int(utilization*100)}%)",
f"break-even ~{be:,.0f} tok/mo",
]
if residency_required:
reasons.append("data residency forces self-host regardless of cost")
return decision, reasons
dec, why = recommend(tokens_per_month=50_000_000, usd_per_1k=0.90,
selfhost_fixed=8_000, utilization=0.6,
residency_required=False)
print(dec)
for r in why: print(" -", r)The right answer is a calculation plus the non-cost overrides (residency), with the numbers attached — not a preference. That is a memo you can defend to finance and to engineering.
✓ Checkpoint — you can move on when you can…
- Build an honest hosted-vs-self-host cost comparison.
- Compute a break-even volume including ops/engineer cost.
- Weigh quality, latency, ops, and data-residency.
- Make and defend a build-vs-buy recommendation.
Knowledge check check yourself
In the API-vs-self-host break-even, what cost do people most often forget, and how does it distort the comparison?
Show answer
Even past the cost break-even point, name factors that can still make a hosted API the right call — and factors that make self-hosting win decisively.