Inference & Cost · Chapter IC1

The inference cost model

Inference — not training — is where the ongoing money goes, and it's memory-bound, not compute-bound. This chapter sets the metrics and mental model the whole track optimizes.

⏱️ ~1.5 hours🧪 1 lab🎯 Beginner→Expert

🌱 Start here — from zero Making models fast & cheap, from scratch — no GPU or systems background assumed — we start with why serving a model even costs money.

When you send a prompt to a model, a computer with a big graphics chip (a GPU) does a lot of math to produce each word, one word at a time. Doing that quickly and cheaply for many users is inference optimization. The surprising part: the GPU spends most of its time moving data around in memory, not doing math — so most tricks in this section are about using memory smarter, not computing faster.

The words you'll hear (in plain terms):

Term	What it actually means
inference	using a trained model to generate output (as opposed to training it).
GPU	a chip good at the parallel math models need; the expensive part of serving.
token	a chunk of text (~¾ of a word) — models read and write in tokens, and you're billed per token.
latency vs throughput	latency = how fast for one user; throughput = how many users at once. They trade off.
quantization	storing the model's numbers in fewer bits so it's smaller and faster — with a small accuracy cost.

What you need before starting:

The MLOps infra chapter gives helpful background but isn't required.
Python basics; the ability to run a script and read its output.
A GPU helps for the hands-on serving labs, but every concept reads fine without one.

New to the topic? Read this box, then take the chapters in order — each section is tagged essential → expert so you always know the depth you're at.

Learning objectives

Explain why LLM serving is memory-bound, not compute-bound, at generation time.
Define the metrics that matter: TTFT, TPOT, throughput, and cost per 1k tokens.
Separate the two phases — prefill vs decode — and why they behave differently.
Reason about the latency-vs-throughput tradeoff every later chapter tunes.

▶ Runnable companionThe code in this lesson is also saved under code/ic1-foundations/ in the course, with a README. Run the scripts or copy the configs directly.

Why inference is the hard part essential

Training gets the headlines; inference is where the ongoing money goes. And it's counterintuitive: generating text is memory-bandwidth-bound, not compute-bound. The GPU spends most of its time moving weights and cache, not doing math — which is why the whole track is about memory and scheduling, not faster matrix multiply.

🗺️ How to read this diagram

This strip shows the life of one request as it flows left to right through a model server. The whole track is about speeding up the last two boxes.

Prompt in (first box) — the text you send. Nothing has been computed yet.
Prefill (parallel) — the model reads your entire prompt in one shot. Because it can work on all prompt tokens at the same time, this stage is fast per token and compute-bound (the GPU's math units are the bottleneck).
Decode (1 tok at a time) — now the model writes the answer one token per step. Each step must re-read the model's weights and its memory of the prompt, so it's memory-bound (moving data, not math, is the bottleneck) and much slower per token.
Tokens out — the finished tokens are streamed back to you as they're produced, which is why chat replies appear word by word.
Read the arrows as "then": prompt → prefill → decode → output. The two middle boxes are colored differently on purpose — they behave completely differently and are optimized with different tricks.

In short: One prefill for the whole prompt, then many small decode steps. Nearly every optimization in this track (KV-cache, batching, speculative decoding) is aimed at the slow decode box.

Prefill vs decode — remember this splitPrefill processes the whole prompt at once (compute-bound, fast per token). Decode generates one token per forward pass, re-reading all weights + KV-cache each time (memory-bound, slow). Almost every optimization targets the decode phase.

The metrics that matter intermediate

Metric	What it measures	Who cares
TTFT	time to first token (≈ prefill)	interactive UX
TPOT	time per output token (decode speed)	streaming feel
Throughput	total tokens/sec across all requests	cost efficiency
Cost / 1k tokens	$ per 1000 tokens served	the bill

The key tension: latency (fast for one user) and throughput (cheap across many users) pull in opposite directions. Batching more requests raises throughput but can raise per-request latency. Every later chapter is a different point on this curve.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab IC1.1 · Measure your baseline

measure.pyimport time
# Minimal latency probe against any OpenAI-compatible endpoint (hosted or local).
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")  # e.g. a vLLM server

t0 = time.time(); first = None; n = 0
stream = client.chat.completions.create(
    model="local-model", stream=True,
    messages=[{"role": "user", "content": "Explain KV-cache in 3 sentences."}],
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        if first is None: first = time.time() - t0      # TTFT
        n += 1
total = time.time() - t0
print(f"TTFT: {first*1000:.0f} ms | TPOT: {(total-first)/max(n-1,1)*1000:.1f} ms | tokens: {n}")

▶ How this works

This little script measures how fast a model server answers, so you have real numbers before you try to make anything faster. It talks to any endpoint that speaks the common OpenAI API shape (a hosted API, or a local server like vLLM).

client = OpenAI(base_url=..., api_key="x") points the client at your server. For a local server the key can be a dummy value like "x".
t0 = time.time() starts a stopwatch. first will hold the time the first token arrived; n counts tokens.
stream=True asks for the reply token by token instead of all at once. The for chunk in stream: loop then receives those pieces as they come.
The first time a chunk of text shows up, we record first = time.time() - t0 — that is TTFT (time to first token, basically the prefill time). Every chunk bumps n.
The final print reports TTFT, then TPOT = the remaining time divided by the other tokens (average time per output token during decode), plus the total token count.

What the output means: One line like TTFT: 210 ms | TPOT: 18.4 ms | tokens: 63. Low TTFT feels snappy to start; low TPOT means the words stream out quickly after that.

Try this: Run it twice against the same endpoint — numbers wobble a little. Then ask a longer question and watch tokens and total time grow while TTFT stays about the same.

Measure before you optimizeYou can't improve what you don't measure. Capture TTFT, TPOT, and tokens/sec on your current setup first — every technique in this track should be judged against that baseline.

Exercise IC1.1 — Profile a real endpoint

Context: The cost model only becomes real once you measure it, and watching throughput rise while per-request latency changes is the latency-vs-throughput tradeoff in action.

Your task: Run the probe against any endpoint you have: record TTFT and TPOT at concurrency 1, then fire 8 requests in parallel and watch throughput rise while per-request latency changes, and plot the tradeoff.

Requirements:

Stream a response and time the first content chunk as TTFT
Compute TPOT from the remaining tokens' timing
Measure at concurrency 1, then at 8 parallel requests
Observe throughput rising as per-request latency changes
Plot the latency-vs-throughput tradeoff

💡 Hint: TTFT is the time to the first streamed chunk; TPOT is the average gap between chunks after that — the Lab IC1.1 probe already measures both.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Cost per 1k output tokensBeginner

Context: Inference is billed per token, so the unit the whole track optimizes is cost per output token — and it scales linearly.

Your task: Given a price of $0.60 per 1M output tokens, write a function that returns the cost of a request producing n output tokens and print the cost of 1,000 tokens.

Requirements:

cost = n / 1_000_000 × price_per_million
Print the cost for 1,000 tokens (and 1M as a sanity check)
Keep the price a parameter with a sensible default
Show cost scales linearly with token count

💡 Hint: It's a single multiply — tokens over a million, times the per-million price.

Show solution

Pure arithmetic — the unit the whole track optimizes:

def cost_usd(n_tokens, price_per_million=0.60):
    return n_tokens / 1_000_000 * price_per_million

print(f"${cost_usd(1000):.6f}")     # $0.000600  -- cost of 1k tokens
print(f"${cost_usd(1_000_000):.2f}")  # $0.60     -- sanity check

Cost scales linearly with tokens, so every token you avoid generating is money saved at fleet scale.

Exercise 2 · TTFT + TPOT = total latencyIntermediate

Context: The two decode metrics are TTFT (time to first token) and TPOT (time per output token); together they set end-to-end latency, and TTFT dominates short replies while TPOT dominates long ones.

Your task: Model end-to-end latency for an n-token response as TTFT + (n−1)×TPOT and print it for a 200-token answer.

Requirements:

Return 0 for a non-positive token count
Use TTFT + (n−1)×TPOT otherwise
Evaluate at n = 1, 50, and 200 tokens
Show TTFT dominates short replies, TPOT dominates long ones
Note streaming hides TPOT from the user

💡 Hint: One token is prefill only (just TTFT); every additional token adds one TPOT, so the formula subtracts one before multiplying.

Show solution

Prefill sets TTFT; each later token costs TPOT. Runnable:

def latency_ms(n_out, ttft_ms=400, tpot_ms=25):
    if n_out <= 0:
        return 0.0
    return ttft_ms + (n_out - 1) * tpot_ms

for n in (1, 50, 200):
    print(n, "tok ->", latency_ms(n), "ms")
# 1 tok -> 400 ms   (just the prefill)
# 200 tok -> 5375 ms
tok_per_s = 1000 / 25
print("decode throughput:", tok_per_s, "tok/s")   # 40.0

TTFT dominates short replies; TPOT dominates long ones. Streaming hides TPOT by showing tokens as they arrive.

Exercise 3 · Prefill is parallel, decode is serialAdvanced

Context: Prefill processes all prompt tokens in roughly one parallel pass; decode runs one serial step per output token — which is why a long prompt is cheap next to a long output.

Your task: Model both phases: prefill as roughly a constant few steps regardless of prompt length, decode as serial per-token steps, then compare long-prompt/short-output vs short-prompt/long-output wall times.

Requirements:

Prefill ≈ a small constant number of steps (parallel/compute-bound)
Decode ≈ out_tokens × step_ms (serial)
Compare a long-prompt/short-output case with a short-prompt/long-output case
Show output length dominates wall time by tens of times
Explain a much longer prompt barely moves wall time

💡 Hint: Prompt length feeds the parallel prefill and nearly cancels out; it's the serial decode loop over output tokens that sets the clock.

Show solution

One deterministic simulation of the two-phase shape:

def wall_ms(prompt_tokens, out_tokens, step_ms=25, prefill_factor=0.4):
    # prefill: all prompt tokens in ~one big parallel step (cheaper per token)
    prefill = prompt_tokens * step_ms * prefill_factor / max(prompt_tokens, 1)
    prefill = step_ms * 3                      # ~a few steps regardless of P (parallel)
    decode = out_tokens * step_ms              # serial: one step per token
    return prefill + decode

a = wall_ms(prompt_tokens=2000, out_tokens=50)   # long prompt, short answer
b = wall_ms(prompt_tokens=50,   out_tokens=2000) # short prompt, long answer
print("long prompt  :", a, "ms")   # ~1325 ms
print("long output  :", b, "ms")   # ~50075 ms
print("output is", round(b / a, 1), "x slower")  # ~37.8x

Prefill is compute-bound and parallel, so a 40x longer prompt barely moves wall time; decode is serial, so output length dominates. This is why the track optimizes decode.

Exercise 4 · Why decode is memory-bound (arithmetic intensity)Expert

Context: Decode re-reads every weight for each token, so the floor on time-per-token is the model size over memory bandwidth (W/BW), independent of FLOPs — decode is memory-bound, and batching is what amortizes that read.

Your task: Compute the TPOT floor for a 14 GB (FP16 7B) model on ~2 TB/s bandwidth, then show batching amortizes it.

Requirements:

Floor per token = W / BW (one full weight read)
Compute it for W ≈ 14 GiB and BW ≈ 2 TB/s (~7 ms/token)
Divide the floor across batch sizes 1/8/32
Show one weight-read serves the whole batch (per-request time falls)
Note quantization (IC2) and batching (IC4) attack exactly this term

💡 Hint: You can't beat W/BW for a single stream, but the same weight read feeds every request in the batch — so divide the floor by batch size.

Show solution

The memory-bandwidth floor is the core insight of IC1:

W  = 14 * 1024**3        # 14 GiB of weights (7B params @ FP16)
BW = 2_000 * 1024**3     # ~2 TB/s HBM bandwidth

floor_s = W / BW         # seconds to read all weights once = one decode step
print(f"TPOT floor: {floor_s*1000:.2f} ms/token")   # ~7.00 ms

# One weight-read serves the whole batch, so per-request cost drops with batch:
for batch in (1, 8, 32):
    per_req = floor_s / batch
    print(f"batch {batch:>2}: {per_req*1000:.3f} ms/token/request")
# batch  1: 7.000   batch 8: 0.875   batch 32: 0.219

Because the bottleneck is moving weights (not math), you cannot beat W/BW per step alone — but batching (IC4) and smaller weights via quantization (IC2) both attack exactly this term.

Exercise 5 · A cost-per-request calculator for capacity planningProfessional

Context: Every serving decision starts from a back-of-envelope cost model, and output tokens dominate the bill because they're both pricier and more numerous.

Your task: Build a planner that, given input/output token counts, prices, and a target QPS, prints cost per request and projected monthly spend.

Requirements:

Cost per request = in×price_in + out×price_out (per-million prices)
Project monthly requests from QPS × seconds/month
Return both per-request cost and monthly spend
Show output tokens dominate the bill
Conclude that trimming verbose generations is the biggest lever

💡 Hint: Two token counts times two per-million prices gives the per-request cost; multiply by requests-per-month for the spend.

Show solution

The spreadsheet every capacity plan starts as, in code:

def plan(in_tok, out_tok, qps, price_in=0.15, price_out=0.60):
    # prices are $/1M tokens
    c_req = (in_tok/1e6)*price_in + (out_tok/1e6)*price_out
    req_month = qps * 60 * 60 * 24 * 30
    return c_req, c_req * req_month

c_req, monthly = plan(in_tok=800, out_tok=250, qps=5)
print(f"cost/request : ${c_req:.6f}")     # $0.000270
print(f"req / month  : {5*60*60*24*30:,}")# 12,960,000
print(f"monthly spend: ${monthly:,.2f}")  # $3,499.20

Output tokens dominate the bill (higher price and usually more of them), so trimming verbose generations is the biggest lever before you touch hardware.

Exercise 6 · Choose a deployment under an SLO and a budgetIndustry scenario

Context: Serving is a constrained-optimization problem: meet an SLO and a budget at a given QPS, and often only the quantized/batched option clears both.

Your task: You must serve a chat feature at p95 TTFT ≤ 500 ms and ≤ $5k/month at ~4 QPS and ~300 output tokens/request. Given three options with cost/latency profiles, write the decision logic and print the pick with its reason.

Requirements:

Model each option as (name, p95 TTFT, monthly cost, note)
Filter to options meeting BOTH the TTFT SLO and the budget
Pick the cheapest survivor
Print the pick + reason, or a “relax one constraint” message if none qualify
Show quantization/batching are what make the SLO affordable

💡 Hint: Filter first on both constraints, then take the min-cost survivor — if the feasible set is empty, you report which constraint to relax.

Show solution

Encode the SLO + budget as a filter, then pick the cheapest survivor — the real deployment decision:

options = [
    # name,            ttft_p95_ms, monthly_usd, note
    ("hosted-api",       350,  6200, "fastest, over budget"),
    ("self-int4-1gpu",   480,  3200, "quantized (IC2), fits 1 GPU"),
    ("self-fp16-2gpu",   300,  7800, "no quant, needs 2 GPUs"),
]
SLO_TTFT, BUDGET = 500, 5000

feasible = [o for o in options if o[1] <= SLO_TTFT and o[2] <= BUDGET]
pick = min(feasible, key=lambda o: o[2]) if feasible else None
print("feasible:", [o[0] for o in feasible])   # ['self-int4-1gpu']
if pick:
    print("PICK:", pick[0], f"(${pick[2]}/mo, {pick[1]} ms p95) —", pick[3])
else:
    print("NO OPTION MEETS SLO + BUDGET — relax one constraint")

The hosted API meets latency but blows the budget; FP16 self-host does too. Quantized single-GPU is the only option inside both constraints — the recurring pattern where IC2/IC4 optimizations are what make the SLO affordable.

✓ Checkpoint — you can move on when you can…

Explain why decode is memory-bound and prefill is compute-bound.
Define TTFT, TPOT, throughput, and cost/1k tokens.
State the latency-vs-throughput tradeoff in one sentence.
Measure a baseline you can optimize against.

Knowledge check check yourself

✓ Knowledge check

Why is LLM decode memory-bandwidth-bound rather than compute-bound, and how does prefill differ?

Show answer

During decode the model generates one token per forward pass and must re-read all its weights (and the KV-cache) each step, so moving data dominates — it's memory-bound and slow per token. Prefill processes the entire prompt in parallel in one pass, saturating the GPU's math units, so it's compute-bound and fast per token.

✓ Knowledge check

Define TTFT and TPOT and say which serving phase each corresponds to.

Show answer

TTFT (time to first token) measures how long until the first token appears, which is essentially the prefill time and drives interactive-start UX. TPOT (time per output token) measures decode speed — the time between subsequent tokens — and drives the streaming feel.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Why inference is the hard part essential

The metrics that matter intermediate

Exercise IC1.1 — Profile a real endpoint

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Inference & cost