Inference & Cost · Project IC

Optimize a serving stack

The capstone: take an open model from a naive baseline to an optimized, benchmarked serving stack — quantize, batch, cache, speculate — and prove the win with numbers.

⏱️ ~5 hours🏗️ Capstone project🎯 Expert

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

a GPU + pip install vllm (serving engine)

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

What you'll buildTake an open model from a naive baseline to an optimized, benchmarked serving stack: quantize it (IC2), serve it on vLLM with continuous batching (IC4) and prompt caching (IC3), optionally speculative decoding (IC5), and prove the win with numbers against IC1's baseline.

Learning objectives

Establish a baseline: TTFT, TPOT, throughput, cost/1k-tokens.
Apply quantization + batching + caching and measure each lever's effect.
Produce a before/after report with the tradeoffs made.
Recommend a production configuration for a stated SLO.

▶ Runnable companionThe code in this lesson is also saved under code/proj-ic-optimize/ in the course, with a README. Run the scripts or copy the configs directly.

The method advanced

Optimize a serving stack, end to end

Baseline (IC1): serve the FP16 model, measure TTFT/TPOT/throughput/cost at concurrency 1 and 32.
Quantize (IC2): switch to AWQ/GPTQ 4-bit; re-measure size, speed, and eval accuracy vs baseline.
Batch (IC4): tune max-num-seqs to your latency SLO; find the throughput knee.
Cache (IC3): add prompt caching for the shared system prompt; measure the cost drop.
Speculate (IC5): if the workload is predictable, add a draft model; check acceptance rate.
Report: a table of each lever's before/after, and a recommended config for the SLO.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

The optimized server

optimized_serve.shpython -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --quantization awq \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching            # reuse shared prefixes' KV (IC3)
# Then run the IC4 load-test harness against it at rising concurrency
# and record the numbers into your before/after table.

▶ How this works

This is the capstone: a single launch command that stacks up almost every optimization from the track at once. Read it as a checklist — each flag is one technique you've already learned, now combined into one production server.

--model mistralai/Mistral-7B-Instruct-v0.3 picks the open model you're optimizing.
--quantization awq applies 4-bit quantization (IC2): smaller and faster to decode.
--max-num-seqs 128 sets the continuous-batching width (IC4): how many requests share each pass.
--gpu-memory-utilization 0.90 gives the engine 90% of GPU memory for weights and the KV-cache (IC3).
--enable-prefix-caching turns on prompt caching (IC3): a shared system prompt or RAG context is prefilled once and reused, not recomputed per request.
The two trailing # comment lines are your homework: point the IC4 load-test harness at this server at rising concurrency and write the results into a before/after table.

What the output means: One optimized server that combines quantization, continuous batching, high memory use, and prefix caching. The real deliverable is the numbers you measure against IC1's plain baseline.

Try this: Turn the flags on one at a time and re-measure after each. That's the whole project: prove what each lever bought you (and what it cost) instead of flipping them all blindly.

Judge every lever against the baselineThe whole point is measured improvement. Each optimization must show its effect on the same metrics — and its cost (quantization's accuracy hit, batching's latency at high concurrency). A recommendation without numbers is an opinion.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Measure a baseline: TTFT, TPOT, throughput, costBeginner

Context: You can't optimize what you haven't measured. The whole project moves four numbers, so capture them first — at concurrency 1 and under load, because a stack that looks fast alone can collapse under concurrency.

Your task: Model a baseline from a load-test's raw timings: TTFT, TPOT, throughput, and cost per 1k output tokens.

Requirements:

Compute average TTFT (time to first token)
Compute TPOT (per-output-token latency)
Compute throughput in tokens/second
Compute cost per 1k output tokens at a set price
Capture at concurrency 1 and 32 so the before/after delta is meaningful

💡 Hint: TPOT is the post-first-token time divided by the remaining tokens; the delta between concurrency levels is the report's whole point.

Show solution

Offline reduction of a load-test into the four numbers the whole project moves:

def baseline(samples, price_per_m=0.60):
    # samples: list of (ttft_ms, out_tokens, total_ms)
    ttft = sum(s[0] for s in samples) / len(samples)
    tpot = sum((s[2]-s[0])/max(s[1]-1,1) for s in samples) / len(samples)
    tot_tokens = sum(s[1] for s in samples)
    tot_secs   = sum(s[2] for s in samples) / 1000
    return {"ttft_ms": round(ttft,1), "tpot_ms": round(tpot,2),
            "throughput_tok_s": round(tot_tokens/tot_secs,1),
            "cost_per_1k": round(1000/1_000_000*price_per_m, 6)}

print(baseline([(400, 200, 5375), (420, 180, 4900)]))
# {'ttft_ms': 410.0, 'tpot_ms': 25.05, 'throughput_tok_s': ~40, 'cost_per_1k': 0.0006}

Capture this at concurrency 1 AND 32 — a stack that looks fast alone can collapse under load, and the report's whole point is the before/after delta.

Exercise 2 · Quantize to 4-bit and account for the memory winIntermediate

Context: Switching to 4-bit halves-or-better the weight footprint (less to read per decode step, more VRAM for KV cache) — but you must re-run the accuracy eval, because 'usually near-lossless' isn't 'always'.

Your task: Model the VRAM before/after and the launch flag, and gate on an accuracy check.

Requirements:

Compute weight VRAM at 16-bit vs 4-bit and the size ratio
Show the memory win (roughly 4x smaller)
An accept-quant gate keeps the quantized model only if accuracy drop is tiny
The vLLM AWQ launch is labelled needs-GPU
Runs the memory math offline

💡 Hint: Weight bytes = params × bits/8; accept only if the accuracy drop is within a small tolerance so you don't quantize away quality.

Show solution

Quantization halves-or-better the weight footprint; the accuracy gate keeps it honest:

def weight_gb(params_b, bits):
    return params_b * 1e9 * (bits/8) / 1024**3

fp16 = weight_gb(7, 16)     # 7B @ 16-bit
awq4 = weight_gb(7, 4)      # 7B @ 4-bit
print(f"fp16 {fp16:.1f} GiB -> awq4 {awq4:.1f} GiB ({fp16/awq4:.1f}x smaller)")
# fp16 13.0 GiB -> awq4 3.3 GiB (4.0x smaller)

def accept_quant(acc_base, acc_quant, max_drop=0.01):
    return (acc_base - acc_quant) <= max_drop     # keep only if drop is tiny

print(accept_quant(0.812, 0.807))   # True

# --- needs GPU: vLLM with AWQ ---
# python -m vllm.entrypoints.openai.api_server \
#   --model mistralai/Mistral-7B-Instruct-v0.3 --quantization awq

Smaller weights mean less to read per decode step (attacking the memory-bandwidth floor) and more VRAM left for KV cache. Always re-run the accuracy eval — 4-bit is usually near-lossless, but "usually" isn't "always".

Exercise 3 · Tune max-num-seqs to the throughput knee under an SLOAdvanced

Context: Batching lifts throughput until latency blows the SLO. Past the knee, each added request pushes p95 over the SLO for no worthwhile throughput gain — so set the flag to the knee, not the max the GPU will accept.

Your task: Sweep max_num_seqs in a throughput/latency model and pick the largest batch that keeps p95 under the SLO.

Requirements:

A model where higher concurrency raises per-token latency but also throughput
Sweep a range of batch sizes
Reject any batch whose p95 latency exceeds the SLO
Pick the largest batch that stays under the SLO (the knee)
The real launch flag is labelled needs-GPU

💡 Hint: Track the best under-SLO throughput as you sweep; the knee is the last batch size before p95 crosses the line.

Show solution

Find the knee offline before you touch the real flag:

def sim(max_seqs, base_tpot_ms=20, slo_ms=8000, out_tokens=300):
    # more concurrency raises per-token latency (contention) but raises throughput
    tpot = base_tpot_ms * (1 + 0.03*max_seqs)
    p95_latency = tpot * out_tokens
    throughput = max_seqs * (1000/tpot)          # tok/s across the batch
    return p95_latency, throughput

best = None
for m in (8, 16, 32, 64, 128, 256):
    lat, thr = sim(m)
    ok = lat <= 8000
    if ok and (best is None or thr > best[1]):
        best = (m, thr)
    print(f"max_num_seqs {m:>3}: p95={lat:6.0f}ms thr={thr:6.0f} {'OK' if ok else 'SLO!'}")
print("knee:", best)     # largest batch under SLO

# --- needs GPU: --max-num-seqs <knee> --gpu-memory-utilization 0.90 ---

Past the knee, each added request pushes p95 over the SLO for no throughput gain worth the latency. Set the flag to the knee, not the max the GPU will accept.

Exercise 4 · Prefix caching: attribute the win to shared contextExpert

Context: When many requests share a long system prompt, prefix caching prefills it once and turns repeated prefill into a cache read. The win scales with prefix length and hit rate — and only exists when requests genuinely share a prefix.

Your task: Model prefix caching and compute the TTFT/compute saved as a function of prefix length and hit rate.

Requirements:

Model prefill cost for the shared prefix plus the unique suffix
On a cache hit, the prefix prefill is skipped
Compute total compute with and without caching over many requests
Report the percentage saved as a function of hit rate and prefix length
The --enable-prefix-caching flag is labelled needs-GPU

💡 Hint: Charge the full prefill only on misses; attribute the TTFT drop to caching explicitly so the lever's value is defensible in the report.

Show solution

Caching the shared prefix converts repeated prefill into a cache read — the win scales with prefix length and hit rate:

def prefill_ms(tokens, per_tok_ms=0.15): return tokens * per_tok_ms

def with_cache(prefix_tokens, unique_tokens, hit_rate, n_reqs):
    cold = prefill_ms(prefix_tokens + unique_tokens)          # no cache
    # on a hit, prefix prefill is skipped:
    hits = int(n_reqs * hit_rate)
    total_no_cache = cold * n_reqs
    total_cache = (cold * (n_reqs - hits)
                   + prefill_ms(unique_tokens) * hits)
    return total_no_cache, total_cache

no, yes = with_cache(prefix_tokens=1800, unique_tokens=60,
                     hit_rate=0.9, n_reqs=100)
print(f"no cache {no:.0f}ms -> cache {yes:.0f}ms ({(1-yes/no)*100:.0f}% saved)")
# ~87% of prefill compute saved when 90% of reqs share the 1800-tok prefix

# --- needs GPU: --enable-prefix-caching ---

Prefix caching only helps when requests genuinely share a prefix (a fixed system prompt, a shared document). Attribute the TTFT drop to it explicitly in the report so the lever's value is defensible.

Exercise 5 · A before/after report with per-lever attributionProfessional

Context: The deliverable is a table that credits each optimization. The delta column tells the next engineer which knob actually paid off, so settled decisions aren't re-litigated.

Your task: Build a report that chains baseline → quant → batch → cache and shows each lever's marginal contribution to throughput and cost.

Requirements:

An ordered chain of stages (baseline, quantize, batch, cache)
Each stage's throughput and cost per 1k
A delta column showing each lever's marginal throughput gain
The framing notes batching is usually the biggest single lever
Runs offline over the stage numbers

💡 Hint: Track the previous stage's throughput to compute each delta; quantization frees the VRAM that lets you batch wider, so order matters.

Show solution

Per-lever attribution is what makes the optimization work reviewable and repeatable:

def report(stages):
    # stages: ordered list of (name, throughput_tok_s, cost_per_1k)
    print(f"{'stage':12} {'thr':>8} {'d-thr':>8} {'$/1k':>10}")
    prev = None
    for name, thr, cost in stages:
        d = "" if prev is None else f"{thr-prev:+.0f}"
        print(f"{name:12} {thr:>8.0f} {d:>8} {cost:>10.5f}")
        prev = thr

report([
    ("baseline", 40,  0.00060),
    ("quantize", 46,  0.00052),
    ("batch=64", 210, 0.00011),
    ("prefix",   240, 0.00010),
])

Batching is usually the biggest single lever; quantization frees the VRAM that lets you batch wider. The delta column tells the next engineer which knob actually paid off, so nobody re-litigates settled decisions.

Exercise 6 · Add speculative decoding for a predictable workloadIndustry scenario

Context: For a narrow, predictable workload, a small draft model proposes tokens the target verifies in parallel. It wins only above a break-even acceptance rate — below it, the draft's cost swamps the benefit.

Your task: Model the acceptance-rate speedup and the break-even where a low acceptance rate makes speculation a net loss.

Requirements:

Model expected accepted tokens per verify step as a function of acceptance rate
Subtract the draft-model overhead
High acceptance yields a real speedup (>1x)
Low acceptance is a net loss (draft overhead exceeds the gain)
Show the speedup across a range of acceptance rates

💡 Hint: Sum the geometric series of accepted tokens over the lookahead and divide by the verify+draft cost; measure the real acceptance rate before enabling it.

Show solution

Speculative decoding trades cheap draft tokens for verified target tokens — it wins only above a break-even acceptance rate:

def spec_speedup(accept_rate, draft_cost_ratio=0.2, lookahead=4):
    # expected accepted tokens per verify step, minus draft overhead
    accepted = sum(accept_rate**i for i in range(1, lookahead+1))
    cost = 1 + draft_cost_ratio          # one target verify + draft passes
    return accepted / cost

for ar in (0.9, 0.7, 0.5, 0.3):
    s = spec_speedup(ar)
    tag = "win" if s > 1 else "LOSS (draft overhead > gain)"
    print(f"accept {ar:.0%}: {s:.2f}x  {tag}")
# high acceptance -> real speedup; low acceptance -> the draft is pure overhead

Speculation pays off on predictable workloads (a domain the draft model models well). Measure the real acceptance rate first: below break-even, the draft model's cost swamps the benefit and you're better off without it.

✓ Checkpoint — you can move on when you can…

Establish a rigorous inference baseline.
Apply quantization, batching, and caching and measure each.
Produce a before/after report with honest tradeoffs.
Recommend a config for a stated latency SLO.

📋 Grade your optimization — is the win real and measured?

Dimension	Meets the bar	Above the bar (staff)
Baseline rigor	A FP16 baseline is measured on TTFT, TPOT, throughput, and cost/1k-tokens at concurrency 1 and 32.	The baseline is reproducible and the harness/warmup/percentiles are documented so numbers are comparable across runs.
Latency/throughput gains	After optimization you show measured improvement on the same metrics vs the baseline.	The throughput knee is found (where p99 breaks the SLO) and gains are reported as a curve, not a single lucky data point.
Cost reduction	$/1k-tokens is recomputed post-optimization and the reduction is quantified against the baseline.	Cost is broken down by lever (quant, batch width, caching) and the cheapest config that meets the SLO is chosen deliberately.
Quality preserved	Quantization's accuracy hit is measured (eval before/after), not assumed to be free.	A quality floor is set and enforced: a config is rejected if it wins on speed but drops eval below the floor.
Per-lever attribution	Each lever (quantize, batch, cache, speculate) is turned on one at a time and its effect measured separately.	You can state what each lever bought and what it cost, and speculative decoding's acceptance rate is checked before crediting it.
Recommendation	You end with a recommended production config for a stated latency SLO, backed by the table.	The recommendation names the deciding tradeoff and the conditions under which you'd choose differently (a recommendation with numbers, not an opinion).

Score each dimension 0 (missing), 1 (meets bar), or 2 (above bar). 0–4: keep building. 5–8: a solid, defensible submission. 9–12: staff-level — you could hand this to a reviewer and defend every call. Any dimension at 0 blocks shipping regardless of the total.

Knowledge check check yourself

✓ Knowledge check

Why does the project require turning optimization levers (quantize, batch, cache, speculate) on one at a time and re-measuring after each?

Show answer

So you can attribute each gain and cost to a specific lever instead of flipping them all blindly. Per-lever attribution is what lets you state what each technique bought and what it cost, rather than guessing.

✓ Knowledge check

Why must quantization's effect be measured on eval accuracy, not just on speed and size?

Show answer

Quantization can silently drop output quality, so its accuracy hit must be measured (eval before/after) rather than assumed free. A config that wins on speed but drops eval below a set quality floor should be rejected.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

The method advanced

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Projects