Inference & Cost · Chapter IC4

Continuous batching & scheduling

Q: Why does batching raise throughput far more than it raises latency for memory-bound decode?

Decode's cost is dominated by reading the model's weights out of memory once per step; a single request leaves the GPU mostly idle waiting on that read. Batching many requests through the same weight-read amortizes that expensive memory access across all of them, so each extra request is nearly free — throughput rises sharply while per-request latency grows only modestly.

Q: Contrast static batching with continuous (in-flight) batching and explain why the latter keeps the GPU busier.

Static batching collects a fixed batch, runs it to completion, then starts the next — but requests finish at different lengths, so the GPU idles on the stragglers. Continuous batching adds and removes requests every decode step: as one sequence finishes, a new one takes its slot, keeping the GPU saturated, which is how vLLM/TGI get their throughput.

Batching amortizes the expensive weight-read across many requests — the biggest throughput lever. Continuous batching keeps the GPU saturated by swapping requests every step.

⏱️ ~1.5 hours🧪 1 lab🎯 Advanced

Learning objectives

Explain why batching is the biggest throughput lever for memory-bound decode.
Distinguish static batching from continuous (in-flight) batching.
Reason about how a scheduler packs requests to keep the GPU busy.
Tune batch/concurrency settings against your latency budget.

▶ Runnable companionThe code in this lesson is also saved under code/ic4-batching/ in the course, with a README. Run the scripts or copy the configs directly.

Why batching wins intermediate

Decode re-reads all the model's weights for every token (IC1). If you're doing that for one request, the GPU is mostly idle waiting on memory. Batch many requests through the same weight-read and you serve them almost for free — throughput rises far more than latency. Batching is the single biggest efficiency lever in serving.

🗺️ How to read this diagram

This shows why serving many users at once is cheaper per user. The costly part of each step is reading the model's weights out of memory; batching lets one weight-read serve lots of requests at the same time.

Many requests (concurrent users) — several people are asking the model something in the same window of time.
Scheduler (packs them) — a traffic controller inside the server groups those requests together into a single batch so they can ride along on the same computation.
One batched forward pass (shared compute) — the model reads its weights once and produces the next token for every request in the batch together. The expensive memory read is shared, so each extra request is nearly free.
Tokens to all (streamed back) — each user gets their own token from that shared pass, streamed back to them.
Read left to right: gather users → pack into a batch → one shared pass → everyone gets output.

In short: Batching = many requests share one weight-read. That's why throughput (users served) shoots up while the cost per user drops — the biggest efficiency lever in this whole track.

Static vs continuous batching intermediate

Static batching waits to collect a fixed batch, runs it to completion, then starts the next — but requests finish at different lengths, so the GPU idles on the stragglers. Continuous (in-flight) batching adds and removes requests every step: as one finishes, a new one takes its slot. This keeps the GPU saturated and is why modern servers (vLLM/TGI) get their throughput.

	Static batching	Continuous batching
When batch changes	per batch	every decode step
GPU utilization	drops on stragglers	stays high
Throughput	good	much better
Used by	naive servers	vLLM, TGI

You mostly configure, not implement, thisYou won't write a scheduler — vLLM/TGI (IC6) do it. Your job is to understand it so you can tune it: max batch size, max concurrent sequences, and how they trade latency for throughput on your traffic.

Tuning to a latency budget advanced

How to tune batching for your workload

Set your latency SLO (e.g. p95 TPOT < 50 ms) — the constraint everything serves.
Raise max concurrent sequences until throughput plateaus or you breach the SLO.
Watch KV-cache memory (IC3): too many sequences and you'll OOM or start evicting.
For bursty traffic, cap concurrency so a spike can't blow the latency budget for everyone.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab IC4.1 · Load-test concurrency

loadtest.pyimport asyncio, time
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")

async def one():
    t0 = time.time()
    await client.chat.completions.create(
        model="local-model", max_tokens=128,
        messages=[{"role": "user", "content": "Summarize continuous batching."}])
    return time.time() - t0

async def bench(concurrency):
    t0 = time.time()
    lat = await asyncio.gather(*[one() for _ in range(concurrency)])
    wall = time.time() - t0
    print(f"conc={concurrency:3d}  p95={sorted(lat)[int(len(lat)*0.95)-1]:.2f}s  "
          f"throughput={concurrency/wall:.1f} req/s")

for c in (1, 4, 16, 64):
    asyncio.run(bench(c))   # watch throughput climb and p95 latency rise

▶ How this works

This script load-tests a server by firing many requests at once and measuring what happens. It shows the core tradeoff of batching: as you send more requests together, you serve more per second (throughput up) but each one can take a bit longer (latency up).

It uses AsyncOpenAI and asyncio so many requests can be in flight at the same time instead of one after another — that's how we create real concurrency.
async def one(): sends a single request and returns how long it took (time.time() - t0). That per-request time is its latency.
bench(concurrency) launches concurrency copies of one() at once with asyncio.gather(...), waits for them all, and times the whole thing (wall).
The print shows p95 latency (sorted(lat)[...0.95...] — the slow-ish request that 95% of users beat) and throughput (concurrency/wall = requests finished per second).
The loop for c in (1, 4, 16, 64): repeats the test at rising concurrency so you can watch the two numbers move in opposite directions.

What the output means: Four lines, one per concurrency level. Throughput (req/s) climbs as concurrency rises; p95 latency stays flat at first, then starts rising once the server is saturated.

Try this: Find the knee: the concurrency where throughput stops climbing much but p95 latency starts spiking. That point is roughly the most load this server should take before users feel it.

Exercise IC4.1 — Find your knee

Context: Every served model has a “knee” where latency spikes for little further throughput, and that knee — not a theoretical max — is your capacity setting.

Your task: Load-test a served model at rising concurrency, plot throughput and p95 latency vs concurrency, and find the knee past which latency spikes for little throughput gain.

Requirements:

Drive rising concurrency against a served model
Record throughput and p95 latency at each level
Plot both against concurrency
Identify the knee where latency spikes for little throughput gain
Use that knee as the capacity setting

💡 Hint: The knee is where the throughput curve flattens while the latency curve turns sharply up — read it off the two plots together.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Throughput rises with batch sizeBeginner

Context: Batching shares one weight-read across many requests, and since a decode step is memory-bound its time barely changes with batch — so throughput rises almost linearly.

Your task: Model it simply: a step takes base_ms regardless of batch, so throughput = batch / step-time; print tokens/sec for batch 1, 8, 32.

Requirements:

throughput = batch / (step_ms / 1000)
Treat step time as constant with batch (memory-bound)
Print tok/s for batches 1, 8, 32
Show throughput rising almost linearly with batch
Frame batching as the single biggest efficiency lever

💡 Hint: If the step time is fixed, throughput is just batch over step time — the linear rise is the whole insight.

Show solution

The core batching win in three lines:

def throughput(batch, step_ms=25):
    return batch / (step_ms / 1000)   # tokens per second (1 tok/req/step)

for b in (1, 8, 32):
    print(f"batch {b:>2}: {throughput(b):.0f} tok/s")
# batch  1:   40 tok/s
# batch  8:  320 tok/s
# batch 32: 1280 tok/s

Because the step time barely changes with batch (it is dominated by reading weights), throughput rises almost linearly — the single biggest efficiency lever in serving.

Exercise 2 · Latency vs throughput tradeoffIntermediate

Context: Bigger batches raise throughput but each request waits for the whole batch, so batch size is a knob you set against an SLO, not one you maximize blindly.

Your task: Model per-request latency as step-time growing slightly with batch and show the classic curve — throughput up, per-request latency up.

Requirements:

Make step time grow slowly with batch (a small per-extra term)
Loop over batches 1/8/32/128
Print step time, per-token latency, and throughput each
Show throughput climbs ~40× while latency only ~triples
Conclude batch size is set against an SLO, not maximized

💡 Hint: Let step time creep up gently with batch — throughput still wins big, but the latency creep is why you tune rather than max the batch.

Show solution

Show both curves so the tradeoff is explicit:

def step_ms(batch, base=25, per_extra=0.4):
    return base + batch * per_extra    # step grows slowly with batch

for b in (1, 8, 32, 128):
    s = step_ms(b)
    thru = b / (s/1000)
    print(f"batch {b:>3}: step {s:5.1f} ms | latency {s:5.1f} ms/tok | {thru:6.0f} tok/s")
# batch   1: step  25.4 |  25.4 ms/tok |    39 tok/s
# batch 128: step  76.2 |  76.2 ms/tok |  1680 tok/s

Throughput climbs ~43x from batch 1 to 128 while per-token latency only triples — but that latency growth is real, so the batch size is a knob you set against an SLO, not maximize blindly.

Exercise 3 · Static vs continuous batching utilizationAdvanced

Context: Static batching waits for the slowest request in the batch; continuous batching swaps a finished slot immediately — which is where vLLM/TGI get their throughput.

Your task: Simulate a batch of requests with different output lengths and compare GPU-step utilization for static vs continuous batching.

Requirements:

Use output lengths with one straggler (e.g. [3,3,3,10])
Static utilization = useful work / (requests × max length) ≈ ~half
Continuous utilization ≈ ~100% (finished slots freed and refilled)
Explain short requests hold idle slots under static batching
Connect continuous batching to vLLM/TGI throughput

💡 Hint: Under static batching the short requests sit idle until the longest finishes; continuous batching refills those slots the moment they free up.

Show solution

A deterministic sim of the two schedulers over discrete steps:

lengths = [3, 3, 3, 10]   # output tokens per request; one straggler

# Static: batch runs until the LONGEST finishes; short reqs idle their slot after done
steps_static = max(lengths)
work = sum(lengths)
slots = len(lengths) * steps_static
print("static:   util =", f"{work/slots*100:.0f}%",  # 47%
      f"({work} useful / {slots} slot-steps)")

# Continuous: a finished slot is refilled; here no queue, so idle slots simply free up.
# Utilization = useful work / (slot-steps actually occupied)
occupied = sum(lengths)   # each slot occupied exactly as long as it runs
print("continuous: util =", f"{work/occupied*100:.0f}%")  # 100% of occupied slot-steps

The straggler forces static batching to hold idle slots for 7 extra steps, wasting ~53% of the GPU. Continuous batching never holds a finished slot idle, which is why vLLM/TGI get their throughput.

Exercise 4 · Size the batch to a latency budgetExpert

Context: Choosing the batch size is the actual tuning task: find the largest batch that still meets a p95 TPOT budget — the math behind vLLM's max-num-seqs.

Your task: Given a p95 TPOT budget and the step-time-vs-batch model, find the largest batch that still meets the budget.

Requirements:

Reuse the step-time-grows-with-batch model
Increase batch while the step time stays within the budget
Sweep several budgets and report each max batch
Show a looser budget permits a bigger batch and more throughput
Tie the result to vLLM's max-num-seqs

💡 Hint: Walk the batch up until the modeled step time would breach the budget, then stop — the last passing batch is your max-num-seqs.

Show solution

Search the knob against the SLO — exactly the tuning move:

def step_ms(batch, base=25, per_extra=0.4):
    return base + batch * per_extra

def max_batch_under(budget_ms):
    b = 1
    while step_ms(b + 1) <= budget_ms:
        b += 1
    return b, round(step_ms(b), 1)

for budget in (40, 60, 100):
    b, s = max_batch_under(budget)
    print(f"budget {budget} ms/tok -> max batch {b} (step {s} ms)")
# budget 40 -> max batch 37
# budget 100 -> max batch 187

A looser latency budget lets you pack a bigger batch and win more throughput. You configure max-num-seqs; this is the math behind choosing it.

Exercise 5 · Configure continuous batching in vLLM (real, needs GPU)Professional

Context: Continuous batching is on by default in vLLM — you don't write the scheduler, you tune two batch knobs against your SLO.

Your task: Map the tuning knobs to real vLLM flags: show the launch with the batching-relevant flags, note which chapter each maps to, and label it as needing a GPU.

Requirements:

Launch the vLLM OpenAI-compatible server
Set --max-num-seqs (IC4: max concurrent sequences)
Set --max-num-batched-tokens (IC4: token budget per step)
Set --gpu-memory-utilization (IC3: VRAM for weights + KV)
Note continuous batching is default: raise for throughput, lower if p95 breaches

💡 Hint: You configure, not code — the two batch flags plus the memory-utilization flag are the whole tuning surface.

Show solution

The real launch — needs a GPU + pip install vllm:

# serve.sh — continuous batching is on by default in vLLM
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --max-num-seqs 128 \          # IC4: max concurrent sequences in a batch
  --max-num-batched-tokens 8192 \  # IC4: token budget per scheduler step
  --gpu-memory-utilization 0.90    # IC3: VRAM for weights + KV-cache

# Then point any OpenAI client at http://localhost:8000/v1

You do not write the scheduler — vLLM does continuous batching for you. Your job is the two batch knobs: raise them for throughput, lower them if p95 latency breaches the SLO.

Exercise 6 · Tune a service that misses its p95 under a traffic spikeIndustry scenario

Context: Under a traffic spike, latency and capacity are separate problems: shrink the batch to protect p95, then scale replicas horizontally for QPS — one knob can't fix both.

Your task: At peak QPS your p95 TPOT breaches 60 ms and the queue grows. Given a batch model, decide whether to lower max batch (protect latency) or add replicas (protect throughput) and print a reasoned plan.

Requirements:

Show the current batch breaches the p95 budget
Find the largest safe batch under the budget
Compute per-replica throughput vs required throughput (peak QPS × out tokens)
Ceil-divide to size the replica count
Conclude: shrink batch for latency, scale replicas for QPS

💡 Hint: The safe batch fixes p95; the throughput gap it leaves is closed by replicas — two levers for two independent problems.

Show solution

Diagnose whether the pressure is latency or capacity, then act:

def step_ms(batch, base=25, per_extra=0.4):
    return base + batch * per_extra

peak_qps, out_tok = 40, 200
current_batch = 128
p95 = step_ms(current_batch)
print("current p95 step:", p95, "ms")   # 76.2 ms -> breaches 60 ms

# Option A: cap batch to meet latency
safe_batch = 1
while step_ms(safe_batch+1) <= 60: safe_batch += 1
cap_thru = safe_batch / (step_ms(safe_batch)/1000)
need_thru = peak_qps * out_tok
print(f"cap batch to {safe_batch}: {cap_thru:.0f} tok/s vs need {need_thru} tok/s")
replicas = -(-need_thru // int(cap_thru))
print(f"=> cap batch AND run {replicas} replicas to hold both SLO and QPS")

Capping the batch fixes p95 but drops per-replica throughput below demand, so latency and capacity are separate problems: shrink the batch to protect p95, then scale replicas horizontally to carry the QPS. One knob cannot fix both.

✓ Checkpoint — you can move on when you can…

Explain why batching raises throughput more than latency.
Contrast static and continuous batching.
Describe how a scheduler keeps the GPU saturated.
Tune concurrency to a latency SLO and find the knee.

Knowledge check check yourself

✓ Knowledge check

Why does batching raise throughput far more than it raises latency for memory-bound decode?

Show answer

Decode's cost is dominated by reading the model's weights out of memory once per step; a single request leaves the GPU mostly idle waiting on that read. Batching many requests through the same weight-read amortizes that expensive memory access across all of them, so each extra request is nearly free — throughput rises sharply while per-request latency grows only modestly.

✓ Knowledge check

Contrast static batching with continuous (in-flight) batching and explain why the latter keeps the GPU busier.

Show answer

Static batching collects a fixed batch, runs it to completion, then starts the next — but requests finish at different lengths, so the GPU idles on the stragglers. Continuous batching adds and removes requests every decode step: as one sequence finishes, a new one takes its slot, keeping the GPU saturated, which is how vLLM/TGI get their throughput.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Why batching wins intermediate

Static vs continuous batching intermediate

Tuning to a latency budget advanced

Exercise IC4.1 — Find your knee

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Inference & cost