Continuous batching & scheduling

Why batching wins intermediate

Here's a number that reorganizes how you think about serving. Take a 7B model on a good GPU. Send it one request and it might spit out 40 tokens a second. Send it thirty-two requests at the same time and it doesn't slow to a crawl — it happily produces well over a thousand tokens a second, and each of those thirty-two users still sees roughly the same speed they'd get alone. You just served 32× the work for almost the same money. That is not a trick or an optimization at the margins; it is the reason GPU inference is affordable at all, and it's why the same model costs one lab pennies per million tokens and costs a naive self-hoster a fortune.

Why is that even possible? Here's the analogy. Imagine a librarian who has to haul one gigantic, heavy encyclopedia from the basement up to the desk every single time someone asks a question — then carry it back down. If one person is waiting, the librarian spends almost the whole time on the stairs and only a heartbeat actually answering. That trip is fixed cost: it's the same whether one person is at the desk or fifty. So the smart move is obvious — let a crowd gather, haul the encyclopedia up once, and answer everyone's question from that single trip. The model's weights are that encyclopedia; reading them out of GPU memory is the trip up the stairs (this is the memory-bound reality you met in IC1). One weight-read can answer the "what's the next token?" question for the whole crowd in nearly the same time it takes to answer for one — so throughput climbs almost linearly with batch size while each user's latency barely moves.

The mistake this kills: newcomers instinctively measure a served model by firing one request and timing it, then multiply that latency by their expected traffic to "size" the hardware. That math is wildly pessimistic — it assumes every request pays for its own stair-climb, when in reality they share. The right mental model is the opposite: a GPU running one request is mostly idle, and your job in this whole chapter is to fill that idle time with other people's requests. The concept below, continuous batching, is just the cleverest way to keep that crowd at the desk as densely packed as possible.

Decode re-reads all the model's weights for every token (IC1). If you're doing that for one request, the GPU is mostly idle waiting on memory. Batch many requests through the same weight-read and you serve them almost for free — throughput rises far more than latency. Batching is the single biggest efficiency lever in serving.

Static vs continuous batching intermediate

Batching is a huge win — but the naive way to batch quietly throws half of it away. Here's the problem in one picture. You gather four requests, run them together, and start streaming tokens. But requests don't all finish at the same time: one user asked for a one-line answer and stops after 3 tokens; another asked for an essay and keeps going for 200. If your batch is fixed — the four ride together from start to finish — then the three short requests sit in their seats, done, doing nothing, while the whole batch waits for the essay-writer to finish. Their GPU slots are occupied but idle. That's static batching, and on real traffic (where output lengths vary wildly) it can waste half your GPU or more.

The analogy: a bus versus a taxi rank. Static batching is a bus that leaves full, drives the whole route, and won't pick anyone up until every original passenger has been dropped off and it returns to the depot — even if three seats emptied at the first stop. Continuous (in-flight) batching is a taxi rank at the airport: the instant one cab pulls away with its fare, the next person in line steps into that spot. No slot ever sits empty while someone is waiting. Concretely, continuous batching re-decides the batch every single decode step: at each step it checks which sequences finished, evicts them, and pulls waiting requests in to fill the freed slots. The batch is a living, churning set, not a frozen group.

The common misconception is that continuous batching is a fundamentally different or heavier algorithm — some exotic parallelism. It isn't. It's the same shared weight-read you already understand, just with the membership of the batch refreshed on every token instead of once per group. That one change — add and evict per step rather than per batch — is the entire difference between "good" throughput and the throughput that makes vLLM and TGI worth running. Note the honest scope, though: you almost never implement this yourself. The scheduler lives inside the serving engine (IC6). What you own is understanding it well enough to reason about why your utilization graph looks the way it does and which knob to turn — which is exactly what the table and the tuning section below give you.

Static batching waits to collect a fixed batch, runs it to completion, then starts the next — but requests finish at different lengths, so the GPU idles on the stragglers. Continuous (in-flight) batching adds and removes requests every step: as one finishes, a new one takes its slot. This keeps the GPU saturated and is why modern servers (vLLM/TGI) get their throughput.

Tuning to a latency budget advanced

If batching is so good, why not just crank the batch size to the moon? Because throughput and latency pull in opposite directions, and someone is always paying the latency bill. Every request you add to a batch makes each decode step a hair slower for everyone in it — and each user feels that as their tokens arriving a little later. Push the batch too high and you'll post gorgeous throughput numbers on a dashboard while your actual users watch their answers stutter out one slow token at a time. Tuning batching isn't "find the maximum"; it's "find the biggest batch that still keeps your slowest users fast enough." That constraint — the speed you promise users — is the thing everything else serves.

The analogy is a rideshare pool. Packing four strangers into one car is efficient for the driver (throughput), but every extra pickup and drop-off makes your ride longer (latency). There's a sweet spot: pool enough to be cheap, but not so much that riders bail. For a served model that promise is written as a latency SLO — a service-level objective like "95% of tokens arrive within 50 ms of the last one" (that per-token gap is TPOT, time-per-output-token, from IC1). You tune the batch up against that ceiling and stop the moment you'd breach it. Two numbers to hold in your head as you read the steps: the batch is how densely you pack one GPU, and replicas are how many GPUs you run — they solve different problems, and confusing them is the classic tuning mistake.

The mistake to avoid: reacting to a latency spike by adding more hardware, or reacting to a capacity shortfall by shrinking the batch. They're crossed wires. If p95 latency is breaching, the batch is too big — shrink it, even though that lowers per-GPU throughput. If you simply can't keep up with the request rate, no batch setting saves you — add replicas. One knob protects the promised speed; the other protects raw capacity. Keep them straight and the four-step recipe below reads like common sense; mix them up and you'll thrash. Now, concretely:

How to tune batching for your workload

1. Set your latency SLO (e.g. p95 TPOT < 50 ms) — the constraint everything serves.
2. Raise max concurrent sequences until throughput plateaus or you breach the SLO.
3. Watch KV-cache memory (IC3): too many sequences and you'll OOM or start evicting.
4. For bursty traffic, cap concurrency so a spike can't blow the latency budget for everyone.

loadtest.pyimport asyncio, time
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")

async def one():
    t0 = time.time()
    await client.chat.completions.create(
        model="local-model", max_tokens=128,
        messages=[{"role": "user", "content": "Summarize continuous batching."}])
    return time.time() - t0

async def bench(concurrency):
    t0 = time.time()
    lat = await asyncio.gather(*[one() for _ in range(concurrency)])
    wall = time.time() - t0
    print(f"conc={concurrency:3d}  p95={sorted(lat)[int(len(lat)*0.95)-1]:.2f}s  "
          f"throughput={concurrency/wall:.1f} req/s")

for c in (1, 4, 16, 64):
    asyncio.run(bench(c))   # watch throughput climb and p95 latency rise

def throughput(batch, step_ms=25):
    return batch / (step_ms / 1000)   # tokens per second (1 tok/req/step)

for b in (1, 8, 32):
    print(f"batch {b:>2}: {throughput(b):.0f} tok/s")
# batch  1:   40 tok/s
# batch  8:  320 tok/s
# batch 32: 1280 tok/s

def step_ms(batch, base=25, per_extra=0.4):
    return base + batch * per_extra    # step grows slowly with batch

for b in (1, 8, 32, 128):
    s = step_ms(b)
    thru = b / (s/1000)
    print(f"batch {b:>3}: step {s:5.1f} ms | latency {s:5.1f} ms/tok | {thru:6.0f} tok/s")
# batch   1: step  25.4 |  25.4 ms/tok |    39 tok/s
# batch 128: step  76.2 |  76.2 ms/tok |  1680 tok/s

lengths = [3, 3, 3, 10]   # output tokens per request; one straggler

# Static: batch runs until the LONGEST finishes; short reqs idle their slot after done
steps_static = max(lengths)
work = sum(lengths)
slots = len(lengths) * steps_static
print("static:   util =", f"{work/slots*100:.0f}%",  # 47%
      f"({work} useful / {slots} slot-steps)")

# Continuous: a finished slot is refilled; here no queue, so idle slots simply free up.
# Utilization = useful work / (slot-steps actually occupied)
occupied = sum(lengths)   # each slot occupied exactly as long as it runs
print("continuous: util =", f"{work/occupied*100:.0f}%")  # 100% of occupied slot-steps

def step_ms(batch, base=25, per_extra=0.4):
    return base + batch * per_extra

def max_batch_under(budget_ms):
    b = 1
    while step_ms(b + 1) <= budget_ms:
        b += 1
    return b, round(step_ms(b), 1)

for budget in (40, 60, 100):
    b, s = max_batch_under(budget)
    print(f"budget {budget} ms/tok -> max batch {b} (step {s} ms)")
# budget 40 -> max batch 37
# budget 100 -> max batch 187

# serve.sh — continuous batching is on by default in vLLM
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --max-num-seqs 128 \          # IC4: max concurrent sequences in a batch
  --max-num-batched-tokens 8192 \  # IC4: token budget per scheduler step
  --gpu-memory-utilization 0.90    # IC3: VRAM for weights + KV-cache

# Then point any OpenAI client at http://localhost:8000/v1

def step_ms(batch, base=25, per_extra=0.4):
    return base + batch * per_extra

peak_qps, out_tok = 40, 200
current_batch = 128
p95 = step_ms(current_batch)
print("current p95 step:", p95, "ms")   # 76.2 ms -> breaches 60 ms

# Option A: cap batch to meet latency
safe_batch = 1
while step_ms(safe_batch+1) <= 60: safe_batch += 1
cap_thru = safe_batch / (step_ms(safe_batch)/1000)
need_thru = peak_qps * out_tok
print(f"cap batch to {safe_batch}: {cap_thru:.0f} tok/s vs need {need_thru} tok/s")
replicas = -(-need_thru // int(cap_thru))
print(f"=> cap batch AND run {replicas} replicas to hold both SLO and QPS")

Knowledge check check yourself