Serving engines: vLLM & TGI
A serving engine ships every technique in this track behind one OpenAI-compatible API. Stand up vLLM, tune it, and your existing client code just works — now on hardware you control.
- a GPU +
pip install vllm(serving engine)
Learning objectives
- Stand up a production serving engine (vLLM) with an OpenAI-compatible API.
- Explain what the engine gives you: PagedAttention, continuous batching, quantization, spec-decoding.
- Reason about tensor/pipeline parallelism for models that don't fit one GPU.
- Point the course code at your own endpoint with a one-line change.
code/ic6-serving/ in the course, with a README. Run the scripts or copy the configs directly.The engine ties it together advanced
IC2–IC5 were techniques; a serving engine is where they ship. vLLM and TGI (Text Generation Inference) bundle PagedAttention (IC3), continuous batching (IC4), quantization (IC2), and speculative decoding (IC5) behind a single server — usually with an OpenAI-compatible API, so your existing client code just works.
This shows where all the earlier tricks actually live. A serving engine (vLLM or TGI) sits between your app and the GPU, speaking the same API a hosted service would — so your existing code doesn't change, but now you run the model yourself.
- Your app (unchanged client) — the same client code you'd use against a hosted API. Nothing special is needed.
- OpenAI-compatible API (/v1/chat) — the engine offers the same HTTP interface most hosted providers use. That compatibility is why you can switch by changing one URL.
- vLLM / TGI (batching + paging) — the engine itself. This box quietly runs everything from IC2–IC5: PagedAttention (IC3), continuous batching (IC4), quantization (IC2), speculative decoding (IC5).
- GPU(s) (quantized weights) — the actual hardware doing the math, holding the (often quantized) model weights and the KV-cache.
- Read left to right: your request enters through a familiar API, the engine applies all the optimizations, and the GPU does the work.
In short: A serving engine = one server that bundles every technique in this track behind a standard API. You configure it; you don't rebuild the optimizations yourself.
Stand up vLLM advanced
serve.sh# Serve any HF model with an OpenAI-compatible API on :8000
pip install vllm
python -m vllm.entrypoints.openai.api_server \
--model mistralai/Mistral-7B-Instruct-v0.3 \
--quantization awq \ # IC2: 4-bit
--max-num-seqs 128 \ # IC4: continuous-batch width
--gpu-memory-utilization 0.90 # IC3: how much VRAM for weights+KV-cache
This one command starts a full production server for a model, with the key optimizations turned on via flags. Each flag maps to a chapter you've already read — so you're configuring the techniques, not coding them.
pip install vllminstalls the engine. The next command launches its built-in OpenAI-compatible web server (vllm.entrypoints.openai.api_server).--model mistralai/Mistral-7B-Instruct-v0.3chooses which model to serve — any model from Hugging Face by name.--quantization awqturns on 4-bit quantization (IC2) to shrink the model and speed up decode.--max-num-seqs 128sets how many requests the continuous batcher (IC4) will pack together at once.--gpu-memory-utilization 0.90lets the engine use up to 90% of GPU memory for weights plus the KV-cache (IC3) — higher fits more concurrent requests but leaves less safety headroom.
What the output means: A server listening on port 8000 that any OpenAI-style client can call. It's applying quantization, batching, and paged KV-cache automatically.
Try this: Drop --quantization awq to serve the full-precision model instead. It needs more GPU memory and is a bit slower per token — a direct feel for what IC2 buys you.
client.pyfrom openai import OpenAI
# The ONLY change from a hosted API: the base_url. Same client, same calls.
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.3",
messages=[{"role": "user", "content": "Say hello from my own GPU."}],
)
print(resp.choices[0].message.content)
This proves the payoff of an OpenAI-compatible engine: your client code is identical to calling a hosted API — you only swap the address it points at.
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")is the whole difference:base_urlnow points at your own server, and the key can be a dummy value because your local server doesn't check it.client.chat.completions.create(...)is the exact same call you'd make to a hosted provider — same method, same arguments.print(resp.choices[0].message.content)reads the reply the same way too.
What the output means: The model's greeting, generated on your own GPU instead of a vendor's — with zero changes to how you call it.
Try this: Take any earlier lab in the course that used a hosted client and change only its base_url to this one. It should just work — that's the point of the compatible API.
When the model doesn't fit one GPU expert
Big models exceed a single GPU's memory. Tensor parallelism splits each layer across GPUs (fast interconnect needed); pipeline parallelism puts different layers on different GPUs. vLLM exposes both as flags — you size, you don't implement.
multi_gpu.shpython -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 # split each layer across 4 GPUs
Some models are too big to fit in one GPU's memory. This command spreads a 70B model across 4 GPUs with a single flag — the engine handles the splitting.
--model meta-llama/Llama-3.1-70B-Instructis a large model that won't fit on one typical GPU.--tensor-parallel-size 4turns on tensor parallelism: each layer of the model is sliced across 4 GPUs that work on it together. This needs a fast link between the GPUs (NVLink) because they exchange data constantly.- You choose the number to match how many GPUs you have; you never write the splitting logic yourself — the engine does it.
What the output means: The same OpenAI-compatible server as before, but now big enough to host a 70B model by pooling the memory of 4 GPUs.
Try this: The alternative is pipeline parallelism (different layers on different GPUs), which tolerates slower links. Tensor parallel = split each layer; pipeline = split the stack of layers.
| Parallelism | Splits | Needs |
|---|---|---|
| Tensor | each layer across GPUs | fast interconnect (NVLink) |
| Pipeline | layers across GPUs | tolerates slower links |
Exercise IC6.1 — Serve and repoint
Context: The capstone of the track is standing up your own engine and proving an earlier lab talks to it with a single base-URL change — then comparing its cost/1k-tokens to a hosted API.
Your task: Serve an open model with vLLM (quantized, continuous batching), point any earlier course lab at it by changing only base_url, confirm it works, then load-test it and compare cost/1k-tokens to a hosted API.
Requirements:
- Serve an open model with vLLM (quantized, continuous batching on)
- Repoint an earlier lab by changing only
base_url - Confirm it works end-to-end
- Load-test it (IC4) to find capacity
- Compare cost per 1k tokens against a hosted API
💡 Hint: Reuse the launch flags and the base-URL swap from the ladder; the only new work is the load test and the cost-per-1k comparison.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: vLLM exposes an OpenAI-compatible API, so migrating from a hosted API to a self-hosted engine is a single base-URL change — the request path and body are unchanged.
Your task: Show the before/after client config (concept-level, stdlib) and print the two base URLs.
Requirements:
- Hold hosted and self-hosted config as two dicts (base_url + key)
- Write a selector that returns one or the other
- Print both base URLs
- Show the same
/v1/chat/completionspath and body apply - Client code is unchanged — only the base URL differs
💡 Hint: The whole migration is swapping base_url; the endpoint path and payload stay identical, so a selector over two dicts is enough.
Show solution
The whole migration is one line — modeled with plain dicts:
hosted = {"base_url": "https://api.provider.com/v1", "key": "sk-hosted-..."}
selfhosted = {"base_url": "http://localhost:8000/v1", "key": "not-needed-locally"}
def client_config(use_self_host):
return selfhosted if use_self_host else hosted
print(client_config(False)["base_url"]) # https://api.provider.com/v1
print(client_config(True)["base_url"]) # http://localhost:8000/v1
# same /v1/chat/completions path, same request body -> code is unchangedBecause the engine speaks the same API, switching from a hosted service to your own vLLM server is a config change, not a rewrite.
Context: A serving engine isn't a new algorithm — it packages the whole IC track (PagedAttention, continuous batching, quantization, speculative decoding) behind one server.
Your task: Map each optimization to the chapter that introduced it and print a capability report so a newcomer sees the engine is IC2–IC5 behind one API.
Requirements:
- Map PagedAttention→IC3, continuous batching→IC4, quantization→IC2, speculative decoding→IC5, OpenAI API→IC6
- Store the mapping and iterate it
- Print a formatted capability report
- Show the engine is packaging, not a new algorithm
💡 Hint: It's a dict from feature to chapter — the report is just its formatted iteration, driving home that you've already seen every piece.
Show solution
Make the bundling explicit as data:
engine = {
"PagedAttention": "IC3 — KV-cache as paged virtual memory",
"continuous batching": "IC4 — swap finished slots every step",
"quantization": "IC2 — AWQ/GPTQ 4-bit weights",
"speculative decoding": "IC5 — draft+verify",
"OpenAI-compatible API":"IC6 — client code unchanged",
}
for feature, why in engine.items():
print(f"- {feature:22} {why}")A serving engine is not a new algorithm; it is the packaging that ships every technique in the track behind a single configurable server.
Context: When a model exceeds one GPU's VRAM you split it: tensor parallelism splits each layer across GPUs, pipeline parallelism splits layers across GPUs — and quantization can drop the GPU count first.
Your task: Write a helper that computes GPUs needed and recommends a parallelism strategy.
Requirements:
- Required VRAM =
params × bytes/param × overhead - Ceil-divide by per-GPU memory for the GPU count
- Recommend single GPU (1), tensor parallelism w/ NVLink (≤8), or pipeline across nodes (>8)
- Show INT4 vs FP16 dropping a 70B model from several GPUs to one
💡 Hint: Compute the memory need, divide by the card size, and let the resulting count pick the strategy — quantizing the weights first can make the count 1.
Show solution
Size the model, then choose a split:
import math
def plan_parallelism(params_b, bytes_per, gpu_gb, overhead=1.3):
need = params_b * 1e9 * bytes_per / 1024**3 * overhead # params_b in billions
n = max(1, math.ceil(need / gpu_gb)) # ceil-div GPUs
if n == 1:
strat = "single GPU"
elif n <= 8:
strat = f"tensor parallel (TP={n}) — fast intra-node NVLink"
else:
strat = f"pipeline parallel across nodes (TP*PP={n})"
return round(need,1), n, strat
print(plan_parallelism(70, 2, 80)) # 70B FP16 on 80GB cards
# (169.5, 3, 'tensor parallel (TP=3) — fast intra-node NVLink')
print(plan_parallelism(70, 0.5, 80)) # 70B INT4
# (42.4, 1, 'single GPU')Quantization (IC2) can drop a model from needing 3 GPUs to fitting on 1. When it still overflows, tensor parallelism within a node is the first split; pipeline parallelism across nodes is for the very largest models.
Context: The serving engine gives you a capacity envelope that stitches the track together — quantized weights (IC2), KV-cache per request (IC3), and batching (IC4).
Your task: Given quantized weight size, GPU VRAM, and per-request KV, estimate max concurrent sequences and rough throughput.
Requirements:
- Free memory = VRAM − weights
- Max concurrency = free memory // per-request KV-cache
- Throughput ≈ concurrency / step time (tok/s at full batch)
- Combine IC2 weights + IC3 KV + IC4 batching into one envelope
💡 Hint: Subtract the weights, divide the rest by per-request KV for concurrency, then divide by step time for throughput — the three chapters chained.
Show solution
One calculator that stitches IC2 + IC3 + IC4 together:
def capacity(gpu_gb, weight_gb, kv_per_req_gb, step_ms=25, ):
free = gpu_gb - weight_gb
concurrency = int(free // kv_per_req_gb)
throughput = concurrency / (step_ms/1000) # tok/s at full batch
return free, concurrency, throughput
free, conc, thru = capacity(gpu_gb=80, weight_gb=3.3, kv_per_req_gb=1.0)
print(f"free for KV: {free:.1f} GB") # 76.7 GB
print(f"max concurrency: {conc}") # 76
print(f"peak throughput: {thru:.0f} tok/s") # 3040 tok/sAn INT4 7B model leaves almost the whole 80 GB card for KV-cache, so concurrency (and thus throughput) is high — the payoff of stacking quantization, paging, and batching in one engine.
Context: The production launch turns on the track's optimizations through flags — you configure the engine, you don't reimplement the optimizations.
Your task: Show the production vLLM launch that enables the optimizations via flags, mapping each to its chapter, using only real flags and labelling it as needing a GPU.
Requirements:
- Launch
vllm.entrypoints.openai.api_server - Set
--quantization(IC2),--max-num-seqs(IC4) - Set
--max-model-lenand--gpu-memory-utilization(IC3) - Set
--tensor-parallel-size(IC6) - Verify with a
curlto/v1/models
💡 Hint: Each flag is one of the earlier chapters made operational — annotate the launch line and confirm it's up with a models query.
Show solution
The real serve command — needs a GPU + pip install vllm:
# serve.sh — every flag maps to a chapter
python -m vllm.entrypoints.openai.api_server \
--model mistralai/Mistral-7B-Instruct-v0.3 \
--quantization awq \ # IC2: 4-bit weights
--max-num-seqs 128 \ # IC4: continuous-batch width
--max-model-len 8192 \ # IC3: caps KV-cache per request
--gpu-memory-utilization 0.90 \ # IC3: VRAM for weights + KV
--tensor-parallel-size 1 # IC6: 1 GPU here; raise to split
# Verify: curl http://localhost:8000/v1/modelsOne command starts a full production server with quantization, paging, and continuous batching enabled. You configure the engine; you do not reimplement the optimizations.
Context: Choosing a serving topology is a constrained minimization: serve a 70B model at the QPS and p95 TTFT budget on the fewest GPUs — quantization often drops it from several cards to one.
Your task: You must serve a 70B model at 8 QPS under a p95 TTFT budget, minimizing GPU count. Given quantization and parallelism options, write the decision logic that lands on the cheapest topology that fits and meets the SLO.
Requirements:
- Size GPUs needed per option (fp16 TP, awq-int4 single, awq-int4 TP2)
- Filter to options meeting the TTFT SLO
- Pick the minimum by GPU count, breaking ties on TTFT
- Show FP16 needs several GPUs while INT4 fits on one
- Among INT4 ties, the lower-latency topology wins
💡 Hint: Filter on the SLO first, then minimize on (GPU count, TTFT) — quantizing collapses the count, and the tie-break picks the faster layout.
Show solution
Search topologies for the cheapest one that both fits and meets latency:
import math
def gpus_needed(params_b, bytes_per, gpu_gb=80, overhead=1.3):
need = params_b * 1e9 * bytes_per / 1024**3 * overhead # params_b in billions
return max(1, math.ceil(need / gpu_gb))
options = [
# name, bytes_per, est_ttft_ms
("fp16 TP", 2, 300),
("awq-int4 single", 0.5, 460),
("awq-int4 TP2", 0.5, 320), # int4 fits 1 GPU; TP2 is optional, not needed
]
SLO_TTFT = 500
plans = []
for name, bpp, ttft in options:
g = gpus_needed(70, bpp)
if ttft <= SLO_TTFT:
plans.append((g, ttft, name))
best = min(plans) # fewest GPUs, then fastest TTFT
print("feasible:", [(name, g) for g, _, name in plans])
# feasible: [('fp16 TP', 3), ('awq-int4 single', 1), ('awq-int4 TP2', 1)]
print("PICK:", best[2], f"({best[0]} GPU(s), {best[1]} ms TTFT)")
# PICK: awq-int4 TP2 (1 GPU, 320 ms TTFT) — cheapest GPU count, best TTFT among tiesFP16 needs 3 GPUs; INT4 quantization drops the 70B model onto a single card. Both INT4 options tie at one GPU, so the tie-break is latency — TP2 gives the better TTFT for the same GPU count. Quantization is what makes the single-GPU topology possible at all.
✓ Checkpoint — you can move on when you can…
- Stand up vLLM with an OpenAI-compatible API.
- Explain which IC techniques the engine bundles.
- Choose tensor vs pipeline parallelism for a large model.
- Repoint existing client code to your endpoint with one line.
Knowledge check check yourself
A serving engine like vLLM exposes an OpenAI-compatible API. What does that compatibility buy you, and which IC techniques does the engine bundle behind it?
Show answer
Contrast tensor parallelism and pipeline parallelism for serving a model too big for one GPU, including their interconnect needs.