Serving engines: vLLM & TGI

The engine ties it together advanced

Picture the last five chapters as a workshop full of specialist power tools. Quantization shrinks the weights, the paged KV-cache stops memory fragmenting, continuous batching keeps the GPU busy, speculative decoding drafts tokens ahead. Each one is real and each one is finicky — and if you tried to wire them together yourself, by hand, in your own inference loop, you'd spend months on plumbing and still ship something slower and buggier than what a team of specialists already maintains. That is the mistake this section exists to prevent: reinventing the serving stack instead of adopting one.

A serving engine is the pre-assembled workbench. Think of it the way you think about a database: you don't write your own B-tree, page cache and query planner every time you need to store rows — you install PostgreSQL and spend your energy on the schema. A serving engine is that same bargain for inference. vLLM and TGI (Text Generation Inference) are the two you'll meet most; TensorRT-LLM (NVIDIA's, squeezing the last drop of throughput out of their GPUs with compiled kernels) and SGLang (strong on structured output and prefix reuse) round out the field. What they share is the important part: they bundle PagedAttention (IC3), continuous batching (IC4), quantization (IC2), and speculative decoding (IC5) behind a single server — no single engine invented those techniques, they package and productionize them.

Here's the detail that makes the whole thing click, and it's easy to undersell: most of these engines expose an OpenAI-compatible API. That means the HTTP shape of a request — the /v1/chat/completions path, the messages array, the streaming format — is the same one the hosted providers use. So the client code you already wrote against a hosted API keeps working; you change the address it points at and nothing else. The engine is doing wildly sophisticated work behind that door (paging attention memory, reshuffling batches every single decode step), but the door itself is boring and familiar on purpose. Your job shifts from implementing optimizations to configuring them — picking flags, sizing memory, choosing how many GPUs — which is exactly the leverage you want. The diagram below traces one request through that stack: familiar API in, all the tricks in the middle, GPU at the end.

IC2–IC5 were techniques; a serving engine is where they ship. vLLM and TGI (Text Generation Inference) bundle PagedAttention (IC3), continuous batching (IC4), quantization (IC2), and speculative decoding (IC5) behind a single server — usually with an OpenAI-compatible API, so your existing client code just works.

Stand up vLLM advanced

Brace yourself for an anticlimax — the good kind. After a whole track of KV-cache paging and batch schedulers, you might expect standing up a production LLM server to be a weekend of YAML and CUDA errors. It's one command. That gap between "how hard it sounds" and "how hard it is" is the entire value proposition of a serving engine, and it's worth pausing on: the difficulty didn't vanish, it got absorbed — moved from your keyboard into code the vLLM team maintains and battle-tests across thousands of deployments.

The mental model to hold is a car dashboard, not an engine rebuild. You're not machining pistons; you're turning knobs. Each flag on the launch line is one dial, and — this is the satisfying part — every dial maps back to a chapter you've already read. --quantization is IC2. --max-num-seqs is the continuous-batch width from IC4. --gpu-memory-utilization is how much VRAM you're handing to weights plus the paged KV-cache from IC3. You already understand what each knob does physically; here you just learn its name on this particular dashboard. So read the lab below less like new material and more like a legend that labels controls you can already reason about.

The common mistake here is a memory one, and it bites on day one. The instinct is to crank --gpu-memory-utilization toward 1.0 to "use the whole card" — and then the server either refuses to start or crashes under a burst of concurrent requests. Remember why from IC3: that budget covers the model weights and the KV-cache that grows with every active sequence and every token they generate. Leave it maxed and there's no headroom for the cache when traffic spikes. The engine defaults are conservative for exactly this reason; nudge them up while watching real load, don't slam them to the ceiling and hope. The same discipline applies to --max-num-seqs: a bigger batch is more throughput right up until the KV-cache for all those sequences won't fit, at which point requests queue or get preempted. Configuring a serving engine well is mostly this one skill — spending your fixed VRAM budget wisely across weights, cache, and concurrency — which is precisely why the earlier chapters came first.

serve.sh# Serve any HF model with an OpenAI-compatible API on :8000
pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --quantization awq \              # IC2: 4-bit
  --max-num-seqs 128 \             # IC4: continuous-batch width
  --gpu-memory-utilization 0.90     # IC3: how much VRAM for weights+KV-cache

client.pyfrom openai import OpenAI

# The ONLY change from a hosted API: the base_url. Same client, same calls.
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[{"role": "user", "content": "Say hello from my own GPU."}],
)
print(resp.choices[0].message.content)

When the model doesn't fit one GPU expert

Sooner or later you hit a wall that no flag can tune away: the model is simply bigger than the card. A 70B model in FP16 wants roughly 140 GB of weights before you've stored a single token of KV-cache — and the biggest GPUs today hold 80 GB. You can't page your way out of this one; the weights themselves don't fit. Quantization (IC2) buys you a lot of runway here — INT4 can drop that 70B model onto a single card — but eventually a model arrives that overflows even the best-quantized single GPU, and then you have to split the model itself across several. That splitting is what this section is about, and there are two fundamentally different ways to do it.

The analogy that makes them stick is a restaurant kitchen. Tensor parallelism is the whole line of cooks working on the same dish at the same time — one chops, one sears, one plates — all on one order simultaneously. Each layer of the model is sliced across the GPUs, and they compute that single layer together, which means they're constantly passing partial results back and forth. That chatter is relentless, so tensor parallelism demands a fast interconnect (NVLink) between the GPUs; put them on a slow link and the talking swamps the working. Pipeline parallelism is the assembly line instead: dish one starts at station A, moves to station B, then C — each GPU owns a different stretch of the model's layers, and activations flow down the chain. Far less cross-talk, so it tolerates slower links (even across separate machines), but it introduces "bubbles" where downstream GPUs idle waiting for upstream work.

Here's the part people get backwards, so say it plainly: you do not implement any of this. You will never write the code that slices a weight matrix across four devices or schedules activations down a pipeline — that is exactly the sort of thing the serving engine exists to hide. In vLLM, tensor parallelism is a single number: --tensor-parallel-size 4. Your entire job is sizing — figuring out how many GPUs the model needs (weights ÷ card capacity, rounded up, with overhead) and picking the split that matches your hardware's interconnect. The rule of thumb worth memorizing: tensor-parallel within one machine where NVLink is fast, pipeline-parallel across machines where the link is slower. The common trap is reaching for multi-GPU too early — always try quantization first, because turning a 3-GPU model into a 1-GPU model is cheaper and simpler than any parallelism scheme. Only split when a single quantized card genuinely can't hold the weights.

Big models exceed a single GPU's memory. Tensor parallelism splits each layer across GPUs (fast interconnect needed); pipeline parallelism puts different layers on different GPUs. vLLM exposes both as flags — you size, you don't implement.

multi_gpu.shpython -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4      # split each layer across 4 GPUs

hosted = {"base_url": "https://api.provider.com/v1", "key": "sk-hosted-..."}
selfhosted = {"base_url": "http://localhost:8000/v1", "key": "not-needed-locally"}

def client_config(use_self_host):
    return selfhosted if use_self_host else hosted

print(client_config(False)["base_url"])  # https://api.provider.com/v1
print(client_config(True)["base_url"])   # http://localhost:8000/v1
# same /v1/chat/completions path, same request body -> code is unchanged

engine = {
    "PagedAttention":       "IC3 — KV-cache as paged virtual memory",
    "continuous batching":  "IC4 — swap finished slots every step",
    "quantization":         "IC2 — AWQ/GPTQ 4-bit weights",
    "speculative decoding": "IC5 — draft+verify",
    "OpenAI-compatible API":"IC6 — client code unchanged",
}
for feature, why in engine.items():
    print(f"- {feature:22} {why}")

import math
def plan_parallelism(params_b, bytes_per, gpu_gb, overhead=1.3):
    need = params_b * 1e9 * bytes_per / 1024**3 * overhead   # params_b in billions
    n = max(1, math.ceil(need / gpu_gb))   # ceil-div GPUs
    if n == 1:
        strat = "single GPU"
    elif n <= 8:
        strat = f"tensor parallel (TP={n}) — fast intra-node NVLink"
    else:
        strat = f"pipeline parallel across nodes (TP*PP={n})"
    return round(need,1), n, strat

print(plan_parallelism(70, 2, 80))    # 70B FP16 on 80GB cards
# (169.5, 3, 'tensor parallel (TP=3) — fast intra-node NVLink')
print(plan_parallelism(70, 0.5, 80))  # 70B INT4
# (42.4, 1, 'single GPU')

def capacity(gpu_gb, weight_gb, kv_per_req_gb, step_ms=25, ):
    free = gpu_gb - weight_gb
    concurrency = int(free // kv_per_req_gb)
    throughput = concurrency / (step_ms/1000)   # tok/s at full batch
    return free, concurrency, throughput

free, conc, thru = capacity(gpu_gb=80, weight_gb=3.3, kv_per_req_gb=1.0)
print(f"free for KV: {free:.1f} GB")     # 76.7 GB
print(f"max concurrency: {conc}")        # 76
print(f"peak throughput: {thru:.0f} tok/s")  # 3040 tok/s

# serve.sh — every flag maps to a chapter
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --quantization awq \                 # IC2: 4-bit weights
  --max-num-seqs 128 \                 # IC4: continuous-batch width
  --max-model-len 8192 \               # IC3: caps KV-cache per request
  --gpu-memory-utilization 0.90 \      # IC3: VRAM for weights + KV
  --tensor-parallel-size 1              # IC6: 1 GPU here; raise to split

# Verify:  curl http://localhost:8000/v1/models

import math
def gpus_needed(params_b, bytes_per, gpu_gb=80, overhead=1.3):
    need = params_b * 1e9 * bytes_per / 1024**3 * overhead   # params_b in billions
    return max(1, math.ceil(need / gpu_gb))

options = [
    # name,              bytes_per, est_ttft_ms
    ("fp16 TP",          2, 300),
    ("awq-int4 single",  0.5, 460),
    ("awq-int4 TP2",     0.5, 320),   # int4 fits 1 GPU; TP2 is optional, not needed
]
SLO_TTFT = 500
plans = []
for name, bpp, ttft in options:
    g = gpus_needed(70, bpp)
    if ttft <= SLO_TTFT:
        plans.append((g, ttft, name))
best = min(plans)                       # fewest GPUs, then fastest TTFT
print("feasible:", [(name, g) for g, _, name in plans])
# feasible: [('fp16 TP', 3), ('awq-int4 single', 1), ('awq-int4 TP2', 1)]
print("PICK:", best[2], f"({best[0]} GPU(s), {best[1]} ms TTFT)")
# PICK: awq-int4 TP2 (1 GPU, 320 ms TTFT) — cheapest GPU count, best TTFT among ties

Knowledge check check yourself