Local & Open Models · Chapter LM3

Production serving with vLLM

Q: What two vLLM flags are the main memory dials when you hit an out-of-memory error, and what does each control?

--gpu-memory-utilization (e.g. lower to 0.80) sets the fraction of GPU memory vLLM may use for weights + KV-cache, and --max-num-seqs sets the continuous-batch width (how many requests are batched at once). Lowering either reduces memory pressure.

Ollama got you running; vLLM gets you serving. The IC6 production engine — batching, paging, quantization — applied to self-hosting open models behind an OpenAI-compatible API.

⏱️ ~1.5 hours🧪 2 labs🎯 Advanced

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

a GPU + pip install vllm (serving engine)

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Explain why production self-hosting needs vLLM, not Ollama.
Stand up vLLM with an OpenAI-compatible API for an open model.
Size the GPU and set the key serving flags.
Connect your app with a one-line change.

▶ Runnable companionThe code in this lesson is also saved under code/lm3-vllm-serving/ in the course, with a README. Run the scripts or copy the configs directly.

From laptop to server intermediate

Ollama got you running; vLLM gets you serving. It's the production engine from IC6 — PagedAttention, continuous batching, quantization — behind the same OpenAI-compatible API. This chapter is the self-hosting view of the inference track.

🗺️ How to read this diagram

This pipeline shows how a raw open model becomes a real service that many people can use at once. Read it left to right — each box feeds the next.

The first box (Open model weights) is just the downloaded model files, usually from Hugging Face ("HF") — data, not yet a running service.
The second box (vLLM server) is the engine that loads those weights and runs them efficiently. "batching+paging" means it serves many requests together and manages GPU memory cleverly, so it's fast under load.
The third box (OpenAI-compatible API) is the doorway vLLM opens — a /v1/chat endpoint that looks exactly like OpenAI's, so existing code works against it.
The last box (Your app + many users) is the point: real traffic hits your app, which calls the API — and the client code didn't have to change.

In short: Weights → vLLM engine → OpenAI-style API → your app. vLLM is the middle piece that turns a model file into a fast service many users can share.

Stand it up intermediate

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab LM3.1 · Serve for production

serve.shpip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \                 # 4-bit (IC2)
  --max-num-seqs 128 \                # continuous-batch width (IC4)
  --gpu-memory-utilization 0.90 \     # weights + KV-cache budget (IC3)
  --host 0.0.0.0 --port 8000

▶ How this works

This one command turns an open model into a production web service. It's a python -m ... launcher with several flags — the \ at the end of each line just means "the command continues on the next line". Read the flags top to bottom; each is one knob.

pip install vllm installs the serving engine (needs a GPU).
--model meta-llama/Llama-3.1-8B-Instruct names which open model to serve — this is its Hugging Face path, which vLLM downloads.
--quantization awq loads a 4-bit compressed version so the model fits in less GPU memory (the idea from the quantization lesson, IC2).
--max-num-seqs 128 sets how many requests it will batch together at once — higher means more throughput for many users.
--gpu-memory-utilization 0.90 lets vLLM use 90% of the GPU's memory, split between the model weights and the working cache.
--host 0.0.0.0 --port 8000 exposes the API on port 8000 so other machines can reach it.

What the output means: A long-running server: it prints startup logs, loads the model, then waits for requests at http://your-server:8000/v1.

Try this: If you hit an out-of-memory error, lower --gpu-memory-utilization (say to 0.80) or --max-num-seqs — those two flags are your main memory dials.

Lab LM3.2 · Same client, new base_url

client.pyfrom openai import OpenAI
client = OpenAI(base_url="http://your-server:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello from my own server."}])
print(r.choices[0].message.content)

▶ How this works

This is the same lesson as Ollama, one level up: connecting to your production vLLM server is also just a base_url change. The code you already wrote keeps working against a model you now host yourself.

OpenAI(base_url="http://your-server:8000/v1", api_key="none") points the standard client at your vLLM server (port 8000) instead of a cloud provider. No real key is needed, so "none" is fine.
client.chat.completions.create(model="meta-llama/Llama-3.1-8B-Instruct", messages=[...]) sends a chat request. The model name must match what the server was started with in the previous lab.
r.choices[0].message.content pulls the reply text out and prints it — identical to how you'd read any OpenAI-style response.

What the output means: The model's greeting prints — served by your own GPU box, not a paid API.

Try this: Replace your-server with the real host/IP of your vLLM machine. If it's on the same computer, use localhost.

Sizing the GPU advanced

Memory must hold weights + KV-cache + overhead. Use the IC1–IC3 math: a 4-bit 8B model is ~5–6 GB of weights, leaving the rest of the card for KV-cache (which sets how many concurrent requests you can serve). gpu-memory-utilization tunes that split.

This is literally IC6 againIf LM3 feels like a repeat of IC6, that's the point: inference optimization and self-hosting are one skill. The Inference track taught the levers; here you apply them to run open models you control.

Exercise LM3.1 — Serve and load-test

Context: The point of vLLM is throughput under load — and a load test against Ollama on the same model makes the gap unmistakable.

Your task: Serve an 8B model with vLLM (quantized, batched), point a course lab at it, then run a load test at rising concurrency and record throughput and p95 latency; compare to Ollama.

Requirements:

Serve the model with vLLM (quantized + batched)
Drive rising concurrency with a load test
Record throughput and p95 latency at each level
Run the same test against Ollama on the same model
Confirm vLLM wins badly under load

💡 Hint: The divergence shows up as concurrency climbs — single requests look similar, but batching pulls ahead under real load.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Why vLLM, not Ollama, for productionBeginner

Context: Ollama runs; vLLM serves. The production engine adds continuous batching, PagedAttention, and quantization — exactly what a single-user tool doesn't optimize for.

Your task: Write a helper that names what vLLM adds for production versus Ollama's single-user focus.

Requirements:

Name continuous batching, PagedAttention, and quantization
Give the production benefit of each
Note the OpenAI-compatible API so existing clients work unchanged
Print the feature→benefit list

💡 Hint: These are the IC6 throughput concepts; the framing is why each matters under concurrent load, not the implementation.

Show solution

vLLM is the production engine (IC6 concepts). Runnable:

ADDS = {
    "continuous batching": "serve many requests together, high GPU utilization",
    "PagedAttention":       "manage KV-cache memory in pages, fit more concurrency",
    "quantization":         "run bigger models in less VRAM",
    "OpenAI-compatible API":"existing client code works unchanged",
}
def why_vllm():
    for feature, benefit in ADDS.items():
        print(f"- {feature}: {benefit}")

why_vllm()

These throughput features are exactly what a single-user tool like Ollama does not optimize for.

Exercise 2 · Stand up vLLM (serving config)Intermediate

Context: Standing up vLLM behind an OpenAI-compatible API comes down to a launch command with a few key flags — model, quantization, batch width, context length.

Your task: Write the vLLM launch command with those key flags, labeled as needing a GPU.

Requirements:

Show python -m vllm.entrypoints.openai.api_server (or equivalent)
Include the model, a quantization flag, a batch-width flag (--max-num-seqs), and a context-length flag
Include a GPU-memory-utilization flag
Comment what each flag controls
Label it as needing a GPU + pip install vllm

💡 Hint: The flags map to concepts: quantization shrinks weights, --max-num-seqs is the continuous-batch width, --max-model-len caps context — it exposes a /v1/chat/completions endpoint.

Show solution

This is serving config (needs a GPU + pip install vllm):

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \        # 4-bit weights (IC2) -> less VRAM
  --max-num-seqs 128 \        # continuous-batch width (concurrency)
  --max-model-len 8192 \      # context length ceiling
  --gpu-memory-utilization 0.90  # fraction of VRAM vLLM may use

It opens a /v1/chat/completions endpoint that looks exactly like OpenAI's — the whole point of the engine.

Exercise 3 · Size the GPU for a model + concurrencyAdvanced

Context: VRAM must hold weights plus KV cache for every concurrent sequence — and it's usually the KV cache, not the weights, that forces a concurrency cap.

Your task: Write a calculator: weights (bytes/param by quant) + per-sequence KV cache × max_num_seqs, and say whether it fits a given GPU.

Requirements:

Weight bytes/param vary by quant (fp16 / awq / int8)
Per-sequence KV cache scales with layers, hidden size, and context length (K and V)
Total = weights + per-seq KV × concurrency
Return the total and whether it fits the GPU
Show a realistic 8B AWQ config on a mid-size card

💡 Hint: KV per sequence is ~2 (K and V) × layers × hidden × ctx × bytes; concurrency multiplies it, which is why it dominates.

Show solution

Total VRAM = weights + KV cache * concurrency. Runnable:

BYTES = {"fp16": 2, "awq": 0.5, "int8": 1}   # ~bytes per weight param

def weights_gb(params_b, quant):
    return params_b * 1e9 * BYTES[quant] / (1024**3)

def kv_gb_per_seq(layers, hidden, ctx_len, bytes_per=2):
    # 2 (K and V) * layers * hidden * ctx * bytes, in GB
    return 2 * layers * hidden * ctx_len * bytes_per / (1024**3)

def fits(gpu_gb, params_b, quant, layers, hidden, ctx, seqs):
    total = weights_gb(params_b, quant) + kv_gb_per_seq(layers, hidden, ctx) * seqs
    return total, total <= gpu_gb

# 8B AWQ, 32 layers, hidden 4096, ctx 8192, 32 concurrent seqs, on a 24GB card
total, ok = fits(24, 8, "awq", 32, 4096, 8192, 32)
print(f"needs ~{total:.1f} GB -> {'FITS' if ok else 'TOO BIG'} on 24GB")

KV cache scales with concurrency and context — it, not the weights, is what usually forces you to cap --max-num-seqs.

Exercise 4 · Tune max-num-seqs to fit VRAMExpert

Context: You should set --max-num-seqs from the VRAM math, not by guessing — too high and vLLM OOMs under load, too low and the GPU sits idle.

Your task: Write a solver that returns the largest batch width (max_num_seqs) that fits a fixed GPU and model.

Requirements:

Compute the VRAM budget left after weights (with a reserve)
Divide the remaining budget by the per-sequence KV cost
Return the integer max concurrency (0 if the weights alone don't fit)
Show the recommended --max-num-seqs for a card

💡 Hint: It's the fit calculation solved for concurrency: (usable VRAM − weights) / KV-per-seq, floored.

Show solution

Solve for the concurrency ceiling. Runnable:

BYTES = {"fp16": 2, "awq": 0.5}
def weights_gb(pb, q): return pb * 1e9 * BYTES[q] / (1024**3)
def kv_per_seq(layers, hidden, ctx): return 2*layers*hidden*ctx*2 / (1024**3)

def max_seqs(gpu_gb, params_b, quant, layers, hidden, ctx, reserve=0.10):
    budget = gpu_gb * (1 - reserve) - weights_gb(params_b, quant)
    if budget <= 0:
        return 0                       # weights alone don't fit
    return int(budget // kv_per_seq(layers, hidden, ctx))

n = max_seqs(24, 8, "awq", 32, 4096, 8192)
print(f"set --max-num-seqs {n} to stay within 24GB")

Set --max-num-seqs from the math, not by guessing — too high and vLLM OOMs under load; too low and you waste the GPU.

Exercise 5 · Connect your app with a one-line changeProfessional

Context: Because vLLM speaks the OpenAI API, connecting an app is a base_url swap — one line.

Your task: Show the one-line change (needs the server) and validate the request payload offline.

Requirements:

Build/validate the chat payload with the standard library, no server
Show the real call pointing an OpenAI client at the vLLM host's /v1 endpoint
Only the base_url changes in app code
Label the real call as needing the vLLM server running

💡 Hint: Same OpenAI client, new base_url; the self-host swap is one line because vLLM mimics the API.

Show solution

Validate the payload offline (runnable):

def chat_payload(model, prompt):
    return {"model": model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 256}
print(chat_payload("meta-llama/Llama-3.1-8B-Instruct", "hi"))

The only app change (needs the vLLM server running):

from openai import OpenAI
client = OpenAI(base_url="http://your-vllm-host:8000/v1", api_key="not-needed")
r = client.chat.completions.create(**chat_payload("meta-llama/Llama-3.1-8B-Instruct", "hi"))
print(r.choices[0].message.content)

Same OpenAI client, new base_url — the self-host swap is one line because vLLM mimics the API.

Exercise 6 · Capacity plan: GPUs for a target QPSIndustry scenario

Context: Capacity planning turns vLLM's per-GPU throughput into a replica count and a monthly bill — the input to the LM5 build-vs-buy decision.

Your task: Given a target requests/second, per-GPU throughput, and headroom, compute how many GPU replicas you need and the monthly GPU cost.

Requirements:

Never plan a GPU at 100% — apply a headroom factor to per-GPU QPS
Replicas = ceil(target QPS / effective per-GPU QPS)
Monthly cost = replicas × hourly rate × hours/month
Show a worked example with the replica count and dollar figure

💡 Hint: math.ceil the ratio after de-rating per-GPU QPS by headroom, then multiply replicas by the hourly rate over a month.

Show solution

Capacity = ceil(target QPS / per-GPU QPS), then cost it. Runnable:

import math

def plan(target_qps, per_gpu_qps, gpu_hourly_usd, headroom=0.30):
    effective = per_gpu_qps * (1 - headroom)      # never run a GPU flat out
    replicas = math.ceil(target_qps / effective)
    monthly = replicas * gpu_hourly_usd * 24 * 30
    return replicas, round(monthly, 2)

replicas, cost = plan(target_qps=40, per_gpu_qps=12, gpu_hourly_usd=1.20)
print(f"{replicas} GPU replicas, ~${cost}/mo")
# 40 / (12*0.7)=~4.76 -> 5 replicas; 5*1.20*720 = $4320/mo

vLLM's batching sets per-GPU QPS; capacity planning turns that into replica count and a bill — the input to the LM5 build-vs-buy call.

✓ Checkpoint — you can move on when you can…

Explain why prod self-hosting uses vLLM over Ollama.
Stand up vLLM with an OpenAI-compatible API.
Size the GPU with the IC1–IC3 memory math.
Connect an app with a one-line change.

Knowledge check check yourself

✓ Knowledge check

Why does production self-hosting call for vLLM rather than Ollama, and which IC-track techniques does vLLM apply?

Show answer

Ollama is built for local, low-traffic dev; vLLM is the production engine for high-throughput serving of many concurrent users. It applies the inference-track levers from IC6 — PagedAttention (KV-cache paging), continuous batching, and quantization — behind the same OpenAI-compatible API.

✓ Knowledge check

What two vLLM flags are the main memory dials when you hit an out-of-memory error, and what does each control?

Show answer

--gpu-memory-utilization (e.g. lower to 0.80) sets the fraction of GPU memory vLLM may use for weights + KV-cache, and --max-num-seqs sets the continuous-batch width (how many requests are batched at once). Lowering either reduces memory pressure.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

From laptop to server intermediate

Stand it up intermediate

Sizing the GPU advanced

Exercise LM3.1 — Serve and load-test

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Local & open models