Production serving with vLLM

From laptop to server intermediate

Here's the trap that catches almost everyone the first time. You wire up Ollama on your laptop in the last chapter, the model answers, everything feels production-ready — so you copy that same setup onto a GPU box, point your app at it, and open it to a handful of colleagues. It's great for the first person. It's fine for the second. By the tenth simultaneous user it's crawling, requests are timing out, and the GPU is somehow only 20% busy while everyone waits. Nothing is broken. You just used a single-user tool for a many-user job, and those are two genuinely different problems.

The reason is worth internalizing, because it's the whole justification for this chapter. Think of Ollama as your own home kitchen: perfect for cooking one meal for yourself, on demand, right now. A production serving engine is a restaurant line — the same stoves, but organized so one crew can plate fifty orders at once without any single diner waiting fifty times as long. The bottleneck in LLM serving is almost never raw compute; it's how cleverly you share the GPU across concurrent requests. A naive server processes requests one at a time, so the expensive card sits mostly idle between tokens. The fix is to keep many requests in flight together — and that is exactly the machinery a dev tool doesn't bother building.

The common mistake is assuming "it works locally" means "it will scale," and reaching for more GPUs when throughput sags. More hardware rarely helps a single-stream server, because the problem is utilization, not capacity — you're paying for a whole card and using a sliver of it. The right move is a smarter engine on the same card. That engine is vLLM, and it earns its place through three techniques you already met in the Inference track: continuous batching (slot new requests into the running batch the instant a slot frees up, instead of waiting for a fixed batch to finish), PagedAttention (manage the KV-cache in small pages like an operating system manages RAM, so memory isn't wasted on padding and you fit far more concurrent sequences), and quantization (compress the weights so a bigger model — or more cache headroom — fits on the card you have).

Ollama got you running; vLLM gets you serving. It's the production engine from IC6 — PagedAttention, continuous batching, quantization — behind the same OpenAI-compatible API. This chapter is the self-hosting view of the inference track.

Stand it up intermediate

Brace yourself for an anticlimax: the single hardest-sounding thing in this chapter — running a production LLM server — is one command. Not a Dockerfile, not a Kubernetes manifest, not a framework you have to learn. You install vLLM, you point it at a model name, you pick a port. That's genuinely it for a first server. The complexity people expect here mostly lives in the flags, and the flags are just the IC-track levers wearing command-line clothes.

It helps to picture what that one command sets in motion. vLLM downloads the open weights from Hugging Face (once — they're cached after that), loads them onto the GPU, and then stands up a web server that speaks the OpenAI wire protocol. That last part is the quiet genius of the design: instead of inventing its own API that you'd have to write a new client for, vLLM mimics the exact HTTP shape OpenAI uses — same /v1/chat/completions path, same JSON request and response. So every tool, SDK, and snippet in the world that already talks to OpenAI can talk to your box with zero code changes beyond the address. You'll see that pay off in the very next lab.

The mistake to avoid is treating the flags as magic incantations to copy blindly. Each one is a deliberate memory-versus-throughput decision you're making on the model's behalf: --quantization trades a little quality for a much smaller memory footprint; --max-num-seqs sets how wide the continuous batch runs (bigger = more concurrent users, but more KV-cache memory); --gpu-memory-utilization hands vLLM a slice of the card and tells it "budget weights and cache inside this." Read the launch command below not as boilerplate but as four tuning knobs — and know that when something goes wrong, it's almost always one of these two: not enough memory (turn the dials down) or not enough throughput (turn them up, if memory allows). That single tension is the entire craft of serving.

serve.shpip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \                 # 4-bit (IC2)
  --max-num-seqs 128 \                # continuous-batch width (IC4)
  --gpu-memory-utilization 0.90 \     # weights + KV-cache budget (IC3)
  --host 0.0.0.0 --port 8000

client.pyfrom openai import OpenAI
client = OpenAI(base_url="http://your-server:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello from my own server."}])
print(r.choices[0].message.content)

Sizing the GPU advanced

The question that will actually decide your bill. "Will this model fit on this GPU, and how many people can it serve at once?" — you cannot answer it by eyeballing the model's parameter count, and getting it wrong is expensive in both directions. Guess too small and vLLM crashes with an out-of-memory error the moment real traffic arrives. Guess too big and you rent a monster card that runs at 15% and quietly bleeds money. Sizing is the difference between a service that survives launch day and one that either falls over or wastes your budget.

Here's the mental model that makes it click. GPU memory (VRAM) is a fixed drawer, and three things compete for the space: the weights (the model itself, loaded once), the KV-cache (scratch memory that grows with every concurrent request and every token of context), and a bit of overhead (the framework, activations, CUDA itself). Most beginners fixate on the weights because that's the number on the model card — but the weights are the easy part; they're a one-time, fixed cost. It's the KV-cache that's the surprise, because it scales with concurrency. Picture a parking lot: the building (weights) takes a fixed footprint, but every car you let in (each concurrent request) needs its own space, and the context length is how big each car is. Run out of parking and the next car — the next user — simply can't get in.

The common trap is sizing only for the weights and then being baffled when the server OOMs under load even though "the model clearly fits." It fit when idle; it didn't fit once thirty users each demanded their own slice of KV-cache. So the real formula is: weights are your entry ticket, and whatever VRAM is left over — divided by the per-request cache cost — is your concurrency ceiling. That's precisely why --max-num-seqs and --gpu-memory-utilization are the two dials you reach for: one caps how many cars you admit, the other decides how much of the drawer vLLM is allowed to claim. Work the numbers below rather than trusting a vibe — the Practice ladder turns this exact reasoning into a calculator you can reuse on any card.

Memory must hold weights + KV-cache + overhead. Use the IC1–IC3 math: a 4-bit 8B model is ~5–6 GB of weights, leaving the rest of the card for KV-cache (which sets how many concurrent requests you can serve). gpu-memory-utilization tunes that split.

ADDS = {
    "continuous batching": "serve many requests together, high GPU utilization",
    "PagedAttention":       "manage KV-cache memory in pages, fit more concurrency",
    "quantization":         "run bigger models in less VRAM",
    "OpenAI-compatible API":"existing client code works unchanged",
}
def why_vllm():
    for feature, benefit in ADDS.items():
        print(f"- {feature}: {benefit}")

why_vllm()

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \        # 4-bit weights (IC2) -> less VRAM
  --max-num-seqs 128 \        # continuous-batch width (concurrency)
  --max-model-len 8192 \      # context length ceiling
  --gpu-memory-utilization 0.90  # fraction of VRAM vLLM may use

BYTES = {"fp16": 2, "awq": 0.5, "int8": 1}   # ~bytes per weight param

def weights_gb(params_b, quant):
    return params_b * 1e9 * BYTES[quant] / (1024**3)

def kv_gb_per_seq(layers, hidden, ctx_len, bytes_per=2):
    # 2 (K and V) * layers * hidden * ctx * bytes, in GB
    return 2 * layers * hidden * ctx_len * bytes_per / (1024**3)

def fits(gpu_gb, params_b, quant, layers, hidden, ctx, seqs):
    total = weights_gb(params_b, quant) + kv_gb_per_seq(layers, hidden, ctx) * seqs
    return total, total <= gpu_gb

# 8B AWQ, 32 layers, hidden 4096, ctx 8192, 32 concurrent seqs, on a 24GB card
total, ok = fits(24, 8, "awq", 32, 4096, 8192, 32)
print(f"needs ~{total:.1f} GB -> {'FITS' if ok else 'TOO BIG'} on 24GB")

BYTES = {"fp16": 2, "awq": 0.5}
def weights_gb(pb, q): return pb * 1e9 * BYTES[q] / (1024**3)
def kv_per_seq(layers, hidden, ctx): return 2*layers*hidden*ctx*2 / (1024**3)

def max_seqs(gpu_gb, params_b, quant, layers, hidden, ctx, reserve=0.10):
    budget = gpu_gb * (1 - reserve) - weights_gb(params_b, quant)
    if budget <= 0:
        return 0                       # weights alone don't fit
    return int(budget // kv_per_seq(layers, hidden, ctx))

n = max_seqs(24, 8, "awq", 32, 4096, 8192)
print(f"set --max-num-seqs {n} to stay within 24GB")

def chat_payload(model, prompt):
    return {"model": model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 256}
print(chat_payload("meta-llama/Llama-3.1-8B-Instruct", "hi"))

from openai import OpenAI
client = OpenAI(base_url="http://your-vllm-host:8000/v1", api_key="not-needed")
r = client.chat.completions.create(**chat_payload("meta-llama/Llama-3.1-8B-Instruct", "hi"))
print(r.choices[0].message.content)

import math

def plan(target_qps, per_gpu_qps, gpu_hourly_usd, headroom=0.30):
    effective = per_gpu_qps * (1 - headroom)      # never run a GPU flat out
    replicas = math.ceil(target_qps / effective)
    monthly = replicas * gpu_hourly_usd * 24 * 30
    return replicas, round(monthly, 2)

replicas, cost = plan(target_qps=40, per_gpu_qps=12, gpu_hourly_usd=1.20)
print(f"{replicas} GPU replicas, ~${cost}/mo")
# 40 / (12*0.7)=~4.76 -> 5 replicas; 5*1.20*720 = $4320/mo

Knowledge check check yourself