Local & Open Models · Project LM

Self-host an open model

The capstone: serve a quantized open model behind an OpenAI-compatible API, run real course code against it, benchmark vs a hosted API, and write a build-vs-buy memo from the numbers.

⏱️ ~5 hours🏗️ Capstone project🎯 Advanced→Expert

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

a GPU + pip install vllm (serving engine)
pip install "llama-cpp-python[server]" — runs on CPU / Apple Silicon, no GPU needed

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

What you'll buildA self-hosted open model behind an OpenAI-compatible API that a course lab uses unchanged — quantized (IC2), served on vLLM (LM3/IC6), benchmarked against a hosted API, with a break-even memo (LM5). Proof you can run models on infra you control.

Learning objectives

Choose and serve an open model appropriate to your hardware.
Point real course code at it with a one-line change.
Benchmark cost and latency vs a hosted API.
Write a build-vs-buy recommendation from the numbers.

▶ Runnable companionThe code in this lesson is also saved under code/proj-lm-selfhost/ in the course, with a README. Run the scripts or copy the configs directly.

The method advanced

Self-host and benchmark, end to end

LM1: pick a model+quant for your hardware (start 8B, 4-bit).
LM3/LM4: serve it — vLLM if you have a GPU, GGUF/llama.cpp if not — on an OpenAI-compatible API.
Repoint: change one earlier lab's base_url to your endpoint; confirm it works.
IC1/IC4: benchmark TTFT, TPOT, throughput at concurrency 1 and 32.
LM5: compute cost/1k-tokens vs a hosted API and the break-even volume.
Memo: recommend hosted vs self-host for this workload, with the deciding factor named.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

The self-host stack

selfhost.sh# GPU path (vLLM):
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct --quantization awq \
  --max-num-seqs 128 --gpu-memory-utilization 0.90 --port 8000

# No-GPU path (llama.cpp):
python -m llama_cpp.server --model ./llama-3.1-8b.Q4_K_M.gguf --n_gpu_layers 0

# Then point any lab at http://localhost:8000/v1 and run the IC4 load-test.

▶ How this works

This is the capstone command sheet: it shows both ways to stand up your own model server (with or without a GPU) and how to plug a real course lab into it. Pick the path that matches your hardware.

GPU path — the python -m vllm.entrypoints.openai.api_server ... block is the vLLM server from LM3: it serves Llama-3.1-8B, --quantization awq keeps it small (4-bit), --max-num-seqs 128 batches many requests, and --port 8000 exposes the API.
No-GPU path — python -m llama_cpp.server --model ...gguf --n_gpu_layers 0 is the llama.cpp server from LM4: it runs a GGUF file entirely on the CPU (0 GPU layers).
Either path ends at the same address, http://localhost:8000/v1, so the final comment applies to both: point any earlier course lab at that URL and run the load-test to measure it.

What the output means: A running OpenAI-compatible endpoint on port 8000 — from either engine — that your existing labs can call unchanged.

Try this: Run the same lab against both your endpoint and a hosted API, then compare speed and cost. Those numbers are exactly what your build-vs-buy memo (LM5) is built from.

The deliverable is the decisionThe point isn't just that you can self-host — it's that you can say, with numbers, when you should. A working endpoint plus a defensible break-even memo is the real output.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Pick a model + quant that fits your hardwareBeginner

Context: The first self-hosting decision is fit: model, precision, and engine follow from your hardware. An 8B at 4-bit needs only a few GiB of weights — match the engine to the metal you actually have.

Your task: Compute the weight VRAM for 8B at 4-bit and pick the serving path from available memory.

Requirements:

Compute weight VRAM = params × bits/8
Add a runtime overhead margin
Pick vLLM on GPU when VRAM is sufficient
Fall back to llama.cpp GGUF on CPU/Apple Silicon when the GPU is too small
Runs offline for a given VRAM figure

💡 Hint: Add ~20% overhead on top of raw weights before comparing to available VRAM; the comparison chooses the engine.

Show solution

The first decision is fit — model, precision, and engine follow from your hardware:

def weight_gb(params_b, bits): return params_b*1e9*(bits/8)/1024**3

def pick_path(params_b, bits, gpu_vram_gb):
    need = weight_gb(params_b, bits) * 1.2      # +20% runtime overhead
    if gpu_vram_gb >= need:
        return f"vLLM on GPU (need {need:.1f} GiB, have {gpu_vram_gb})"
    return f"llama.cpp GGUF on CPU/Apple Silicon (GPU too small: {need:.1f} GiB)"

print(pick_path(8, 4, gpu_vram_gb=24))   # vLLM on GPU
print(pick_path(8, 4, gpu_vram_gb=0))    # llama.cpp GGUF path

Llama-3.1-8B at 4-bit needs ~5 GiB of weights; it runs on a modest GPU via vLLM or entirely on CPU/Apple Silicon via llama.cpp. Match the engine to the metal you actually have.

Exercise 2 · Serve behind an OpenAI-compatible APIIntermediate

Context: Bring the model up on an OpenAI-compatible /v1 endpoint on both paths. Standardizing on the OpenAI protocol means the benchmark harness and every earlier lab talk to it with zero client changes — the endpoint is the only thing that moved.

Your task: Write the two labelled launch lines (vLLM and llama.cpp) and a runnable client sanity call.

Requirements:

A vLLM launch line (labelled needs-GPU)
A llama.cpp GGUF launch line for the CPU path (labelled)
Both expose an OpenAI-compatible /v1 endpoint
A provider-agnostic client call runs once a server is up
The client code is identical regardless of path

💡 Hint: Point an OpenAI client at localhost:8000/v1; the client doesn't care which engine is behind it.

Show solution

Both engines expose the same protocol, so the client code is identical regardless of path:

# --- needs GPU: vLLM ---
# python -m vllm.entrypoints.openai.api_server \
#   --model meta-llama/Llama-3.1-8B-Instruct --quantization awq \
#   --max-num-seqs 128 --port 8000

# --- CPU path: llama.cpp (GGUF) ---
# python -m llama_cpp.server \
#   --model ./llama-3.1-8b.Q4_K_M.gguf --n_gpu_layers 0 --port 8000

# client is provider-agnostic once a server is up:
from openai import OpenAI                    # pip install openai
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
r = client.chat.completions.create(model="local",
    messages=[{"role": "user", "content": "say hi in 3 words"}])
print(r.choices[0].message.content)

Standardizing on the OpenAI protocol means the benchmark harness and every earlier lab talk to your self-hosted model with zero client changes — the endpoint is the only thing that moved.

Exercise 3 · Repoint an earlier lab and benchmark at concurrency 1 and 32Advanced

Context: Prove real course code runs against your endpoint, then load-test it. Concurrency 1 shows single-user latency; concurrency 32 shows the throughput you're actually paying the GPU for — both feed the cost model.

Your task: Build a concurrency benchmark that reports TTFT/TPOT/throughput at 1 and 32 concurrent clients.

Requirements:

Repoint an earlier lab at your endpoint to prove real code runs
Benchmark at concurrency 1 and 32
Report TTFT, TPOT, and aggregate throughput at each level
Model contention (per-token latency rises with concurrency)
Both numbers feed the cost model

💡 Hint: A box you only ever run at concurrency 1 is wildly overpriced per token — that's why the aggregate throughput at 32 matters.

Show solution

Model the concurrency benchmark offline; the real harness fires the same shape at your live endpoint:

def bench(concurrency, base_tpot_ms=22, out_tokens=256):
    tpot = base_tpot_ms * (1 + 0.02*(concurrency-1))    # contention
    ttft = 350 + 4*concurrency
    per_req_ms = ttft + tpot*out_tokens
    agg_throughput = concurrency * (out_tokens / (per_req_ms/1000))
    return {"conc": concurrency, "ttft_ms": round(ttft),
            "tpot_ms": round(tpot,1), "agg_tok_s": round(agg_throughput)}

for c in (1, 32):
    print(bench(c))
# conc 1  : modest throughput, low latency
# conc 32 : ~10-20x aggregate throughput, higher TTFT/TPOT

Concurrency 1 shows single-user latency; concurrency 32 shows the throughput you're actually paying the GPU for. Both numbers feed the cost model — a box you only ever run at concurrency 1 is wildly overpriced per token.

Exercise 4 · Cost per 1k tokens and the break-even volumeExpert

Context: Self-hosting is a fixed GPU cost; an API is per-token. Self-hosting only wins at high, steady utilization — an idle GPU still bills — and the break-even volume is the single number a build-vs-buy memo turns on.

Your task: Compute self-hosted $/1k tokens from throughput + GPU hourly, and the monthly volume where self-hosting undercuts the API.

Requirements:

Self-host $/1k = GPU hourly over tokens-per-hour at your throughput
It's fixed regardless of volume; the API scales linearly
Compute the monthly break-even token volume
Return no break-even when self-host is never cheaper at that utilization
Runs offline over throughput, GPU hourly, and API price

💡 Hint: Tokens/hour = throughput × 3600; if self-host $/1k already beats the API there's no break-even, otherwise amortize the monthly GPU cost against it.

Show solution

The break-even is where fixed GPU cost divided by tokens equals the API's per-token price:

def self_cost_per_1k(agg_tok_s, gpu_hourly):
    tok_per_hour = agg_tok_s * 3600
    return gpu_hourly / tok_per_hour * 1000

def breakeven_tokens_month(gpu_hourly, api_price_per_1k, agg_tok_s):
    monthly_gpu = gpu_hourly * 24 * 30
    # self-host $/1k is fixed regardless of volume; API scales linearly
    sc = self_cost_per_1k(agg_tok_s, gpu_hourly)
    if sc >= api_price_per_1k:
        return None                       # never cheaper at this utilization
    # you must run enough tokens to amortize the GPU below API cost
    return monthly_gpu / (api_price_per_1k/1000)

sc = self_cost_per_1k(agg_tok_s=6000, gpu_hourly=2.0)
print(f"self-host ${sc:.5f}/1k vs API $0.00060/1k")
print("break-even tokens/mo:", breakeven_tokens_month(2.0, 0.00060, 6000))

Self-hosting only wins at high, steady utilization — an idle GPU still bills. The break-even volume is the single number a build-vs-buy memo turns on.

Exercise 5 · Write the build-vs-buy memo from the numbersProfessional

Context: Turn benchmarks into a decision. The deciding factor is rarely raw price — it's volume steadiness, latency, data residency, or whether anyone can operate the GPU at 3am. Name it explicitly so the decision survives review.

Your task: Produce a memo that states the deciding factor and recommends build or buy with the numbers behind it.

Requirements:

Consider volume vs break-even, latency, data residency, and ops headcount
Collect the reasons that actually favour each side
Recommend BUILD or BUY from those factors
Every recommendation cites a measured number
Runs offline over the decision inputs

💡 Hint: Gate BUILD on having someone to operate the box; volume or residency can force build, but zero ops staff pushes toward buy.

Show solution

The memo is a structured decision, not a vibe — every recommendation cites a measured number:

def memo(monthly_tokens, breakeven, self_p95, api_p95,
         residency_required, ops_headcount):
    reasons = []
    volume_favors = monthly_tokens >= (breakeven or float("inf"))
    if volume_favors: reasons.append(f"volume {monthly_tokens:,} >= break-even")
    if self_p95 < api_p95: reasons.append("self-host p95 lower")
    if residency_required: reasons.append("data must stay in-house")
    if ops_headcount == 0: reasons.append("no ops staff -> favors buy")
    decision = "BUILD" if (volume_favors or residency_required) and ops_headcount else "BUY"
    return {"decision": decision, "reasons": reasons}

print(memo(monthly_tokens=5_000_000_000, breakeven=3_000_000_000,
           self_p95=900, api_p95=700, residency_required=True, ops_headcount=2))

The deciding factor is rarely raw price — it's usually volume steadiness, latency, data residency, or whether you have anyone to operate the GPU at 3am. Name it explicitly so the decision survives review.

Exercise 6 · Add a health check, autoscale trigger, and failover to the APIIndustry scenario

Context: A self-hosted box will occasionally OOM or die. Resilience turns a hobby endpoint into production infra — degrade to the API rather than go dark, with the API as a pricier-but-always-up safety net.

Your task: Add a health probe, an autoscale trigger on queue depth, and automatic failover to the hosted API.

Requirements:

A health check that reflects GPU state and queue depth
An autoscale trigger that computes replicas needed from queue depth
Failover routes to the hosted API when the box is unhealthy
The system degrades to the API rather than going dark
Demonstrate healthy routing, a scale decision, and a failover

💡 Hint: Route to self-host while healthy and to the API otherwise; the API keeps the product up while the GPU recovers.

Show solution

Resilience turns a hobby endpoint into production infra — degrade to the API rather than go dark:

class Router:
    def __init__(self, api_fallback): self.api = api_fallback; self.healthy = True
    def health(self, gpu_ok, queue_depth, max_queue=200):
        self.healthy = gpu_ok and queue_depth <= max_queue
        return self.healthy
    def should_scale(self, queue_depth, per_replica=64):
        return max(1, -(-queue_depth // per_replica))    # replicas needed
    def route(self, gpu_ok, queue_depth):
        if self.health(gpu_ok, queue_depth):
            return "self-host"
        return "API failover"                # degrade, don't die

r = Router(api_fallback="hosted")
print(r.route(gpu_ok=True,  queue_depth=50))    # self-host
print(r.should_scale(queue_depth=300))          # 5 replicas
print(r.route(gpu_ok=False, queue_depth=10))    # API failover

Health + autoscale + failover is the minimum operational story for depending on a box you own. The API becomes your safety net: pricier per token, but it keeps the product up while the GPU recovers.

✓ Checkpoint — you can move on when you can…

Serve an open model appropriate to your hardware.
Run real course code against your endpoint.
Benchmark cost and latency vs a hosted API.
Recommend build-vs-buy from the numbers.

📋 Grade your self-host — can you say when to build vs buy, with numbers?

Dimension	Meets the bar	Above the bar (staff)
Correct serving	An open model is served behind an OpenAI-compatible endpoint and a real course lab calls it unchanged (one-line `base_url` swap).	The engine matches the hardware (vLLM on GPU, llama.cpp/GGUF on CPU) and the endpoint handles the lab's real request shapes without special-casing.
Quantization tradeoff	A quant level appropriate to the hardware is chosen (e.g. 8B, 4-bit) and it loads and runs.	The quality cost of the quant is measured on the task, and the choice is justified as the smallest that still meets the quality bar.
Throughput measured	TTFT, TPOT, and throughput are benchmarked at concurrency 1 and 32 against the endpoint.	The measurement uses a proper load-test harness and reports tail latency, so the throughput claim holds under real concurrency.
Cost vs API	Cost/1k-tokens for the self-host is computed and compared to a hosted API for the same workload.	The break-even volume is calculated (including GPU/idle time), so you can name the traffic level where self-host wins.
The decision (memo)	A build-vs-buy memo recommends hosted or self-host for this workload and names the deciding factor.	The memo is defensible: it states the assumptions, the sensitivity to volume, and when the recommendation would flip.

Score each dimension 0 (missing), 1 (meets bar), or 2 (above bar). 0–4: keep building. 5–8: a solid, defensible submission. 9–12: staff-level — you could hand this to a reviewer and defend every call. Any dimension at 0 blocks shipping regardless of the total.

Knowledge check check yourself

✓ Knowledge check

Why does the project put the self-hosted model behind an OpenAI-compatible API and repoint an earlier lab's base_url at it?

Show answer

Because it lets real, unchanged course code run against your own model with a one-line change — proving the endpoint works in a real workflow. The OpenAI-compatible interface means no client rewrite is needed.

✓ Knowledge check

What decides the build-vs-buy recommendation in the closing memo?

Show answer

The measured numbers: cost/1k-tokens self-hosted vs a hosted API, plus latency, and the break-even volume where self-hosting becomes cheaper. The memo names the deciding factor for this specific workload rather than a generic preference.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

The method advanced

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Projects