Working with LLMs: APIs, SDKs, Parameters & Open-Source Models
The practical mechanics of driving an LLM: how the request/response actually works, what every parameter does, how SDKs wrap it, and the real decision between a hosted API (like Anthropic's) and an open-source model you run yourself. This is the plumbing under every other chapter.
Learning objectives
- Compare closed APIs vs open-source models for a task.
- Reason about the tradeoffs: capability, cost, control, privacy.
- Route between models by requirement.
- Set a model-sourcing policy for a team.
code/pe4-llms-apis-oss/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.1 · The landscape essential
You can consume LLMs three ways: closed APIs (Claude, GPT — most capable, easiest), open-weight models you host (Llama, Mistral, Qwen — control + privacy), or managed open models (hosted OSS). The right choice is per-task, not ideological.
| Option | Best at | Cost of |
|---|---|---|
| Closed API | capability, zero ops | per-token fee, data leaves |
| Self-hosted open | privacy, control, fixed cost | ops + a capability gap |
| Managed open | middle ground | less control than self-host |
2 · The core tradeoffs essential
Four axes decide it: capability (closed frontier still leads on hard tasks), cost (open wins at high volume if you have GPUs/ops), control (open = you own the model/version), privacy/residency (open = data never leaves). Weight them by your actual constraints.
3 · Intermediate — route by requirement intermediate
choose_source.pydef choose_source(needs_frontier, data_sensitive, high_volume, has_gpu_ops):
if data_sensitive and (has_gpu_ops or not needs_frontier):
return "self-hosted open (data stays in-house)"
if needs_frontier:
return "closed API (capability)"
if high_volume and has_gpu_ops:
return "self-hosted open (cost at scale)"
return "closed API (simplest)"
print(choose_source(True, False, False, False)) # hard task -> API
print(choose_source(False, True, True, True)) # sensitive + volume + ops -> self-host
print(choose_source(False, False, False, False)) # simple -> API
closed API (capability)
self-hosted open (data stays in-house)
closed API (simplest)This lab turns "which kind of LLM should I use?" into a small decision function. There are three real options: a closed API (like Claude — most capable, someone else runs it, you pay per use and your data leaves your servers), a self-hosted open model (you run an open-weight model like Llama on your own GPUs — data stays in-house, fixed cost, more work), and the simplest default. The function reads four yes/no facts about your task and returns the recommendation.
- The inputs are four true/false flags:
needs_frontier(is this a genuinely hard task that needs the most capable model?),data_sensitive(must the data never leave your infrastructure?),high_volume(huge number of calls?), andhas_gpu_ops(do you have the GPUs and staff to run a model yourself?). - The checks run top to bottom, and the first match wins. First: if the data is sensitive and you can either self-host or the task isn't that hard, keep it in-house —
self-hosted open (data stays in-house). Privacy is checked first because it's usually a hard legal requirement, not a preference. - Next:
if needs_frontier:— a genuinely hard task goes to theclosed API, because the top hosted models still lead on the hardest reasoning. - Next:
if high_volume and has_gpu_ops:— lots of calls and the ability to run your own model means self-hosting saves money at scale. - If none of those fire, the last line returns
closed API (simplest)— when nothing forces your hand, the easiest option wins. The threeprint(...)lines call the function with different fact patterns so you can see each branch in action.
What the output means: Three lines, one per test call. A hard task returns the API; sensitive + high-volume + you-have-ops returns self-hosted; a plain task returns the API as the simplest choice. In short: the logic routes each task to the right source instead of picking one model for everything.
Try this: Change the third call to choose_source(False, True, False, False) (sensitive data, but no GPU ops and not a hard task) and predict the answer before running. It should still say self-hosted — because sensitive data must stay in-house even when you'd rather not run it yourself.
4 · Advanced — the capability/cost frontier advanced
Open models have closed much of the gap on common tasks but still trail the frontier on the hardest reasoning. The economic crossover: below some volume, an API is cheaper (no fixed GPU/ops cost); above it, self-hosting wins. Know your break-even (Local Models track).
breakeven.pydef breakeven(api_cost_per_call, self_host_fixed_monthly, self_host_marginal):
# find monthly call volume where self-host becomes cheaper
denom = api_cost_per_call - self_host_marginal
if denom <= 0: return "self-host never cheaper at these rates"
return f"break-even at ~{round(self_host_fixed_monthly/denom):,} calls/month"
print(breakeven(api_cost_per_call=0.01, self_host_fixed_monthly=3000, self_host_marginal=0.0005))
break-even at ~315,789 calls/monthThe previous lab decided which source; this one answers the money question: at what call volume does running your own model become cheaper than paying an API per call? An API charges a fee every single call but has no fixed cost. Self-hosting has a big fixed monthly cost (renting GPUs, whether you use them or not) plus a tiny marginal cost per call. Below some volume the API wins; above it, self-host wins. That tipping point is the break-even.
- The three inputs are:
api_cost_per_call(what the API charges you each call, e.g.0.01= one cent),self_host_fixed_monthly(your flat monthly GPU bill, e.g.3000dollars), andself_host_marginal(the tiny extra cost of one more call on your own server, e.g.0.0005). denom = api_cost_per_call - self_host_marginalis how much you save per call by self-hosting instead of calling the API (here0.01 - 0.0005 = 0.0095per call).- The guard
if denom <= 0:handles the case where the API is already as cheap (or cheaper) per call — then self-hosting never catches up, no matter the volume, so it returns a plain-English message instead of a number. - Otherwise
self_host_fixed_monthly / denomdivides the fixed monthly bill by the per-call saving: "how many calls of savings do I need to pay off that fixed cost?" Theround(...)and:,just make it a clean number with comma separators. - The
print(breakeven(...))call passes the values by name (keyword arguments) so the numbers are self-documenting — you can read what each one is without checking the function definition.
What the output means: break-even at ~315,789 calls/month. Below ~316k calls a month the API is cheaper (you avoid the $3,000 fixed bill); above it, self-hosting wins because the per-call saving has covered that fixed cost. This one number is what turns "self-hosting feels cheaper" into an actual, defensible decision.
Try this: Drop the fixed cost to self_host_fixed_monthly=500 and re-run — the break-even falls sharply, because a smaller fixed bill is paid off with far fewer calls. Then set self_host_marginal=0.01 (equal to the API cost) and watch the guard message appear: with no per-call saving, self-hosting can never win.
5 · Professional — hybrid architectures professional
Real systems mix sources: a cheap open/small model handles the high-volume easy majority, escalating hard cases to a frontier API; sensitive data routes to a self-hosted model. This captures most of the cost/privacy benefit without sacrificing capability where it matters.
6 · Tech-lead — a model-sourcing policy tech-lead
A lead sets the policy: default source, what data may go to which providers (compliance), the break-even threshold for self-hosting, and an abstraction layer so switching sources is config (the provider-agnostic client from K7/DF4).
Exercise PE4.1 — Source the models for a system
Context: Sourcing decisions come alive across a mix of workloads with different needs — sensitive extraction, a high-volume classifier, and a hard reasoning task pull in different directions.
Your task: For three workloads (sensitive-data extraction, a high-volume classifier, a hard reasoning task), pick a source per workload, compute the self-host break-even, and write a one-paragraph sourcing policy including what data may leave your infra.
Requirements:
- Pick a source per workload with the routing logic
- Compute the self-host break-even volume
- Write a one-paragraph sourcing policy
- State explicitly which data classes may leave your infra
💡 Hint: Let privacy decide the sensitive workload before cost or capability, then use the break-even to justify the high-volume one.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: The three sourcing options — closed API, self-hosted open, managed open — trade along the same axes. Sourcing is a compliance and cost decision, not a preference.
Your task: Given what a project cares about most, name the sourcing option that best fits and what it costs you.
Requirements:
- Cover closed API, self-hosted open, and managed open
- State what each option is best at
- State what each option costs you (per-token fee, ops, control)
- Return a readable summary per option
- Make clear closed APIs lead on capability but your data leaves
💡 Hint: A lookup keyed by option is enough — the point is that each choice trades capability, privacy, ops, and cost against each other.
Show solution
The sourcing table as a lookup (pure stdlib):
OPTIONS = {
"closed API": ("capability, zero ops", "per-token fee, data leaves"),
"self-hosted open": ("privacy, control, fixed cost", "ops + capability gap"),
"managed open": ("middle ground", "less control than self-host"),
}
def describe(option):
best, cost = OPTIONS[option]
return f"{option}: best at {best}; you pay in {cost}"
for o in OPTIONS:
print(describe(o))Each option trades along the same axes: closed APIs lead on capability with zero ops but your data leaves and you pay per token; self-hosted open keeps data in-house at fixed cost but you own the ops and accept a capability gap. Sourcing is a compliance + cost decision, not a preference.
Context: The routing order matters: privacy is a hard compliance constraint checked first, then frontier capability, then cost at scale. When nothing forces a choice, the simplest option — a closed API — wins.
Your task: Encode the routing function that checks privacy first, then capability, then cost, and run it on three workloads.
Requirements:
- Check data sensitivity first as a compliance override
- Force in-house serving for sensitive data
- Route frontier-capability needs to a closed API
- Route high-volume-with-ops to self-hosted for cost
- Default to the simplest option and demonstrate three workloads
💡 Hint: Privacy is evaluated before capability and cost because it is a constraint, not a tradeoff — sensitive data forces in-house serving.
Show solution
The routing decision, exactly as the lesson defines it (pure logic):
def choose_source(needs_frontier, data_sensitive, high_volume, has_gpu_ops):
if data_sensitive and (has_gpu_ops or not needs_frontier):
return "self-hosted open (data stays in-house)"
if needs_frontier:
return "closed API (capability)"
if high_volume and has_gpu_ops:
return "self-hosted open (cost at scale)"
return "closed API (simplest)"
print(choose_source(True, False, False, False)) # closed API (capability)
print(choose_source(False, True, True, True)) # self-hosted open (in-house)
print(choose_source(False, False, False, False)) # closed API (simplest)Privacy is evaluated first because it is a hard compliance constraint, not a tradeoff: sensitive data forces in-house serving. Only after that do capability and cost-at-scale decide, and when nothing forces a choice the simplest option (a closed API) wins.
Context: Self-hosting has a fixed monthly cost plus a small marginal cost; APIs cost per call. The break-even volume makes "should we self-host?" a number, not an opinion.
Your task: Implement breakeven() — the monthly volume where self-hosting gets cheaper — and interpret the ~316k-calls number.
Requirements:
- Compute break-even as fixed cost divided by per-call savings
- Guard the case where marginal cost meets or exceeds the API price
- Return the break-even volume in calls per month
- Interpret the ~316k result against a realistic price set
- State that below break-even the API wins and above it self-host wins
💡 Hint: The denominator is the per-call saving (API price minus marginal); if it is non-positive, self-hosting can never win.
Show solution
The break-even calculator, as the lesson defines it (pure arithmetic):
def breakeven(api_cost_per_call, self_host_fixed_monthly, self_host_marginal):
denom = api_cost_per_call - self_host_marginal
if denom <= 0:
return "self-host never cheaper at these rates"
return f"break-even at ~{round(self_host_fixed_monthly/denom):,} calls/month"
print(breakeven(api_cost_per_call=0.01, self_host_fixed_monthly=3000,
self_host_marginal=0.0005))
# break-even at ~315,789 calls/monthBelow ~316k calls/month the API's pay-per-use wins; above it the fixed GPU cost amortizes and self-hosting is cheaper. If the marginal cost ever meets or exceeds the API price, self-hosting can never win — the break-even makes "should we self-host?" a number, not an opinion.
Context: Hybrid architectures route easy, high-volume requests to a cheap (often open) model and hard requests to a frontier API. Most traffic is easy, so this collapses the bill while keeping quality where it matters.
Your task: Model the hybrid router and the blended cost, showing the saving grow as the easy fraction rises.
Requirements:
- Route easy requests to a cheap model and hard ones to the frontier API
- Use illustrative per-call prices
- Compute the blended cost over a mixed stream
- Compare against sending everything to the frontier model
- Report the percent saved and note the need for a provider-agnostic client
💡 Hint: The saving tracks the easy fraction — a provider-agnostic client layer is what lets the router send each request wherever it is cheapest to answer well.
Show solution
Hybrid routing with a blended-cost readout (pure stdlib):
COST = {"cheap_open": 0.0005, "frontier_api": 0.01} # $ per call (illustrative)
def route(difficulty):
return "frontier_api" if difficulty == "hard" else "cheap_open"
def blended(requests):
total = sum(COST[route(r)] for r in requests)
return round(total, 4)
mix = ["easy"] * 95 + ["hard"] * 5
all_frontier = ["hard"] * 100
print("hybrid (95% easy):", blended(mix)) # tiny
print("all frontier :", blended(all_frontier)) # baseline
print("saving:", round(100*(blended(all_frontier)-blended(mix))/blended(all_frontier)), "%")Most traffic is easy, so routing it to a cheap open model and reserving the frontier API for the hard tail collapses the bill while keeping quality where it matters. The hybrid needs a provider-agnostic client layer so the router can send each request wherever it is cheapest to answer well.
Context: A sourcing policy is a compliance matrix plus a default plus a break-even threshold, encoded so every route is auditable — not left to a developer's judgement call.
Your task: Implement a policy that maps data classes to allowed sources and refuses to send restricted data to a closed API.
Requirements:
- Map each data class (public, internal, restricted) to allowed sources
- Never allow restricted data to leave your infra
- Honour a desired source when the data class permits it
- Fall back to an allowed source when the desired one is disallowed
- Return whether the route was permitted, and demonstrate a blocked case
💡 Hint: The matrix is the contract — a closed API must be unreachable for restricted data, and the fallback should prefer self-hosted.
Show solution
The sourcing policy as an auditable gate (pure logic):
ALLOWED = {
"public": {"closed_api", "managed_open", "self_hosted"},
"internal": {"managed_open", "self_hosted"},
"restricted": {"self_hosted"}, # never leaves our infra
}
DEFAULT = "closed_api"
def route_with_policy(data_class, desired):
allowed = ALLOWED[data_class]
if desired in allowed:
return {"source": desired, "ok": True}
fallback = "self_hosted" if "self_hosted" in allowed else next(iter(allowed))
return {"source": fallback, "ok": False,
"note": f"{desired} not allowed for {data_class}; using {fallback}"}
print(route_with_policy("public", "closed_api"))
print(route_with_policy("restricted", "closed_api")) # blocked -> self_hostedThe policy makes sourcing a documented, enforceable decision: a compliance matrix says which data class may reach which source, a default covers the common case, and restricted data can never be routed to a closed API. Encoding it means every route is auditable, not left to a developer's judgment call.
Context: As lead, a workload has crossed the break-even volume and needs to move in-house. You don't flip a switch — you route everything through an abstraction layer, then shadow and canary the open model against the API baseline so the capability gap is measured, not assumed.
Your task: Plan the migration behind a provider-agnostic layer, decide the cutover, and quantify the cost and capability tradeoff before committing.
Requirements:
- Quantify API spend versus self-host spend at the given volume
- Put all calls behind a provider-agnostic client first
- Shadow then canary the self-hosted model against the API baseline
- Ramp only if the measured quality gap is acceptable, keeping the API as fallback
- Return the spend comparison, quality gap, and phased plan
💡 Hint: The break-even proves cost; the shadow/canary comparison against the API baseline is what decides whether the migration actually ships.
Show solution
A migration plan that quantifies the tradeoff before cutover (pure stdlib):
def migration_plan(monthly_calls, api_cost, fixed, marginal, quality_gap):
api_spend = monthly_calls * api_cost
self_spend = fixed + monthly_calls * marginal
cheaper = self_spend < api_spend
steps = [
"1. Put all calls behind a provider-agnostic client (base_url swap)",
"2. Shadow self-hosted model on live traffic; compare quality offline",
"3. Canary 5% to self-hosted; watch quality + latency vs API baseline",
"4. Ramp only if quality gap is acceptable; keep API as fallback via gateway",
]
return {"api_spend": round(api_spend), "self_spend": round(self_spend),
"self_cheaper": cheaper, "quality_gap": quality_gap, "plan": steps}
import json
print(json.dumps(migration_plan(500_000, 0.01, 3000, 0.0005, quality_gap="~3% on hard set"),
indent=2))A lead does not flip a switch: route everything through an abstraction layer first, then shadow and canary the open model against the API baseline so the capability gap is measured, not assumed. The break-even proves cost, but the quality comparison — with the API kept as gateway fallback — decides whether the migration actually ships.
✓ Checkpoint — you can move on when you can…
- Compare closed API vs open/self-hosted.
- Weigh capability/cost/control/privacy.
- Route by requirement; find the break-even.
- Set a compliance-aware sourcing policy.
Knowledge check check yourself
The choose_source logic checks data sensitivity before capability. Why is privacy checked first, and what does the lesson say sourcing fundamentally is?
Show answer
In the API-vs-self-host break-even model, what does the break-even volume represent, and when can self-hosting 'never' win?