Working with LLMs: APIs, SDKs, Parameters & Open-Source Models

1 · The landscape essential

Here's a decision you'll be asked to defend in your first architecture review, and "I just used the one everyone uses" is not an answer that survives it. A team ships a feature on a frontier API because it's the fastest path to a demo. It works beautifully. Then the invoice arrives, or legal asks where the customer data went, or a single provider outage takes the whole product down for an afternoon — and suddenly the model you reached for reflexively is the thing being questioned in a postmortem. The choice of where your intelligence comes from is an engineering decision with cost, legal, and reliability consequences, and this chapter is about making it on purpose.

The most useful reframing: you are not choosing a model, you are choosing a supply chain. Think of it like sourcing electricity for a factory. You can buy from the grid (someone else builds the plant, you pay per kilowatt-hour, and you accept that the meter is always running and the utility knows your usage) — that's a closed API. You can build your own generators on-site (huge upfront cost, staff to run them, but the power never leaves your fence and the marginal cost is tiny) — that's self-hosting an open-weight model. Or you can rent a generator someone else maintains but that runs your fuel — a middle path, the managed open option. Nobody argues that on-site generators are morally superior to the grid; you'd match the source to the load. Model sourcing is the same, and treating it as a tribal identity ("we're an open-source shop") instead of a per-workload calculation is the most common mistake here.

The trap worth naming early: people collapse three genuinely different things into one axis. "Open vs closed" is really three questions stacked on top of each other — who runs the hardware, whether you can see and modify the weights, and whether your data leaves your infrastructure. Those usually move together, but not always: a managed open model gives you open weights without running the hardware, and some closed providers offer in-your-VPC deployments that keep data local. Keep the three questions separate in your head and the fog clears. As you read the table below, notice it isn't ranking the options — there is no winner — it's laying out what each one is best at and what each one costs you, because the right pick is entirely a function of which cost you can least afford to pay for the task in front of you.

You can consume LLMs three ways: closed APIs (Claude, GPT — most capable, easiest), open-weight models you host (Llama, Mistral, Qwen — control + privacy), or managed open models (hosted OSS). The right choice is per-task, not ideological.

2 · The core tradeoffs essential

The reason this decision feels hard is that there is no single "best" — every option is the best at something and the worst at something else. That's not a flaw in the choices; it's the definition of a tradeoff, and the engineers who struggle here are the ones hunting for a universally correct answer that doesn't exist. The skill isn't finding the winner. The skill is naming the axes, then figuring out which one your project cannot afford to lose — because once you know your binding constraint, the decision usually makes itself.

An analogy that keeps people honest: choosing a model source is like choosing where to live relative to your office. Downtown (the frontier API) is the most capable location — everything's close, zero maintenance — but you pay premium rent forever and you're surrounded by strangers. The suburbs (self-hosting) mean a long commute and you fix your own plumbing, but the mortgage is fixed, the yard is yours, and nobody's watching. There is no "correct" place to live; there's only the right place for someone with your budget, commute tolerance, and privacy needs. Anyone who tells you one is objectively better is selling something. The four axes below are just the rent, the commute, the ownership, and the privacy — priced for LLMs.

The common mistake is to weight all four axes equally and average them into a mushy "it depends." They are not equal, and crucially they are not even the same kind of thing. Capability and cost are tradeoffs — dials you can turn, trading a bit of one for a bit of the other. But privacy/residency is often not a dial at all: it's a hard constraint handed to you by a regulator or a contract, a yes/no gate that overrides everything else regardless of how much cheaper or smarter the alternative would be. That asymmetry is why the routing code you'll meet in §3 checks data sensitivity first, before it ever looks at capability or cost. Constraints filter; tradeoffs optimize. Get that order wrong and you'll build a beautifully cost-optimized system that's also illegal. Read the four axes below not as a scorecard to total up, but as a list where you first ask "which of these is non-negotiable for me?" and only then optimize the rest.

Four axes decide it: capability (closed frontier still leads on hard tasks), cost (open wins at high volume if you have GPUs/ops), control (open = you own the model/version), privacy/residency (open = data never leaves). Weight them by your actual constraints.

3 · Intermediate — route by requirement intermediate

Once you accept there's no single best source, a scary question follows: does that mean picking one is a judgement call I have to make fresh every time, hoping I remember all four axes? No — and that's the whole point of this section. The move that separates a hobby project from a system is turning the decision from a vibe into an algorithm: a small, explicit function that takes the facts about a task and returns the source, the same way every time, in an order you can defend in a review. When the routing lives in code instead of in someone's head, it's testable, auditable, and it doesn't quietly drift as different developers make different calls on similar tasks.

Think of it like a triage nurse at an emergency room. The nurse doesn't treat patients in the order they walked in, and they don't decide by gut — they run a fixed protocol: check the life-threatening symptoms first (the hard constraints), then severity, then convenience. A chest pain jumps the queue over a sprained ankle every time, because the protocol encodes what must be checked first. Our router is that protocol for LLM requests: it checks the one thing that can override everything — is this data sensitive? — before it ever weighs whether the task is hard or high-volume. The ordering isn't arbitrary; it mirrors §2's insight that a constraint filters before a tradeoff optimizes.

The mistake to avoid: writing the checks in the wrong order and letting a cheaper or smarter option "win" a request that should never have left your infrastructure in the first place. If you check capability before privacy, a hard task on sensitive data routes to the frontier API — and you've just leaked regulated data to optimize a metric you weren't even allowed to optimize. That's why the function below is a cascade of early returns where the first matching rule wins: privacy first as an override, then frontier capability, then cost-at-scale, and finally the simplest default when nothing forces your hand. Read it as a priority list, not a pile of independent ifs — the order is the policy.

choose_source.pydef choose_source(needs_frontier, data_sensitive, high_volume, has_gpu_ops):
    if data_sensitive and (has_gpu_ops or not needs_frontier):
        return "self-hosted open (data stays in-house)"
    if needs_frontier:
        return "closed API (capability)"
    if high_volume and has_gpu_ops:
        return "self-hosted open (cost at scale)"
    return "closed API (simplest)"

print(choose_source(True, False, False, False))    # hard task -> API
print(choose_source(False, True, True, True))       # sensitive + volume + ops -> self-host
print(choose_source(False, False, False, False))    # simple -> API

closed API (capability)
self-hosted open (data stays in-house)
closed API (simplest)

4 · Advanced — the capability/cost frontier advanced

"Self-hosting is cheaper" is one of the most expensive sentences in AI engineering — because it's true right up until it isn't, and the people who say it rarely know which side of the line they're on. A team sees the per-call price of an API, multiplies by their traffic, gets a scary number, and concludes they should run their own model to save money. Sometimes they're right. Often they've forgotten that a GPU rack costs the same whether it serves ten calls a day or ten million, and at their actual volume the "cheap" self-hosted option is burning money on idle hardware. This section gives you the one number that ends the argument: the break-even volume.

Here's the intuition, and it's the same maths as deciding whether to buy a car or take taxis. Taxis (the API) cost you nothing when you're not riding, but every trip has a fare — perfect if you travel occasionally. A car (self-hosting) hits you with a fixed monthly payment whether it leaves the driveway or not, plus a few cents of petrol per trip. If you drive constantly, the car wins; if you barely drive, the taxis win. There's a specific number of trips per month where the two cost exactly the same — that is your break-even, and above it you should own, below it you should rent. Nobody buys a car "because cars are cheaper"; they buy one because they cross the break-even. LLM sourcing is identical: a fixed monthly GPU bill versus a per-call fee, meeting at a crossover volume you can actually compute.

Two subtleties trip people up, so name them now. First, capability and cost are separate questions and you must not let one answer the other: open models have closed most of the gap on common work but still trail the frontier on the hardest reasoning, so "self-hosting is cheaper at our volume" does not mean "self-hosting is good enough for our hardest task" — those are two different tests, and §5's hybrid design exists precisely because the honest answer is often "cheap model for the easy majority, frontier for the hard tail." Second, the break-even only exists if self-hosting is actually cheaper per call in the first place. If the API's per-call price is already at or below your self-hosted marginal cost, no amount of volume will ever let you catch up — the fixed cost never gets paid off, and the right answer is simply "keep using the API." The calculator below computes the crossover and guards that impossible case, turning "self-hosting feels cheaper" into a defensible number.

Open models have closed much of the gap on common tasks but still trail the frontier on the hardest reasoning. The economic crossover: below some volume, an API is cheaper (no fixed GPU/ops cost); above it, self-hosting wins. Know your break-even (Local Models track).

breakeven.pydef breakeven(api_cost_per_call, self_host_fixed_monthly, self_host_marginal):
    # find monthly call volume where self-host becomes cheaper
    denom = api_cost_per_call - self_host_marginal
    if denom <= 0: return "self-host never cheaper at these rates"
    return f"break-even at ~{round(self_host_fixed_monthly/denom):,} calls/month"

print(breakeven(api_cost_per_call=0.01, self_host_fixed_monthly=3000, self_host_marginal=0.0005))

break-even at ~315,789 calls/month

5 · Professional — hybrid architectures professional

The whole "open vs closed" debate contains a hidden false premise: that you have to pick one. You don't. The most cost-effective production systems in the wild aren't loyal to any single source — they route each request to whichever model can answer it well for the least money, and it turns out that's a different model for different requests. Once you internalize that you're allowed to mix, the agonizing all-or-nothing decision from the earlier sections dissolves into something much easier: send most traffic somewhere cheap, and reserve the expensive frontier for the handful of requests that actually need it.

The reason this works so well comes down to a fact about real traffic that surprises people: the difficulty of your requests is wildly lopsided. Picture a hospital again — the vast majority of visits are routine (a checkup, a prescription refill) that a nurse practitioner handles fine, and only a small tail needs the specialist surgeon whose time costs ten times as much. Staffing the whole hospital with surgeons would be absurd; so would sending every request to the frontier model. If 95% of your queries are "classify this ticket" or "extract this field," a small, cheap, often open model nails them — and because that 95% is where nearly all your volume lives, moving it off the frontier collapses your bill while leaving quality untouched on the 5% that genuinely needs the big model. You're not sacrificing capability; you're spending it only where it's scarce.

The catch — and the reason this is a "professional" tier and not a beginner trick — is that a hybrid only stays sane if the routing and the model-switching are cleanly separated. The mistake is to hardcode "call the cheap model here, call the frontier there" all over your codebase; then every new model, every price change, every provider outage means hunting through business logic to rewire calls. The discipline is an abstraction layer: your code asks for "an answer to this request," and a single provider-agnostic client decides which model gets it, so the router's policy is one thing you can change in one place. That same interface is what §6 turns into a documented policy and what lets the migration in Exercise 6 swap a self-hosted model in behind the scenes without touching a caller. Hold that idea — route freely, but switch behind one door — as you read how the blended cost falls.

Real systems mix sources: a cheap open/small model handles the high-volume easy majority, escalating hard cases to a frontier API; sensitive data routes to a self-hosted model. This captures most of the cost/privacy benefit without sacrificing capability where it matters.

6 · Tech-lead — a model-sourcing policy tech-lead

Everything so far has been about making a good sourcing decision. This section is about the harder problem: making sure the right decision happens every time, on every request, even the ones written by a developer who's never read this chapter. A single engineer weighing the axes carefully is a good start. A team of twenty, each making that call independently under deadline pressure, is a data breach waiting to happen — someone will eventually send restricted data to a closed API "just to unblock the demo," and they won't be wrong to, because nobody told them not to. The job of a lead isn't to make the best individual choice; it's to make the correct choice the default and the wrong one impossible.

The reframing that matters: a sourcing policy is not documentation, it's infrastructure. Compare it to how a building handles fire safety. You don't post a memo saying "please avoid fires and exit calmly" and trust everyone to remember — you install fire doors that physically won't let the blaze spread and exit signs that make the right path obvious under stress. A policy that lives only in a wiki page is the memo; a policy encoded as a gate in your provider-agnostic client is the fire door. When "restricted data may not reach a closed API" is a rule the code enforces, it doesn't matter who's writing the feature or how rushed they are — the wrong route simply cannot execute. That's the difference between a policy people should follow and one they can't help but follow.

A real policy has four moving parts, and it's worth seeing them as a set because leaving any one out reopens a gap. First, a default source so the common, unremarkable request has an obvious home nobody has to think about. Second, a compliance matrix mapping each class of data to the sources it's allowed to touch — this is the part that must be enforced, not suggested, because it's usually a legal obligation rather than a guideline. Third, the break-even threshold from §4, written down so "should we self-host this workload yet?" is answered by a number the whole team shares instead of relitigated every quarter. And fourth, the abstraction layer — the provider-agnostic client from K7/DF4 — that makes swapping a source a config change rather than a code rewrite, which is what keeps the policy adjustable as prices, models, and regulations shift under you. The note and code below turn these four into an auditable gate.

A lead sets the policy: default source, what data may go to which providers (compliance), the break-even threshold for self-hosting, and an abstraction layer so switching sources is config (the provider-agnostic client from K7/DF4).

OPTIONS = {
    "closed API":       ("capability, zero ops", "per-token fee, data leaves"),
    "self-hosted open": ("privacy, control, fixed cost", "ops + capability gap"),
    "managed open":     ("middle ground", "less control than self-host"),
}
def describe(option):
    best, cost = OPTIONS[option]
    return f"{option}: best at {best}; you pay in {cost}"

for o in OPTIONS:
    print(describe(o))

def choose_source(needs_frontier, data_sensitive, high_volume, has_gpu_ops):
    if data_sensitive and (has_gpu_ops or not needs_frontier):
        return "self-hosted open (data stays in-house)"
    if needs_frontier:
        return "closed API (capability)"
    if high_volume and has_gpu_ops:
        return "self-hosted open (cost at scale)"
    return "closed API (simplest)"

print(choose_source(True,  False, False, False))  # closed API (capability)
print(choose_source(False, True,  True,  True))   # self-hosted open (in-house)
print(choose_source(False, False, False, False))  # closed API (simplest)

def breakeven(api_cost_per_call, self_host_fixed_monthly, self_host_marginal):
    denom = api_cost_per_call - self_host_marginal
    if denom <= 0:
        return "self-host never cheaper at these rates"
    return f"break-even at ~{round(self_host_fixed_monthly/denom):,} calls/month"

print(breakeven(api_cost_per_call=0.01, self_host_fixed_monthly=3000,
                self_host_marginal=0.0005))
# break-even at ~315,789 calls/month

COST = {"cheap_open": 0.0005, "frontier_api": 0.01}   # $ per call (illustrative)

def route(difficulty):
    return "frontier_api" if difficulty == "hard" else "cheap_open"

def blended(requests):
    total = sum(COST[route(r)] for r in requests)
    return round(total, 4)

mix = ["easy"] * 95 + ["hard"] * 5
all_frontier = ["hard"] * 100
print("hybrid (95% easy):", blended(mix))          # tiny
print("all frontier     :", blended(all_frontier)) # baseline
print("saving:", round(100*(blended(all_frontier)-blended(mix))/blended(all_frontier)), "%")

ALLOWED = {
    "public":     {"closed_api", "managed_open", "self_hosted"},
    "internal":   {"managed_open", "self_hosted"},
    "restricted": {"self_hosted"},          # never leaves our infra
}
DEFAULT = "closed_api"

def route_with_policy(data_class, desired):
    allowed = ALLOWED[data_class]
    if desired in allowed:
        return {"source": desired, "ok": True}
    fallback = "self_hosted" if "self_hosted" in allowed else next(iter(allowed))
    return {"source": fallback, "ok": False,
            "note": f"{desired} not allowed for {data_class}; using {fallback}"}

print(route_with_policy("public", "closed_api"))
print(route_with_policy("restricted", "closed_api"))   # blocked -> self_hosted

def migration_plan(monthly_calls, api_cost, fixed, marginal, quality_gap):
    api_spend  = monthly_calls * api_cost
    self_spend = fixed + monthly_calls * marginal
    cheaper = self_spend < api_spend
    steps = [
        "1. Put all calls behind a provider-agnostic client (base_url swap)",
        "2. Shadow self-hosted model on live traffic; compare quality offline",
        "3. Canary 5% to self-hosted; watch quality + latency vs API baseline",
        "4. Ramp only if quality gap is acceptable; keep API as fallback via gateway",
    ]
    return {"api_spend": round(api_spend), "self_spend": round(self_spend),
            "self_cheaper": cheaper, "quality_gap": quality_gap, "plan": steps}

import json
print(json.dumps(migration_plan(500_000, 0.01, 3000, 0.0005, quality_gap="~3% on hard set"),
                 indent=2))

Knowledge check check yourself