Buy vs build & model selection
The most expensive AI decisions are made before any code is written: API vs fine-tune vs build, which vendor, and how do we get out if it goes wrong. This lesson gives you a repeatable framework — a decision flow, a vendor scorecard, a total-cost-of-ownership model, and an explicit lock-in and exit strategy — so these choices are defensible rather than fashionable. Frameworks and public pricing structures are attributed; costs are illustrative and must be verified against current vendor pricing.
Learning objectives
- Choose deliberately between API, fine-tuning, and build-from-scratch using a decision flow tied to your actual constraints.
- Score vendors and models with a weighted scorecard instead of a demo-driven gut call.
- Build a total-cost-of-ownership (TCO) model that includes the costs teams forget.
- Reason explicitly about vendor lock-in and design an exit strategy before you sign.
- Present the recommendation as a decision a skeptical exec or board can approve.
1 · The four options, honestly framed
There is a spectrum, not a binary. From least to most ownership:
| Option | What you own | When it wins | The real cost |
|---|---|---|---|
| Hosted API | A prompt and an integration. | Fastest to value; frontier quality; variable load. | Per-token spend; data leaves your boundary; provider dependency. |
| API + retrieval/prompting | Context and orchestration. | You need your data, not new model behavior. | Same as API plus your retrieval infra. |
| Fine-tune / adapter | A customized model artifact. | Consistent format/style/domain; smaller model can hit quality. | Data prep, training, eval, and re-tuning as base models move. |
| Self-host open weights | The weights and the serving stack. | Residency/air-gap; high steady volume; no egress allowed. | GPUs, MLOps, on-call — a real infra team. |
| Build from scratch | Everything, including pre-training. | Almost never, for almost everyone. | $millions and a research org — a moat only if the model is the business. |
2 · A decision flow that ends the debate
Run the constraints in priority order — the first hard constraint that bites usually decides it, the same way the PHI constraint decided deployment in the clinical case study. Read the flow as: eliminate on hard constraints first, then optimize on cost and quality.
Worked reading of the flow: (1) If data legally cannot leave your boundary, you are pushed toward self-hosting regardless of cost. (2) If you need your own knowledge in answers, that's retrieval on top of an API — not fine-tuning; fine-tuning teaches behavior (format, tone, a narrow task), not facts. (3) Only steady, high volume amortizes the fixed cost of a serving team. (4) If your quality bar needs the current frontier, that usually means a hosted or managed-cloud frontier model, since your self-hosted open weights will trail it.
3 · The vendor / model scorecard
Demos are theater; a weighted scorecard forces you to state what you actually value and score every candidate on the same axes. Assign weights first (before you see scores) so you can't rationalize your favorite afterward.
| Dimension | Weight | What you're really measuring |
|---|---|---|
| Quality on your evals | 25% | Score on your task set, not a public leaderboard. |
| Cost at your volume | 20% | Blended $/request at projected load — verify current pricing. |
| Latency / throughput | 15% | p95 latency and tokens/sec under your concurrency. |
| Data & security posture | 15% | Retention, training-on-your-data toggle, region, certifications. |
| Reliability / SLA | 10% | Uptime history, rate limits, incident transparency. |
| Lock-in / portability | 10% | How hard is it to leave? (see §5) |
| Roadmap / support | 5% | Model cadence, deprecation policy, enterprise support. |
4 · Total cost of ownership — the costs teams forget
The sticker price (per-token or GPU-hour) is the cost people quote. TCO is the cost they actually pay. A defensible model includes:
| Cost bucket | API path | Self-host path |
|---|---|---|
| Direct inference | Per-token spend at projected volume | GPU rental/purchase + power |
| Engineering | Integration + prompt/eval upkeep | Serving stack, autoscaling, upgrades |
| On-call / ops | Mostly the vendor's problem | Yours — 24/7 for a critical service |
| Data & eval | Eval set upkeep | Same, plus fine-tune data pipeline |
| Migration / re-tune | Re-test on model updates | Re-tune adapters as base models move |
| Opportunity | Time-to-value is fast | Months of infra before first value |
5 · Lock-in and the exit strategy
Every choice creates dependency; the question is whether you can leave when you need to. Sources of lock-in: prompt/behavior coupling (prompts tuned to one model's quirks), proprietary features (a vendor-specific tool or format), fine-tune artifacts (an adapter you can't move), and data gravity (your embeddings/indexes in one store). You reduce lock-in the same way you reduce coupling in code: an abstraction seam and a tested escape route.
✓ Checkpoint — you can move on when you can…
- State the five points on the buy-vs-build spectrum and one situation where each wins.
- Explain why fine-tuning is the wrong tool for adding your company's knowledge.
- Assign weights to a vendor scorecard and justify why quality-on-your-evals is weighted highest.
- List three TCO buckets an engineer typically forgets, and say why the self-host crossover is later than it looks.
- Name two sources of lock-in and the abstraction that mitigates each.
A stakeholder says: "Our support answers are wrong because the model doesn't know our products. Let's fine-tune it on our docs." Why is fine-tuning likely the wrong first move, and what would you propose instead?
Show answer
Two vendors are within 3% on your eval set. Vendor A is 20% cheaper today; Vendor B has a documented deprecation policy, region controls, and a clean abstraction path. Which do you lean toward, and how do you defend it to a cost-focused exec?
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: The first skill is refusing the false binary and naming the precise option.
Your task: For five representative scenarios, name the best option on the buy-vs-build spectrum and one sentence of justification.
Requirements:
- A) A startup wants a support bot live in two weeks
- B) A bank cannot let any customer data leave its network
- C) A team needs outputs always in a strict JSON schema in a niche style
- D) A hobby project answers ~50 questions a day
- E) A company's entire product IS a novel foundation model
💡 Hint: Match the option to the dominant constraint: speed, residency, behavior, volume, or moat.
Show solution
A) Hosted API + prompting. Speed-to-value dominates; frontier quality out of the box, no infra.
B) Self-host open weights. A hard residency/egress constraint vetoes external APIs regardless of cost — accept the ops burden.
C) Fine-tune / adapter. The need is consistent behavior (schema + style), which is exactly what fine-tuning is for; likely on top of a smaller model to save cost.
D) Hosted API. 50 requests/day will never amortize a serving team; per-token spend is trivial.
E) Build from scratch (pre-train). The one case that justifies it — the model itself is the differentiated product and the moat. Almost no one else.
Context: A weighted scorecard turns a demo-driven argument into a defensible number.
Your task: Score two candidate models on the seven scorecard dimensions and compute a weighted total, then state the decision.
Requirements:
- Use the weights from §3 (they must sum to 100%)
- Invent plausible per-dimension scores (0–10) for two candidates
- Compute each weighted total and pick a winner
- Note what single dimension, if it changed, would flip the decision
💡 Hint: Set weights before scores; compute weighted sum = Σ(weight × score).
Show solution
Representative fill (scores 0–10):
| Dimension (weight) | Model A | Model B |
|---|---|---|
| Quality on our evals (25%) | 9 | 8 |
| Cost at volume (20%) | 6 | 9 |
| Latency (15%) | 7 | 8 |
| Data/security (15%) | 8 | 7 |
| Reliability (10%) | 8 | 7 |
| Lock-in (10%) | 6 | 8 |
| Roadmap (5%) | 8 | 7 |
Weighted totals: A = .25·9+.20·6+.15·7+.15·8+.10·8+.10·6+.05·8 = 7.50. B = .25·8+.20·9+.15·8+.15·7+.10·7+.10·8+.05·7 = 7.90. Winner: Model B, driven by cost and portability despite A's edge on raw quality.
Flip point: the quality gap is small — if our evals showed A leading by ~2 points instead of 1, or if cost weight dropped below ~12%, A would win. That sensitivity is the honest output of the scorecard.
Context: The self-host-is-cheaper argument usually dies when you add the forgotten costs.
Your task: Build a TCO comparison of an API path vs a self-host path and solve for the monthly request volume at which self-host becomes cheaper.
Requirements:
- API path: a per-request blended cost (verify against current pricing)
- Self-host path: fixed monthly infra + a loaded fraction of an MLOps engineer + on-call
- Write the crossover equation and solve for volume
- State the honest caveat about how the crossover moves
💡 Hint: Crossover: API_per_req × N = fixed_self_host_monthly ⇒ N = fixed / per_req.
Show solution
Illustrative model (verify all numbers). API: assume $0.004 blended per request. Self-host fixed monthly: GPU $3,000 + ~40% of a $200k-loaded MLOps eng ≈ $6,700 + on-call/upgrades $1,300 = $11,000/month fixed, roughly volume-independent until you saturate the box.
Crossover. Self-host wins when fixed < API·N ⇒ N > $11,000 / $0.004 = 2.75 million requests/month (~92k/day). Below that, the API is cheaper on TCO even though its marginal price looks high.
Caveats: the crossover collapses if you already run an MLOps team (fixed cost is sunk), and it rises if throughput per GPU is lower than assumed or if re-tuning is frequent. Every input drifts — recompute with current GPU and token prices, and don't quote a crossover you computed six months ago.
Context: Lock-in is a coupling problem; the fix is an abstraction seam and a tested escape route.
Your task: Design the abstraction layer and exit plan that keeps you portable across model providers.
Requirements:
- Define the interface your app calls instead of a vendor SDK directly
- List what the adapter normalizes (auth, message format, tool-calling, streaming, errors)
- State how your eval set stays provider-agnostic
- Describe the drill that proves you can actually switch
💡 Hint: The seam should make 'swap provider' a config change plus a re-run of your eval set.
Show solution
Interface. The app calls a single internal LLMClient.complete(messages, tools, opts); provider SDKs live only behind adapters implementing it. No vendor type leaks into business code.
Adapter normalizes: auth/config, message/role format, tool-calling schema (the biggest source of divergence), streaming events, token-usage reporting, and error/retry semantics (rate limits, refusals). Prompts live as versioned templates, not inlined per model.
Provider-agnostic evals. The eval harness targets LLMClient, so any provider runs the identical task set; you re-qualify a challenger in an afternoon, not a quarter.
The switch drill. Quarterly, run the full eval set against a second provider through its adapter and record quality/latency/cost. This proves the seam still works, keeps a warm fallback for an outage or deprecation, and gives you real leverage at renewal. An escape route you've never tested is a hope, not a plan.
Context: The deliverable of this whole decision is a one-page memo an exec can approve.
Your task: Write the recommendation memo for a representative 'add an AI assistant to our product' decision.
Requirements:
- State the recommendation up front (BLUF)
- Summarize the decision flow result and the scorecard winner
- Show the TCO at projected volume and the crossover
- Name the lock-in risk and the exit plan in one paragraph
- State what would change the recommendation
💡 Hint: Lead with the decision; put the analysis underneath for those who want it.
Show solution
Recommendation (BLUF): Build the assistant on a hosted frontier API + retrieval, behind our provider-abstraction layer, with a fine-tune deferred until RAG quality is proven. Do not self-host at current volume.
Why. Decision flow: data can leave under our enterprise agreement; the need is our knowledge (→ retrieval, not fine-tune); volume (~200k req/month) is well below the ~2.75M self-host crossover. Scorecard winner: Model B (7.9 vs 7.5), driven by cost and portability.
TCO. ~$800/month inference at projected volume + existing eng time, vs ~$11k/month fixed to self-host — API wins decisively until we ~14× our volume.
Lock-in / exit. Risk is prompt+tool coupling to one provider; mitigated by the abstraction seam and a quarterly switch drill against a second provider, keeping a warm fallback and renewal leverage.
What would change this: a hard residency mandate (→ self-host), a 10×+ volume jump past the crossover (→ revisit self-host), or a proven need for behavior RAG can't give (→ fine-tune). All numbers verified against current pricing as of the memo date.
Context: Representative scenario: an ambitious staff engineer proposes the company self-host and fine-tune an open-weight model to 'own our AI stack and cut costs.' Leadership asks you to adjudicate.
Your task: Evaluate the proposal rigorously and issue a decision with conditions, respecting that the engineer may be right.
Requirements:
- Steelman the proposal — when would it actually be correct?
- Run it through the decision flow and TCO honestly
- Identify the hidden costs and the key assumption the case rests on
- Issue a decision: proceed / pilot / decline, with measurable conditions
- Say how you'd de-risk if you proceed
💡 Hint: The right answer might be 'not yet' with a concrete trigger, not a flat no.
Show solution
Steelman. Self-hosting is genuinely correct if volume is high and steady, if there's a residency/egress constraint, or if the company already runs an MLOps team so the fixed cost is sunk. "Own our stack" also has real strategic value if AI is core to the product.
Decision-flow + TCO reality. Absent a residency mandate, the case rests entirely on volume clearing the crossover (~2.75M req/month in our illustrative model). Current volume is ~200k/month — 14× short. The "cut costs" claim compares GPU-hours to token spend and omits a loaded MLOps salary, on-call, upgrades, and re-tuning as base models move. Corrected, self-host is more expensive today, not less.
Hidden costs / key assumption. The proposal assumes a fine-tuned open model matches frontier quality on our tasks — unverified. It also assumes throughput per GPU we haven't measured. Both are testable.
Decision: decline as a full migration, approve a bounded pilot. Fund a 3-week pilot to (1) fine-tune the candidate and measure it on our held-out eval set vs the incumbent API, and (2) measure real throughput/GPU. Trigger to revisit self-host at scale: sustained volume past ~60–70% of the crossover or a residency mandate.
De-risking if we later proceed: keep the abstraction seam so the API stays a fallback, roll out behind a traffic split, and don't retire the API contract until the self-host path holds quality and reliability for a full quarter.