Standing up a shared AI platform

Meet "Vanta Corp" — a representative composite. A year into the AI gold rush, Vanta has a dozen product teams each shipping LLM features. Every team has independently wired up its own API keys, its own retry logic, its own half-built guardrails, and its own untracked spend. Nobody can answer "what are we spending on AI and on what?", two teams just leaked a key, and each team re-solves prompt-injection defense from scratch — badly. The mandate: stand up a shared AI platform so teams build features, not plumbing. This is the classic platform-engineering job applied to AI, dominated by three forces: you must remove duplicated toil, you must make the safe path the easy path, and you must make spend attributable and bounded.

1 · Framing the platform — what "shared" has to buy

Start with the meeting that always precedes this project. A VP finally asks the question nobody could answer: "What are we spending on AI, and is any of it safe?" The room goes quiet. Twelve teams have twelve sets of provider keys, twelve half-built prompt-injection filters, and twelve invoices buried inside a general cloud bill. Two of those keys have already leaked. That silence is the business case for a platform — and if you've sat in that room, you already understand why "just let each team build it" stops scaling somewhere around the third team.

The trap here is thinking a platform is a technology decision. It isn't — it's a toil-consolidation decision. The useful mental model is a shared office kitchen versus everyone bringing a hot plate to their desk. One kitchen is only worth building if enough people are cooking; if it's just you and a microwave, a shared kitchen is pure overhead with a cleaning rota. A platform earns its keep the same way: only when several teams are each independently re-solving the same unglamorous problems — key management, guardrails, model choice, spend tracking, retries, logging — does centralizing them save more than it costs to run.

So the discipline of this first section is to enumerate the duplication honestly, because each duplicated concern is a column in the platform's value case and, later, a capability the gateway must actually deliver. Name every place a team is doing something that is duplicated (wasteful), dangerous (a security or compliance risk), or unobservable (nobody can see or audit it). The common mistake is to skip this step and jump straight to architecture — you end up building a platform nobody asked for, solving problems teams didn't feel, and then wondering why adoption is zero. If the list you produce is genuinely short, the honest answer is "not yet — ship a shared library and revisit." If it's the long list below, the toil is already burning real money and risk, and the platform pays for itself the day it lands.

A platform is only worth its own team if it removes more toil than it adds. Start by naming the duplicated, dangerous, or unobservable things every product team is doing today — because each one becomes a capability the platform centralizes. If the list is short, you don't need a platform yet; if it's this long, you can't afford not to have one.

2 · The AI gateway — one contract in front of everything

Here's the decision that quietly determines whether the whole platform works. You've agreed to centralize keys, guardrails, routing, and billing — but where does that centralized code actually live? The tempting answer is a shared Python package every team imports. It feels lighter, and teams already know how to pip install. Six months later you push a critical guardrail fix, and you discover half the teams are still pinned to last quarter's version, one team forked it, and you cannot rotate a leaked key without twelve coordinated redeploys. The thing you built to reduce risk has become a fleet of stale copies you can't update.

The fix is to make the platform a gateway, not a library — and the difference is exactly the difference between a toll bridge and a printed map. A map (the library) is copied into every car; once printed, it's frozen, and different drivers carry different editions. A toll bridge (the gateway) is a single structure every car physically drives through — so the moment you change the toll, add a lane, or close a route, every vehicle experiences the change instantly, because they all pass through the same point. Precisely defined: a gateway is a runtime boundary — a live service in the request path — whereas a library is build-time code baked into each caller. Only the runtime boundary can enforce something on every call the instant you change it.

That single architectural choice buys you the platform's whole reason to exist: guardrails that update everywhere at once, keys that rotate without touching a single team, budgets that can actually block a call rather than just chart it, and one uniform log for every request the org makes. The caller, meanwhile, gets radically simpler — it sees a clean endpoint and never touches a provider key, a model name, or a retry loop. The common mistake is to blur the two: teams ship "a bit of shared logic" as a library "for now," and enforcement quietly leaks back out to the callers where it can never be guaranteed. Keep the split sharp — a thin client SDK is fine for ergonomics (nicer function calls, types), but every rule you must guarantee lives server-side in the gateway.

The heart of the platform is an internal AI gateway: a single service every team calls instead of a provider directly. It owns the cross-cutting concerns so no team has to. Read the diagram as one request's life: a team app authenticates to the gateway, which applies guardrails, checks the team's budget, routes to the right model (with caching and fallback), logs everything, and returns a clean response. The caller never sees keys, never sees which model answered, and never has to build any of it.

3 · Model routing — the right model, hidden from the caller

Picture the day a better, cheaper model ships — the exact good news that should be a one-line win. You go to adopt it and discover a dozen product teams have each hard-coded the old model's name deep in their code. Adopting the new one is now a cross-team migration: twelve tickets, twelve redeploys, twelve regression cycles, and a scheduling nightmare. So you don't do it. You stay frozen on last year's model, overpaying, because the upgrade is too expensive to coordinate. That frozen-forever failure is what this section exists to prevent, and it's caused entirely by letting callers name models directly.

The escape is a layer of indirection, and the everyday analogy is ordering coffee by size. You ask for a "large," not for "480 millilitres in the 90mm cup from the third shelf." "Large" is an intent; the café is free to change cups, suppliers, or recipes behind that word without retraining a single customer. Model routing does the same thing: teams request a capability tier — "cheap-fast," "balanced," or "frontier" — which names what they need, not which model provides it. The gateway holds the mapping from tier to concrete model, so a model name is an implementation detail the platform owns, exactly like the cup size behind "large."

Once that indirection exists, the gateway can make the choice smarter than any single team would bother to. It can pick the cheapest model that clears the tier's quality bar (cost control, decided once for everyone), fail over to an alternate provider when one times out (resilience nobody has to build), and pin regulated data classes to an approved region or provider (compliance enforced in one place, not audited twelve times). And because callers depend only on the intent, you can swap the model, add a fallback, or shift the cost/quality tradeoff for the whole org with a single edit and zero caller changes. The mistake to avoid is leaking the model name back out "just so a team can pin a specific version" — the moment one caller depends on a concrete model, you've re-frozen the migration you worked to free. (Model availability and pricing drift constantly; treat the tier-to-model map as something you re-verify against provider docs, not set once.)

With every request flowing through one place, the gateway can make the model choice for the team — by capability, cost, and availability — and change it centrally without a single caller redeploying. Teams request a capability tier ("cheap-fast", "balanced", "frontier"), not a specific model name, so you can swap models, add providers, or fail over transparently.

4 · Cost governance — spend that's attributable and bounded

Every team that has run LLMs in production has a version of this story. A well-meaning engineer ships an agent with a retry loop that, under one bad input, calls a frontier model in a tight cycle overnight. Nobody notices until the monthly bill lands with an extra five figures nobody can explain — and worse, nobody can even say which team or feature caused it, because it's all one undifferentiated pile of provider spend. The lesson lands hard: for AI, cost isn't a reporting problem you clean up at month-end, it's a real-time control problem, and the place to solve it is the one spot every call already passes through — the gateway.

The key distinction, and the one beginners miss, is between attribution and enforcement. Think of the difference between a bank statement and a debit card with a limit. A statement (attribution) tells you afterward where the money went — useful, but the money is already gone. A card limit (enforcement) declines the transaction at the till before you overspend. A cost dashboard is a statement: it changes no behavior, because no one's ability to spend depends on it. The platform's superpower is that it sits in the request path, so it can be the card limit — it can warn, downgrade, throttle, or block a call while it's happening, not report on it later.

Because the gateway authenticates every call, it can tag each request's tokens and cost to a specific team and feature — turning that undifferentiated pile into a per-team, per-feature line item. From there, four more mechanisms layer on top to make spend both bounded and owned: graduated budgets that act as they approach a cap, per-team quotas so one team's runaway loop can't starve everyone else, shared prompt caching that cuts cost across all teams at once, and chargeback that hands each team its own bill so the cost has an owner. The common mistake is to stop at metering — to build beautiful dashboards and declare victory. Dashboards inform; only in-path enforcement changes what happens. The steps below walk the full layering; read them as "attribute first, then act."

The reason finance blessed the platform is this section. Because every call is authenticated and metered, the gateway turns AI spend from an untraceable lump into a per-team, per-feature line item — and, crucially, it can enforce budgets, not just report them. Four mechanisms, layered, give you FinOps for AI: metering, budgets, quotas/rate-limits, and shared caching.

How the gateway governs cost

1. Meter every call — attribute input/output tokens and cost to a team and feature via the auth identity.
2. Per-team budgets — a monthly cap; approaching it alerts the team, exceeding it soft-fails to a cheaper tier or blocks non-critical calls.
3. Quotas + rate limits — per-team RPS/concurrency so one team's runaway loop can't starve everyone.
4. Shared prompt caching — common system prompts/context cached once at the gateway, cutting cost across all teams (verify cache pricing/TTL in provider docs).
5. Chargeback / showback — a monthly per-team report so cost is owned by the team that incurs it.

5 · Self-serve & golden paths — adoption by attraction

This is where most internal platforms quietly die. The gateway works, routing is elegant, cost governance is airtight — and adoption is near zero, because onboarding a team means filing a ticket and waiting two weeks for the platform team to provision access by hand. Meanwhile a product engineer under deadline pressure discovers they can paste a provider key into an environment variable and ship the same afternoon. They will do that every single time. You didn't lose on capability; you lost on friction. A platform competes with the direct provider call, and the direct call is often astonishingly easy — so if your paved road is slower to get onto than going off-road, teams go off-road and you've built expensive shelfware.

The reframe that fixes this is to stop thinking of the platform as a gate and start thinking of it as a product competing for users — and products win by being the easiest and best option, not by being mandatory. Two ideas do the heavy lifting. First, self-serve: a team registers its app in a portal and instantly receives a scoped identity and tier access — no ticket, no human in the loop, no wait. Second, the golden path: a paved, documented, best-practice route for the common cases (a RAG template, an agent tool-loop) where grounding, citations, guardrails, retries, and observability are already wired in and correct. The team supplies only the part that's actually theirs — their corpus, their tools — and ships.

The crucial nuance is that a golden path is not a mandate. Mandates ("you must use the platform") breed resentment and creative workarounds; the moment the paved road is inconvenient, a mandate just teaches teams to hide their shadow IT. A golden path wins by attraction: it's genuinely the fastest, safest way to ship the common case, so teams take it because they'd be foolish not to — guardrails and dashboards come free, and the docs actually work. The escape hatch matters too: a team with a truly exotic need can go off-path, accepting more responsibility for what they then have to build themselves. Most won't want to. The common mistake is investing everything in enforcement and nothing in ergonomics — you can compel a login, but you cannot compel enthusiasm, and durable adoption is built on the latter.

A platform that requires a ticket and a two-week wait to onboard a team will lose to teams calling the provider directly. The goal is self-serve: a team registers, gets a scoped identity, and ships against a golden path — a paved, documented, best-practice route for the common cases — in an afternoon. Golden paths aren't mandates; they're the easiest path, so teams take them by choice.

6 · Platform-team topology & the internal-product mindset

A platform with no clear owner rots in a predictable way. It starts as a heroic side project by two engineers; they get pulled onto feature work; the guardrails drift out of date, nobody answers the support questions, and within a year teams have quietly gone back to direct provider calls. The technology was never the problem — the org design was. How you staff and position the owning team decides whether the platform is a living product or an abandoned tool, and that's an engineering-leadership decision as much as an architectural one.

A useful, publicly known vocabulary for this is Team Topologies, which names a few team shapes and how they should interact. The one insight worth internalizing is its central metric: a platform team's job is to minimize the cognitive load it imposes on the teams it serves. Cognitive load is just "how much a product engineer has to hold in their head to ship a feature." Think of the platform as electrical wiring in a building: a good electrical system lets every tenant plug in an appliance without knowing anything about amperage, grounding, or the substation. A bad one makes every tenant an amateur electrician. The platform succeeds precisely when a product team can ship an AI feature while knowing almost nothing about models, keys, guardrails, or billing — not because those concerns vanished, but because the platform absorbed them behind a smaller surface.

In that vocabulary, the owning group is a platform team providing a self-serve internal product to stream-aligned product teams (the dozen teams shipping features), and its default interaction mode is "X-as-a-Service" — you consume it like a utility, you don't collaborate on its internals. Early on, though, a pure self-serve stance fails: the first adopters need hand-holding to get onto the paved road at all. So the platform team temporarily wears a second hat as an enabling team — pairing closely with the first one or two teams to bootstrap adoption, transfer the patterns, and then deliberately withdraw once the golden path proves itself. The common mistake is either extreme: staying in permanent hands-on mode (you become a bottleneck consultancy that can't scale), or going pure self-serve from day one (nobody can find the on-ramp and adoption stalls). Treat it as a product — measure time-to-first-call and adoption, not lines of code — and staff it to lower cognitive load, not to gatekeep.

A platform needs an owning team, and how you structure it decides whether it thrives. Borrowing the publicly known Team Topologies vocabulary: the platform team is a platform team providing a self-serve internal product to stream-aligned product teams, and it should minimize the cognitive load it imposes — the whole point is to reduce what product teams must know. Early on it also acts as an enabling team, pairing with product teams to bootstrap adoption, then stepping back.

7 · Rollout & the failure modes of a platform

The most common way to kill a good platform is to launch it well. Leadership loves the demo, issues a company-wide mandate — "all AI features must use the platform starting Monday" — and turns on a dozen teams at once against a gateway that has never handled real, concurrent, adversarial traffic. The gateway buckles, a team's launch slips because of your latency, trust evaporates in the first week, and the mandate that was supposed to guarantee adoption instead guarantees resentment. A big-bang mandate front-loads all the risk onto the moment you understand the system least. The alternative is to roll out the way you'd launch any product with real users: earn adoption incrementally.

The pattern is lighthouse → widen → steady state. First, land a single eager "lighthouse" team and co-build their feature on the platform (this is exactly where the enabling-team hat from §6 goes on) — you prove the golden path against real traffic while the blast radius is one team. Then widen by opening self-serve onboarding so more teams pull themselves on because the paved road is now visibly the fastest route (§5). Finally you reach steady state, where new AI features simply start on the platform by default — adoption by attraction, not decree. Each stage de-risks the next: you never expose more teams than the platform has earned.

The second half of this section is the honest part: a platform has its own failure modes, and every one traces back to a design choice made earlier. The single most important is the one you created deliberately — by putting one gateway in every team's request path, you made the org's entire AI capability depend on that one service's uptime. That's a single point of failure (SPOF), and it's the price of centralization. It's a price worth paying, but only if you then treat the gateway like the critical infrastructure it now is: high availability across multiple instances, per-provider circuit breakers and fallback so one provider's outage degrades gracefully instead of cascading, and a tested runbook for the day it does break. The mistake is to celebrate the elegance of "one contract in front of everything" and forget that "everything" now shares one fate. The table below pairs each failure mode with the specific earlier decision that defuses it — read it as a self-audit, not a list of scary outcomes.

Roll the platform out the way you'd roll out any internal product: land a lighthouse team, prove the golden path, then widen — never a mandate-first big bang. And a platform has its own failure modes, each mapping to a design choice already made above.

Team -> Gateway:
  { service_identity, feature_id, tier: cheap-fast|balanced|frontier, messages }
Gateway does:  auth -> guardrails -> budget check -> route(tier,cost,availability)
               -> provider call (cached, with fallback) -> log + meter
Gateway -> Team:
  { response, usage: {tokens_in, tokens_out, cost, model_used} }

route(tier, data_class):
  candidates = TIER_MAP[tier]                 # e.g. balanced -> [modelA, modelB]
  candidates = filter_by_policy(candidates, data_class)   # region/provider pins
  for m in order_by_cost(candidates):
     if available(m):  return call_with_fallback(m, rest=candidates)
  raise NoModelAvailable                       # circuit-breaker / degrade