Standing up a shared AI platform

Meet "Vanta Corp" — a representative composite. A year into the AI gold rush, Vanta has a dozen product teams each shipping LLM features. Every team has independently wired up its own API keys, its own retry logic, its own half-built guardrails, and its own untracked spend. Nobody can answer "what are we spending on AI and on what?", two teams just leaked a key, and each team re-solves prompt-injection defense from scratch — badly. The mandate: stand up a shared AI platform so teams build features, not plumbing. This is the classic platform-engineering job applied to AI, dominated by three forces: you must remove duplicated toil, you must make the safe path the easy path, and you must make spend attributable and bounded.

1 · Framing the platform — what "shared" has to buy

A platform is only worth its own team if it removes more toil than it adds. Start by naming the duplicated, dangerous, or unobservable things every product team is doing today — because each one becomes a capability the platform centralizes. If the list is short, you don't need a platform yet; if it's this long, you can't afford not to have one.

2 · The AI gateway — one contract in front of everything

The heart of the platform is an internal AI gateway: a single service every team calls instead of a provider directly. It owns the cross-cutting concerns so no team has to. Read the diagram as one request's life: a team app authenticates to the gateway, which applies guardrails, checks the team's budget, routes to the right model (with caching and fallback), logs everything, and returns a clean response. The caller never sees keys, never sees which model answered, and never has to build any of it.

3 · Model routing — the right model, hidden from the caller

With every request flowing through one place, the gateway can make the model choice for the team — by capability, cost, and availability — and change it centrally without a single caller redeploying. Teams request a capability tier ("cheap-fast", "balanced", "frontier"), not a specific model name, so you can swap models, add providers, or fail over transparently.

4 · Cost governance — spend that's attributable and bounded

The reason finance blessed the platform is this section. Because every call is authenticated and metered, the gateway turns AI spend from an untraceable lump into a per-team, per-feature line item — and, crucially, it can enforce budgets, not just report them. Four mechanisms, layered, give you FinOps for AI: metering, budgets, quotas/rate-limits, and shared caching.

How the gateway governs cost

1. Meter every call — attribute input/output tokens and cost to a team and feature via the auth identity.
2. Per-team budgets — a monthly cap; approaching it alerts the team, exceeding it soft-fails to a cheaper tier or blocks non-critical calls.
3. Quotas + rate limits — per-team RPS/concurrency so one team's runaway loop can't starve everyone.
4. Shared prompt caching — common system prompts/context cached once at the gateway, cutting cost across all teams (verify cache pricing/TTL in provider docs).
5. Chargeback / showback — a monthly per-team report so cost is owned by the team that incurs it.

5 · Self-serve & golden paths — adoption by attraction

A platform that requires a ticket and a two-week wait to onboard a team will lose to teams calling the provider directly. The goal is self-serve: a team registers, gets a scoped identity, and ships against a golden path — a paved, documented, best-practice route for the common cases — in an afternoon. Golden paths aren't mandates; they're the easiest path, so teams take them by choice.

6 · Platform-team topology & the internal-product mindset

A platform needs an owning team, and how you structure it decides whether it thrives. Borrowing the publicly known Team Topologies vocabulary: the platform team is a platform team providing a self-serve internal product to stream-aligned product teams, and it should minimize the cognitive load it imposes — the whole point is to reduce what product teams must know. Early on it also acts as an enabling team, pairing with product teams to bootstrap adoption, then stepping back.

7 · Rollout & the failure modes of a platform

Roll the platform out the way you'd roll out any internal product: land a lighthouse team, prove the golden path, then widen — never a mandate-first big bang. And a platform has its own failure modes, each mapping to a design choice already made above.

Team -> Gateway:
  { service_identity, feature_id, tier: cheap-fast|balanced|frontier, messages }
Gateway does:  auth -> guardrails -> budget check -> route(tier,cost,availability)
               -> provider call (cached, with fallback) -> log + meter
Gateway -> Team:
  { response, usage: {tokens_in, tokens_out, cost, model_used} }

route(tier, data_class):
  candidates = TIER_MAP[tier]                 # e.g. balanced -> [modelA, modelB]
  candidates = filter_by_policy(candidates, data_class)   # region/provider pins
  for m in order_by_cost(candidates):
     if available(m):  return call_with_fallback(m, rest=candidates)
  raise NoModelAvailable                       # circuit-breaker / degrade