Security, privacy & procurement
Enterprise AI lives or dies on questions that never appear in a demo: where does the data live, how is PII/PHI protected, can one tenant see another's data, what happens when prompt injection hits at org scale, and what did we actually sign. This lesson covers data residency, PII/PHI & DLP, tenancy isolation, prompt-injection as an organizational risk, and the vendor-security / legal review (DPA, retention). Public regulations are cited as fact; specifics drift — verify against the current text — and this is engineering-leadership framing, not legal advice.
Learning objectives
- Reason about data residency as a legal property of where bytes live and are processed — distinct from encryption.
- Design a PII/PHI + DLP boundary that redacts before egress and prevents leaks into logs and prompts.
- Enforce tenancy isolation so one customer's data and context can never reach another.
- Treat prompt injection as an organizational risk with layered defenses, not a prompt-wording problem.
- Run a vendor security review and read the procurement/legal terms that matter: DPA, retention, training-on-your-data, sub-processors.
1 · Data residency — where the bytes live
Data residency is a legal/contractual requirement that data be stored and often processed within a specific jurisdiction. Engineers routinely conflate it with encryption — they are different: encrypted data sent to another region still left the region. Residency constraints can veto an entire deployment option (as in the clinical case study), which is why they belong at the top of the design, not the bottom.
| Concept | What it controls | Common misconception |
|---|---|---|
| Residency | Where data is stored/processed | "It's encrypted, so region doesn't matter" — false |
| Sovereignty | Whose law governs the data | Assuming your home law follows the data abroad |
| Encryption | Who can read it | Confuses confidentiality with location |
| Transfer mechanism | Whether a cross-border move is lawful | Assuming any transfer is fine if encrypted |
2 · PII/PHI and the DLP boundary
Personal data (PII) and health data (PHI) carry duties under regimes like GDPR, the DPDP Act, and PHI-specific law. The engineering pattern is a data-loss-prevention (DLP) boundary: sensitive data is detected and redacted before it egresses your trust boundary — before a model call, before a log write, before a third-party system. The order is the design; a leak into logs is as serious as a leak to a model.
3 · Tenancy isolation — one customer never sees another
In multi-tenant AI systems the highest-severity failure is cross-tenant leakage: tenant A's data surfacing in tenant B's answer. It hides in AI-specific places engineers don't check: a shared vector index that isn't filtered by tenant, a cache keyed without the tenant, context bleed when requests are batched, or a fine-tune that memorized one tenant's data and serves it to all.
| Isolation surface | Leak risk | Control |
|---|---|---|
| Vector store / retrieval | Cross-tenant chunks retrieved | Filter every query by tenant id; test it |
| Cache | One tenant's answer served to another | Include tenant id in the cache key |
| Prompt / context | Batching bleeds context across tenants | Never mix tenants in one context window |
| Fine-tune / memory | Model memorizes tenant A, serves tenant B | Don't co-train tenant data without isolation; scope agent memory per tenant |
| Logs / traces | One tenant's data visible in shared logs | Tag + access-control by tenant |
4 · Prompt injection at organizational scale
Prompt injection — untrusted content hijacking the model's instructions — is covered as a security topic elsewhere; here it's an organizational risk. At org scale the blast radius is bigger: an agent with tools can be steered to exfiltrate data, take unauthorized actions, or poison downstream systems, and the attack can arrive through any content the agent reads (a document, an email, a web page, a retrieved chunk). You cannot prompt-word your way out; you contain it with layers and least privilege.
5 · Vendor security review & the contract that matters
Before an AI vendor touches your data, someone must review its security posture and someone must read the contract. These are the questions and clauses that actually decide risk — the ones that belong in procurement, not in an engineer's head.
| Area | The question to ask | Why it matters |
|---|---|---|
| Data retention | How long is our data kept, and can we get zero-retention? | Retained prompts are a breach surface and a compliance issue |
| Training on our data | Is our data used to train their models? Can we opt out? | Your confidential data leaking into a shared model is a real risk |
| Sub-processors | Who else touches our data downstream? | Their vendors become your vendors |
| Certifications | SOC 2 / ISO 27001 / relevant attestations? | Third-party evidence of controls, not just claims |
| DPA | Is there a Data Processing Agreement, and does it fit our regime? | The legal instrument governing personal data — often required by GDPR/DPDP |
| Incident / breach terms | Notification SLA and liability? | Determines what you owe regulators and customers when it goes wrong |
✓ Checkpoint — you can move on when you can…
- Explain why data residency is distinct from encryption and how it can veto a deployment.
- Describe a DLP boundary that protects both ingress and egress, including logs.
- Name three tenancy-isolation surfaces and the control for each, and how you'd test isolation.
- Explain why prompt injection is contained by least privilege and layers rather than prompt wording.
- List four contract/security items to secure before a vendor touches your data.
A vendor says: "Your data is encrypted in transit and at rest, so residency isn't a concern." Why is this a non-sequitur, and what do you actually need to confirm?
Show answer
You're reviewing an AI vendor for a product that handles EU customers' personal data. Name the three contract/posture items you'd treat as non-negotiable and why.
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: The first competence is not confusing three things vendors deliberately blur.
Your task: In your own words, distinguish data residency, data sovereignty, and encryption, with a one-line example of each mattering independently.
Requirements:
- Define each precisely
- Give a scenario where residency is violated even though data is encrypted
- Give a scenario where sovereignty is the issue even if residency is fine
- State the verify-against-current-text caveat
💡 Hint: Residency = where; sovereignty = whose law; encryption = who can read.
Show solution
Residency — where data is stored/processed. Sovereignty — whose law governs it. Encryption — who can read it.
Residency violated despite encryption: an EU customer's records are encrypted but processed on servers in another continent — confidentiality is intact, but the location requirement is broken.
Sovereignty issue with fine residency: data is stored in-region, but the provider is subject to a foreign government's disclosure law that could compel access — the bytes are in the right place, but whose law reaches them is the problem.
Caveat: whether a specific region/mechanism satisfies your obligations is a changing legal fact — verify against the current regulation with legal.
Context: A leak into a log is as serious as a leak to a model; the boundary must cover both directions.
Your task: Design the DLP boundary for an AI feature that handles personal data.
Requirements:
- Show detection/redaction before any egress (model, tool, third party)
- Include an egress check on model outputs
- Specify the logging rule (what may and may not be stored)
- State the fail-closed behavior when data can't be confidently cleaned
💡 Hint: Redaction is first; logging is egress; fail closed when unsure.
Show solution
Ingress. On every request, classify and scan for PII/PHI; redact recognized spans to typed placeholders before the text reaches a model, a tool, or any third party. Report what type was removed, never the raw value.
Egress. Run DLP again on the model's output before it's returned or forwarded — the model can echo, infer, or retrieve PII. Block or re-redact on a hit.
Logging (also egress). Audit records store a salted hash of the user plus PII-free metadata (which sources were used, verdicts, tenant tag) — never the raw prompt or answer.
Fail closed. If the text can't be confidently cleaned (detector low-confidence, or a second-pass check still fires), block egress rather than send/store it. A blocked request is a recoverable inconvenience; a leaked identifier is not.
Context: Isolation you assert in a design doc is worthless; isolation you test in CI is a control.
Your task: Design the automated test that proves one tenant can never see another's data across your AI system's surfaces.
Requirements:
- Seed tenant A with a unique canary string in the corpus/memory
- Exercise retrieval, cache, and context as tenant B
- Assert the canary never appears in B's results
- Say which surfaces this must cover and how it runs in CI
💡 Hint: A canary is a unique token you can search for; if it crosses tenants, isolation failed.
Show solution
Setup. Insert a globally-unique canary (e.g. CANARY-A-9f3c…) into tenant A's corpus, agent memory, and any co-located store.
Exercise as tenant B. Run: (1) retrieval queries designed to surface the canary; (2) a request that would hit the cache with a key that omits tenant (to catch cache-key bugs); (3) a batched request alongside tenant A (to catch context bleed).
Assert. The canary must appear in zero of tenant B's retrieved chunks, cached responses, model context, and logs. Any appearance fails the build.
Coverage & CI. Cover vector store, cache, context assembly, and logs — the four leak surfaces. Run on every change to retrieval, caching, or memory code, and on a schedule against staging. Isolation becomes a tested invariant, not a hope.
Context: At org scale, injection's danger is the blast radius; least privilege shrinks it.
Your task: Design the containment for an agent that reads untrusted content (emails, docs, web) and has tools.
Requirements:
- List the layered defenses (input/output rails, action gate, monitoring)
- Apply least privilege to the agent's read scope and tool permissions
- Show what an injected agent can and cannot do under your design
- State your assumption about whether injection can be fully prevented
💡 Hint: Assume injection sometimes succeeds; design so a successful injection is contained.
Show solution
Assumption first: injection cannot be fully prevented by wording — untrusted content is adversarial. So the design goal is containment, not prevention.
Layers. Input rail delimits/flags untrusted content; the model answers from scoped context; output rail screens for exfiltration patterns and DLP; an action gate means the model only proposes actions and deterministic code decides; monitoring logs every tool call.
Least privilege. The agent can read only the specific tenant/scope it's serving (not the whole store), and its tools are restricted to reversible, low-stakes operations; anything irreversible or customer-facing requires human approval (the HITL tiers from EL4).
Injected-agent capability. Under this design a successful injection can produce a bad draft or attempt a tool call — but it cannot read another tenant's data (scope), exfiltrate to an external tool (not permitted), or take an irreversible action (gated). The blast radius is a caught, logged, reversible attempt.
Context: Someone has to turn a sales deck into a documented risk decision before data flows.
Your task: Produce the vendor-security review checklist and the go/no-go recommendation for a representative AI vendor handling personal data.
Requirements:
- List the security-posture questions and the required evidence
- List the contract clauses to secure (DPA, retention, training opt-out, sub-processors, breach SLA)
- State which items are non-negotiable vs negotiable
- Issue a go / conditional-go / no-go with reasons
💡 Hint: Distinguish 'nice to have' from 'we cannot lawfully proceed without this.'
Show solution
Security posture (with evidence, not claims): current SOC 2 / ISO 27001 report; encryption in transit/at rest; access controls and audit logging; region options for residency; pen-test summary; incident history.
Contract clauses: a DPA fitting our regime; retention terms (ideally zero/short with deletion on request); training-on-our-data opt-out; the sub-processor list + change-notice; breach-notification SLA and liability.
Non-negotiable vs negotiable. Non-negotiable for personal data: a valid DPA, a training opt-out (or contractual guarantee they don't train on it), and a workable residency/transfer basis. Negotiable: exact retention window, support tier, price. Certifications strongly preferred but a recent report + remediation plan can be acceptable.
Recommendation: Conditional-go — proceed once the DPA is executed, training opt-out is confirmed in writing, and legal verifies the residency basis against the current regulation. No-go if the vendor won't provide a DPA or refuses a training opt-out for personal data — those are lawful-basis blockers, not preferences. This is engineering diligence, not legal advice; legal owns the final sign-off.
Context: Representative scenario: a customer reports seeing a snippet that looks like another customer's data in an AI answer. It's a potential cross-tenant breach and possibly a reportable privacy incident. You lead the response.
Your task: Run the incident from detection to prevention, respecting legal obligations.
Requirements:
- State the immediate containment step
- Explain how you confirm scope without making it worse
- Identify the likely technical root causes to check
- Address the legal/notification dimension (in general terms)
- Give the durable fix and the regression control
💡 Hint: Contain first, confirm scope carefully, and loop in legal early — notification may be time-bound.
Show solution
Immediate containment. If a cross-tenant leak is plausible, degrade or disable the affected path (e.g. take the retrieval feature offline or force single-tenant-only mode) to stop further exposure while you investigate. Preserve logs.
Confirm scope carefully. Use the tagged, PII-free logs to determine which tenants and requests were affected — without pulling raw data into new places and widening exposure. Reproduce with a canary in staging, not with real customer data.
Likely root causes to check: a retrieval query missing its tenant filter; a cache key that omitted tenant id; context bleed from batching tenants together; or a fine-tune/memory that co-mingled tenant data. The isolation canary test (if it existed) tells you which surface.
Legal / notification. Engage legal/privacy immediately — a confirmed personal-data exposure may carry time-bound notification duties under regimes like GDPR/DPDP. Engineering confirms the facts; legal decides notification. Verify obligations against the current regulation; this isn't legal advice.
Durable fix + regression control. Fix the specific surface, then add the cross-tenant canary test to CI so this class of leak fails the build forever, plus monitoring/alerting on any tenant-id mismatch in retrieval and cache. The incident ends as a permanent test — isolation becomes an enforced invariant, not a lesson learned twice.