What is an enterprise AI platform architecture?
An enterprise AI platform architecture is the shared services and enforced rules through which every AI product in an organisation reaches models, data and business systems. It is not a model, and not a product you buy. It settles model access, grounding, agent rights, quality and identity once, so the second and third product inherit them.
Without one, each product copies the first direct integration, until there are five ways of choosing a model, five ways of handling personal data and no place to change either.
The model is rarely what stalls enterprise AI programmes; access, trust and control are.
The five layers at a glance
| Layer | The question it settles | Enforce it in | How to test it |
|---|---|---|---|
| 1. AI gateway | Which models run, where, with what filtering? | The gateway; EU geographic inference profiles; two SCPs | From a member account, call a model in a non-EU Region, or through a global profile: denied. |
| 2. Retrieval | Where does an answer come from, and who may see the source? | An owned retrieval API; a citation check; Row-Level Security with FORCE |
Ask what no document answers: it declines. Query as two tenants: no shared rows. |
| 3. Agent actions | What may an agent change, and who approved it? | One MCP action server; approval bound to exact parameters; idempotency keys | Change one parameter after approval: refused. Send twice: one execution. |
| 4. Evaluation and observability | Did this change make answers worse? | Golden-set gates in CI; traces tied to prompt and model versions | Swap the model on a branch: the gate names the cases that dropped. |
| 5. Identity, tenancy, compliance | Who may do what, and can it be erased? | Token exchange (RFC 8693); temporary workload credentials | A user without refund rights asks the agent for one: refused. Erase a test user: no personal data left in vectors, traces or logs. |
Layer 1: the AI gateway is the only path to models
Every model call goes through one service. Products ask for a capability such as “summarise”, never a model name, so a model change is a routing change tested in one place. On Amazon Bedrock the gateway insists on EU geographic inference profiles, which, as of October 2026, keep requests inside the geography; global profiles may route to any supported commercial Region (cross-Region inference).
A gateway governs only the traffic that goes through it; the account covers the rest. Service control policies cap every IAM user and role in member accounts, root user included (not the management account). Use two: one that denies every Region outside the EU, and one that turns global profiles off. Do not trust a profile’s prefix either: AWS says to call GetInferenceProfile from each source Region and check every destination (inference profile support).
The gateway applies Bedrock Guardrails’ sensitive information filters to prompts and responses. They evaluate text only: personal data in tool-call arguments and tool results passes untouched.
The price is one more service on every request path, which must stay highly available. Details in the AI gateway guide.
Layer 2: retrieval cites its sources or declines
Every answer cites the retrieved passages it used, or the system declines. Enforce that in code: a check after generation rejects citations that do not match retrieved passages; an answer without one is a refusal. Keep retrieval behind an API the platform owns, with hybrid search and reranking. PostgreSQL with pgvector, embeddings next to the relational data, is a sound default.
Tenant isolation belongs in the database, with Row-Level Security. Table owners bypass it unless the table uses FORCE ROW LEVEL SECURITY, and superusers and BYPASSRLS roles always do, so the retrieval API connects with a plain role: no superuser, no BYPASSRLS, no table ownership.
Test recall too. With approximate indexes such as HNSW, pgvector’s README says filtering happens after the index scan, and one tenant’s vectors in a shared index can affect another’s recall. In my reading, the tenant policy is such a filter whenever the plan uses the shared index: compare production-path results with an unprepared exact search across tenant sizes, and use iterative index scans (pgvector 0.8.0 or later) or list partitioning by tenant if they differ.
Strict citation raises refusals, as intended. More in the RAG guide.
Layer 3: agents act through one controlled action layer
Reading is low risk and writing is not. Every agent action goes through one MCP action server: versioned tool manifests that say whether a tool writes, human approval of the exact parameters on every write, an idempotency key per approval, preconditions checked at execution and an append-only decision log. The agent holds no rights of its own.
The MCP specification says MCP cannot enforce its security principles at the protocol level, and the tools page says there SHOULD be a human able to deny tool calls. Under the 2026-07-28 revision, current in October 2026, a broken stream loses the in-flight request and the client re-issues it with a new ID (changelog): without your own idempotency key, an approved write can run twice.
The MCP write-access checklist turns the layer into six demo questions, and what the 2026-07-28 MCP specification changes for agents that write reads the revision against them.
Cost: approval on every write limits throughput. See the agent actions guide.
Layer 4: evaluation and observability measure every change
A language-model application changes whenever a prompt, a model, a retrieval setting or a tool manifest changes, and none of those looks risky in a diff. So a golden set of real questions, expected sources and expected refusals runs as a CI gate, and a model swap ships through the same pipeline as code.
Traces close the loop: tie each production call to the prompt and model versions behind it, and turn bad answers into new cases. OpenTelemetry’s GenAI semantic conventions, still marked Development, define spans for inference, retrieval and tool execution. They advise against capturing prompts and outputs by default and, for sensitive data in production, recommend storing them outside the trace under separate access controls. A trace that stores prompts can hold personal data that layer 5 must erase.
A passing gate shows known cases still work, not that the system is correct. Golden sets go stale and invite overfitting, so human review of real conversations stays. The method is in the evaluation guide.
Layer 5: identity, tenancy and compliance
Identity ties the layers together: the AI can do what the user can do, never more. Users authenticate once; tenants are isolated in the database, as in layer 2. Workloads use temporary credentials rather than stored keys, for example IAM roles for service accounts on EKS.
The downstream call carries the user’s rights through OAuth 2.0 Token Exchange. The gateway or action server sends the identity provider the grant type urn:ietf:params:oauth:grant-type:token-exchange, the user’s token as subject_token with its subject_token_type, optionally its own as actor_token, and an audience or resource naming the target system, and receives a token meant for that system. Under delegation the caller keeps its own identity and the act claim records who acted for whom; under impersonation it is indistinguishable from the user, so prefer delegation where your identity provider supports it. An MCP server MUST NOT pass through the token it received (authorization security considerations).
Compliance is designed in. The GDPR asks for data minimisation (Article 5) and gives people a right to erasure (Article 17); on a platform, erasure has to reach vectors, traces and audit records, not only source rows. AI Act obligations depend on the use case: Annex III systems are high-risk unless a derogation applies (Article 6), and providers must make sure people know they are talking to an AI system, unless it is obvious (Article 50). Treat these as design considerations, not legal advice.
Limit: a legacy system behind one shared account cannot enforce the user’s rights.
The order to build it in
- The gateway. It is small, removes the most duplicated work and yields usage data from day one. Leave out what no product has asked for yet.
- Retrieval. Most first use cases need grounded answers. Leave out a dedicated vector store until the corpus demands one.
- Evaluation, before the third product ships. By then one model change touches several products. Leave out a large judge-model suite; a few dozen reviewed cases, refusals included, can gate.
- Agent actions, last. Only once identity and the decision log exist, because a write-capable agent without them is the riskiest thing you can deploy. Leave out any tool exempt from approval.
Identity is not a step: users authenticate from day one, and token exchange must work before the first agent writes.
Failure modes to avoid
- A platform nobody asked for. Build the layer a real product needs, then generalise.
- A gateway that becomes a bottleneck. Keep its contract small and stable, make it highly available, give it an owner.
- Policy only in prompts. A rule that lives only in a system prompt is a suggestion, because prompts can be talked around; make it a citation check, a database policy or an account control.
- No evaluation until something breaks. Without a golden set, every improvement is an opinion.
- Agents with their own privileges. An agent should never be able to do what the user cannot.
In practice
On ENSO’s shared, multi-tenant platform I designed these layers together: a central AI gateway on Amazon Bedrock, RAG on Aurora PostgreSQL with pgvector behind an owned retrieval API, agent actions through one MCP action server with human validation, evaluation gates in CI and identity through Keycloak and OAuth2 Token Exchange. The platform powers four product capabilities. The full case is in the ENSO case study, and the services behind it are described on the enterprise LLM platform page and the AI agents governance page.
Trade-offs and limits
A platform pays off with the second and third product, not the first. It adds services to run, an owner and an availability target, and it concentrates risk: when the gateway or the retrieval API is down, every product is.
A single product with a single model may not need one yet. The account-level residency rule and a golden set do not depend on one.
PostgreSQL with pgvector ties retrieval to one engine and its scaling limits; a very large corpus may need a dedicated vector store.
Human validation on every write limits throughput. Relax it per tool, only where the manifest marks the risk as low.
Infrastructure controls bound what a mistaken model can do; they do not make it right.
Common questions
What are the layers of an enterprise AI platform architecture?
Five: an AI gateway that is the only path to models, retrieval that cites its sources or declines, one controlled action layer for agents, evaluation and observability on every change, and identity with tenancy and compliance. Each is enforced in code and infrastructure, not in prompts.
In what order should you build an enterprise AI platform?
Gateway first: it is small and removes the most duplicated work. Retrieval next, since most first use cases need grounded answers. Evaluation before the third product ships. Agent actions last, once identity and an append-only decision log exist: a write-capable agent without them is the riskiest thing to deploy.
Does a single AI product need a platform?
Not necessarily. A platform pays off with the second and third product, and a single product with a single model may not need one yet. Two things do not depend on a platform: the data-residency rule in the cloud account and a golden set in CI.
How are tenants kept apart on a shared AI platform?
In the database, not the application. On PostgreSQL, Row-Level Security filters every query by tenant, vector search included. Superusers and BYPASSRLS roles always bypass it, even under FORCE ROW LEVEL SECURITY; table owners do unless FORCE is set. Connect the application as none of them. Test with one query run as two tenants.
Sources
- Bedrock, cross-Region inference: geographic and global profiles.
- Bedrock, global cross-Region inference: Region-deny and global-off SCPs.
- Bedrock, inference profile support: checking destinations.
- Bedrock Guardrails, sensitive information filters: text-only scope.
- AWS Organizations, SCPs: maximum permissions.
- PostgreSQL, Row Security Policies: who bypasses them.
- pgvector README: filtering, multitenancy.
- MCP specification 2026-07-28: protocol-level limits.
- MCP, Tools: human in the loop.
- MCP, Key Changes: re-issued requests.
- MCP, authorization security: no token passthrough.
- OpenTelemetry, GenAI spans: spans, content capture.
- RFC 8693, Token Exchange: delegation,
act. - EKS, IAM roles for service accounts: workload credentials.
- GDPR: minimisation, erasure.
- AI Act, Article 6 and Article 50: high-risk, transparency.