Enterprise Agentic RAG Platform
A reference architecture for grounding LLM agents in enterprise data, with retrieval as a data product, an AI gateway and eval gates before release.
The challenge
Getting an LLM to answer from private enterprise data accurately enough to put in front of customers, and being able to prove it still does after every model change.
Most enterprise RAG pilots demo well and then stall, because the hard parts of an agentic platform are not the model. They are keeping the index in step with the source systems, enforcing entitlements at retrieval time rather than in the prompt, and being able to tell whether last week’s change made answers better or worse.
How it works
Retrieval is treated as a data product rather than a library call. It has an owner, a contract, freshness and quality metrics, and a version — which matters because chunking strategy, embedding model and ranking are all breaking changes to everything downstream of them.
Entitlement filtering happens before ranking, not after generation. Filtering after the fact means the model has already seen documents the user cannot, and no amount of prompt instruction reliably prevents that leaking into an answer.
Every model call goes through the AI gateway. That is what makes cost attribution, fallback between providers, and a single audit trail possible without every team implementing their own.
What to get right
- Version the chunking. Treat a chunking change like a schema migration: re-embed, shadow-evaluate, then swap the index.
- Gate releases on evals, not vibes. A golden set of a few hundred real questions catches regressions that spot-checking never will.
- Budget the agent. Steps, tokens and wall-clock time need hard ceilings, or one malformed plan becomes an unbounded bill.
- Log the retrieved context, not just the answer. When an answer is wrong, the question is almost always which documents came back.
How it fits together
Source systems and connectors
CRM, ticketing, wikis, contracts and object storage, pulled on a schedule with change-data-capture where the source supports it rather than re-crawled in full.
Ingestion and enrichment pipeline
Parsing, chunking, metadata extraction and PII classification. The chunking strategy is versioned — changing it invalidates every embedding downstream.
Embedding and vector index
Embeddings generated in batch and kept in step with the source, with the tenant, sensitivity label and source URI stored alongside every vector.
Retrieval service
Hybrid keyword and vector search, filtered by the caller's entitlements before ranking. Retrieval is a data product with its own contract and quality metrics.
AI gateway
One entry point to every model: routing, fallback, per-team rate limits, cost attribution, prompt-injection screening and full request logging.
Agent runtime
Planning, tool calls and memory, executed in a sandbox with an explicit tool allowlist and a hard budget on steps, tokens and wall-clock time.
Evaluation and release gates
Golden datasets and LLM-as-judge scores run in CI. A prompt, model or retrieval change ships only if faithfulness and answer quality hold.
Observability and feedback
Traces spanning retrieval and generation, token and latency budgets per route, and captured user feedback fed back into the eval set.
Typically built with
- LangGraph
- pgvector
- Apache Airflow
- MLflow
- Langfuse
- Kubernetes
