SRE Operations Agentic System: 5 Lessons Learned

· by Ankur Kumar · 10 min read · 6 reads
View as Markdown
SRE Operations Agentic System: 5 Lessons Learned

An SRE and operations agentic system is a team of cooperating agents that helps site reliability engineers investigate production issues by querying the systems they already use, such as observability, OpenShift, application logs and the internal knowledge base. As a team, we learned five lessons building one: measure impact early, start with evals, add guardrails for security and cost, engineer context and memory deliberately, and optimize every component separately.

Why Build an SRE and Operations Agentic System?

The goal was simple: make the SRE and operations teams more efficient. These teams run GenAI engineering platforms and GPU management, and when a production issue hits, they have to move across many systems. They check observability dashboards, inspect OpenShift, read application logs and search the internal knowledge base.

That was manageable when the estate was small. It becomes hard as the number of clusters grows and the number of systems grows with it. Every new cluster and every new tool adds another place an engineer must look before they can form a view of what is wrong.

As the solution architect, I framed the business outcome for the team as shorter time to understand an incident, not a clever demo. That framing shaped every decision below, and everything that follows is what we did and learned together.

What Does the Architecture Look Like?

We built a multi agent, agentic system with one orchestrator agent and a set of extensible subagents. Each subagent uses its own skills and is mapped to its own Model Context Protocol (MCP) servers. MCP is an open standard that lets an agent discover and call tools exposed by external systems.

The orchestrator started with four subagents:

  • Infrastructure agent. Connects to the OpenShift MCP servers and similar infrastructure MCP servers. Each cluster has its own MCP server, so this agent collects data from multiple OpenShift MCPs, selected by environment and cluster.
  • Observability agent. Connects to Grafana and other observability systems through MCP. It aggregates what it collects and processes the metrics before formulating the context it hands to the rest of the system.
  • Capacity agent. Pulls GPU capacity and allocation data from the provisioning system and related sources.
  • Knowledge agent. Fetches content from the internal knowledge base through the existing RAG backed MCP server.

Architecture of an SRE operations agentic system: an orchestrator delegating to infrastructure, observability, capacity and knowledge subagents, each mapped to its own MCP servers

The system also handles enterprise specific nuances, such as internal terminology for departments and cluster naming conventions. An engineer can ask about a team or a cluster in the words they use every day, and the agents resolve that to the right environment, cluster and data source.

The design is extensible by intent. Adding a new system means adding a subagent and its MCP server, not rewriting the orchestrator. For related patterns, see emerging multi agent orchestrator system design patterns.

Lesson 1: Measure the Impact at an Early Stage

Do not wait for a finished system before you find out whether it helps. We worked with the production team to build a curated list of real use cases, then ran the agentic system against them from the first working version.

This gives you a steady feedback loop:

  • You see where the agentic system is useful and where it is not.
  • You learn how to tune system prompts, because real incidents expose weak instructions quickly.
  • You learn which MCP servers return useful data and which add noise.
  • You mature the system progressively, with evidence guiding each change.

The use cases should come from the people who carry the pager. Their incidents are the honest test, and their involvement builds trust in the result.

Lesson 2: Take an Evals First Approach

Evaluation (evals) is the practice of scoring an agentic system’s output against a known baseline. We recommend treating it as a first class part of the build, not a final check.

You need a baseline that is right for your context, and you need to measure the system on several dimensions:

  • Correctness of the answer.
  • Quality of the reasoning and the response.
  • Accuracy of the data the system cites.
  • Any other parameter that matters to your operations team.

A golden dataset, a curated set of questions with verified answers, is what makes this practical. It tells you whether a change to a prompt or a tool made things better or worse. It also keeps its value after launch: run the same evals in production and post production environments so you notice drift as systems, clusters and models change.

For the broader view of how agentic behavior should be designed and judged, agentic design patterns on Google Cloud is a useful companion.

Lesson 3: Put Guardrails in Place for Security and Cost

An agentic system that can reach production systems needs boundaries. We think about two kinds.

Security guardrails keep the system doing legitimate work only. The system should not be usable for malicious purposes, and it should not take actions that could affect the health of production systems. In an SRE setting, read access and diagnosis are very different from change access, so be deliberate about which tools can do what.

We ensured controlled access through role based access control (RBAC), which grants permissions according to a person’s role. Permissions and service account based access follow the role of the user making the request, so the agentic system can only reach the clusters, data and tools that user is already entitled to. The agents never act with broader rights than the person asking.

Cost guardrails address a quieter risk. On a complex problem, an agentic system can keep calling tools and burning tokens without converging. The system has to know when to stop. When it cannot make progress within a defined budget, it should hand over to a human with what it has found so far.

That human in the loop step is not a failure mode. It is a designed outcome, and for a hard incident it is often the right one.

Lesson 4: Manage Context and Memory

An agentic system performs better with the right context. The knowledge base, the tools and every subagent each need the appropriate context to do their part, which is why context engineering matters so much. If you want the fundamentals, we covered them in what is context engineering.

Two points from our experience:

  • Formulating the right context is hard, and it is enterprise specific. Your systems, department terminology, cluster naming conventions, runbooks and failure patterns are not the same as anyone else’s. Expect to customize, and plan time for it.
  • Memory needs both horizons. Short term memory carries the state of the current investigation. Long term memory carries what the system has learned across investigations.

Rather than building memory management from scratch, we would use an agent harness or a ready made solution for short term and long term memory. Memory is important, but it is rarely where your differentiation lives.

Lesson 5: Optimize Each Component Separately

This is the lesson we consider most important. An agentic system has many moving parts, and a weak part drags down the whole. Tune each component on its own, including the MCP servers and the APIs behind them.

Three practices helped:

  1. Give the agentic system quality context. It can only reason well over what it is given, so this stays a first order concern.
  2. Expose the right number of tools from each MCP server. More tools do not make a better agentic system. A large tool list makes selection harder and adds noise, so aim for an optimal set that covers the job and no more.
  3. Quality check the data the tools return. Treat this as part of readiness for the agentic system. If you use a DAG (directed acyclic graph) based flow, add a validation step after each tool call, so the data is checked before it is passed on to the agents.

Here is how the lessons map to what you would actually inspect:

Component What to check Why it matters
System prompts Behavior against golden dataset cases Weak instructions show up as wrong answers
MCP servers Number and design of exposed tools Too many tools degrade tool selection
APIs behind tools Latency, completeness, error handling The agentic system inherits every upstream flaw
Tool output Validation before it reaches the agents Bad data produces confident but wrong answers
Knowledge base Freshness and retrieval quality Stale runbooks mislead the agentic system

What Is on Our Roadmap?

The next step is to make the agentic system self evolving. These are directions we are pursuing, not capabilities we have already delivered:

  • Build the knowledge base as it works. Each time the system helps solve a production issue, the findings feed back into the knowledge base or production playbook, so the next incident starts with more than the last one did.
  • Use long term memory for past experience. The system retrieves how similar incidents were diagnosed and resolved, and applies that experience to the current one.
  • Apply semantic learning for richer context. Learning from the meaning of past incidents, not only keywords, should give each subagent a more enriched context to reason over.
  • Connect more systems. We plan to add MCP servers for cloud providers and for database systems such as Snowflake and Databricks, alongside other infrastructure systems. This extends the same pattern: one new subagent, mapped to its own MCP servers.
  • Load skills dynamically. Instead of carrying every skill all the time, the system loads the skills that fit the production problem in front of it. This also keeps context lean, which supports the context and tool count lessons above.
  • Improve context quality with RLCD or a decision model. Reinforcement Learning for Calibrated Decisions (RLCD) is an approach that outputs fast, structured decisions along with statistically calibrated probabilities. We plan to use it, or a similar decision model, for reranking, which reorders retrieved content so the most relevant items reach the model first. The aim is for each subagent to see better context, not just more of it.
  • Apply security guardrails more deeply. We plan to extend the guardrails we have today as the system gains more systems to reach and more autonomy.
  • Add model routing. Model routing sends each step to the model best suited to it, which helps balance quality against cost.

Each of these should go through the same discipline as the first release: measure on real incidents, evaluate against the golden dataset, and keep the guardrails in place as the system learns.

What This Means for Your Own Build

All five lessons help you reach a first version. Getting it right the first time is very hard. In our experience a ready made agentic solution also does not work easily for this problem, because the value sits in your own systems, your own context and your own operating model.

Our recommendation, given your context and the trade offs involved: build an agentic system for SRE and operations around your enterprise needs. Start with a small set of subagents, test on real incidents, measure with evals, set guardrails, and tune each component. Buying parts such as the agent harness for memory is reasonable. Outsourcing the whole design is where we would be cautious.

If you are building something similar, we would be glad to exchange practitioner perspectives on what has worked for you.