DevOps Engineer to LLMOps/AIOps Platform Engineer
The first AI feature reaches production and the platform team inherits a set of questions nobody has an answer for. Which model version served that request? Why did last night’s bill triple? What stopped the agent from calling the payments API? How do we roll back a prompt?
None of those are model problems. They are platform problems, and they land on the DevOps engineer — usually after the launch.
This path is the answer to those four questions, in seven stages, roughly nineteen weeks at an hour a day. You are not learning to train models. You are learning to run a dependency that answers differently every time you call it.
Stage 0: what already transfers
Almost all of it. CI/CD, infrastructure as code, Kubernetes, secrets management, SLOs and error budgets, on-call, incident response, network policy, cost allocation — every one of those carries over intact. If you have read understand the DevOps ecosystem to apply trending patterns and run clusters in anger, you have the foundation.
Enterprise assessments treat three of those as pass/fail, and they are worth checking yourself against before starting:
| Area | What is expected |
|---|---|
| CI/CD | Hands-on with at least one — Jenkins, GitLab, GitHub Actions, Harness, or your cloud’s native tooling |
| Infrastructure as code | Hands-on with Terraform, or Pulumi, Crossplane or CloudFormation |
| Kubernetes and Docker | AKS, EKS or GKE in production; CKA is the usual proof |
| Observability | Grafana, Prometheus, Dynatrace, AppDynamics, New Relic or Elastic |
| Service mesh | Istio or Linkerd |
The first three are the gate. The last two are expected but rarely the thing that fails an assessment — and both become more important, not less, once AI traffic is in the picture.
What changes is that one component in the middle is non-deterministic, unmetered by default, and capable of taking actions. Every control below is one you already run, pointed at that.
The roadmap
Stage 1: serving and inference — weeks 1–3
Learn: where inference actually runs — hosted API, managed endpoint, or self-hosted; GPU and accelerator basics, memory as the binding constraint, continuous batching, quantisation trade-offs, KV cache, and cold starts on autoscaled GPU nodes.
Build: serve one open-weights model on your own cluster behind a stable endpoint, and load-test it until you know its throughput curve.
Done when: you can say what your p95 latency is at what concurrency, and what it costs per hour idle.
You may well end up using hosted APIs for everything. Do this anyway — it is the only way the cost and latency numbers stop being magic. Inference optimization in the knowledge base covers the techniques.
Stage 2: the AI gateway — weeks 4–6
Learn: the gateway as the control point — model routing and fallback, per-team rate limits and quotas, provider key management, retries and timeouts on a dependency that can hang, semantic caching, and hard budget enforcement.
Build: put every model call in your organisation behind one gateway, with per-team budgets that actually reject when exceeded.
Done when: switching a team from one provider to another is a config change, and nobody can spend money you did not authorise.
This is the highest-leverage thing a platform engineer can build in year one. Everything downstream — observability, cost attribution, guardrails, provider migration — gets easier once there is one chokepoint.
On what to actually run, there are two families. If you want the control plane your organisation already audits, that is Azure API Management’s AI gateway policies, Apigee in front of Vertex AI, or API Gateway and an ALB in front of Bedrock — no new vendor, and the logs land where your auditors already look. If you want something model-aware without building it, LiteLLM, Portkey and Kong AI Gateway ship provider routing, fallback, budgets and semantic caching out of the box.
A second gateway is appearing beside this one, and it is worth planning for now rather than retrofitting: an MCP gateway, doing the same job for the tool surface. One place where tool servers are registered, scoped and audited, instead of every agent holding its own credentials for every system it touches. Same control point, second protocol — and the tool side is the one with the larger blast radius, because tools write.
Stage 3: agent runtime and sandboxing — weeks 7–9
Learn: why an agent is a different deployment target — it executes code, calls tools and makes network requests decided at runtime. Then: process and container isolation, egress allowlists, tool permission models, credential scoping per agent, and step and spend ceilings.
Build: a sandboxed execution environment for agent tool calls, with default-deny egress and a per-run budget.
Done when: an agent that tries to reach an unapproved host fails closed and you get an alert.
Where the agent actually runs is a design decision, not a deployment detail, and the shape of the workload picks it. Short tool calls fit serverless — AWS Lambda, Azure Functions, Cloud Run functions. Long or stateful runs want a container: Fargate, Azure Container Apps, Cloud Run. Multi-step workflows that must survive a restart want durable orchestration — Step Functions or Durable Functions. Each cloud also now offers a managed agent runtime bundling isolation and session state, of which Bedrock AgentCore Runtime is the furthest along. Self-hosting on Kubernetes stays right whenever the sandbox policy has to be yours rather than the vendor’s, which is more often than the vendor documentation suggests.
This area is moving fast and the knowledge base tracks it closely: agent sandboxing, the Kubernetes Agent Sandbox, kagent for Kubernetes-native orchestration, KAOS, and the Agentic Ops Framework. If you already run Kubernetes, this is familiar ground with unfamiliar names — the 5 lessons from applying Kubernetes in the enterprise still apply.
Stage 4: LLM observability — weeks 10–12
Learn: why a request is now a trace with a span per step, the OpenTelemetry GenAI semantic conventions, token accounting attributed to team and feature, capturing prompt and response payloads without capturing PII, and online evaluation sampled from live traffic.
Build: end-to-end tracing for one agentic application — every model call, tool call and retrieval as a span, with tokens and cost on each.
Done when: you can answer “why was this answer slow and what did it cost” from a single trace.
The tooling splits along a line worth understanding before you buy anything. Your cloud’s platform telemetry — Azure Monitor, CloudWatch with X-Ray, Cloud Trace — already holds the infrastructure spans, and once the OTel GenAI conventions are emitted it will carry the model spans beside them. What it does not give you is prompt and response inspection, eval scores and dataset curation attached to the same trace, which is exactly what the LLM-native tools exist for: Langfuse and LangSmith for tracing, MLflow for the model-lifecycle side, and Galileo, W&B Weave or Braintrust where evaluation and observability are one product. Run one from each family and join them on trace ID. Picking a single winner means either losing the infrastructure context or losing the prompt.
The knowledge base’s observability goals and solutions sections map the tooling. If you have run Prometheus and Grafana alongside microservices or worked with Elastic as an observability platform, the shape is familiar — the new part is that a span carries a cost.
Stage 5: release engineering for prompts — weeks 13–15
Learn: what a deployable unit even is now — model version, prompt version, tool definitions and retrieval config, versioned together; canary and shadow traffic for a non-deterministic component; evaluation suites as deploy gates; and rollback when the gate catches something.
Build: a pipeline where changing a prompt runs the eval suite, ships to 5% of traffic, compares quality against the baseline, and promotes or rolls back automatically.
Done when: a prompt change is as boring and as reversible as a config change.
Testing and evaluations and deployment cover the practice. This stage is what separates an LLMOps platform from a wrapper around an API key.
Stage 6: security, governance and FinOps — weeks 16–19
Learn: NIST AI RMF as the risk framework, Google’s SAIF as the control set, agentic red teaming, prompt injection as a supply-chain problem when agents read untrusted content, audit trails that satisfy a regulator, and cost per resolved request as the unit metric.
Build: a monthly platform report — spend by team and feature, cost per resolved request, eval scores over time, and every policy violation caught.
Done when: the AI platform is boring to operate and legible to finance.
One control here does not carry over from your DevOps practice, because it did not previously exist: agent identity. An agent acting on a user’s behalf is neither a service principal nor that user, and treating it as either breaks the audit trail at exactly the moment somebody needs it. Microsoft Entra Agent ID and Agent 365 are the furthest-developed answer — an agent gets a directory identity, a lifecycle and conditional access, the same way a person does. Beside it sit data-plane controls you already have names for: Azure Purview for classifying and applying DLP to what agents are allowed to read, Bedrock Guardrails for content and topic policy enforced at the model boundary rather than in application code, and VPC Service Controls or its equivalents for the perimeter around all of it. Ask early who the actor is in your audit log. If the answer is one shared service account, you do not have an audit trail — you have a log file.
GenOps — the evolution of MLOps for generative AI — and AI governance are the reference for how this is being framed across the industry.
The MLOps question
Assessments almost always list MLOps and LLMOps as preferred rather than required — which is a useful signal about where to spend time. You are not expected to train models. You are expected to know the shape of the machinery, because a platform that serves generative AI eventually inherits some of it:
| Concern | Know one of | Why it reaches you |
|---|---|---|
| MLOps platform | MLflow, Weights & Biases, Dataiku, H2O.ai, DataRobot, or your cloud’s | Where experiments and runs are tracked |
| Model registry and versioning | MLflow registry, cloud equivalents | The thing your deploy pipeline promotes from |
| Fine-tuning | LoRA and adapter approaches, managed tuning services | The alternative to RAG, with a very different cost curve |
| Inference serving | vLLM, Kubeflow, TensorFlow Serving | Stage 1 of this path, from the ML side |
| Model observability | MLflow, LangSmith, Weights & Biases | Drift and quality, adjacent to stage 4 |
Do this last, and only to the depth that lets you have a competent conversation with a data scientist. Fine-tuning in the knowledge base is a good primer on the one decision that most often reaches the platform team — because “should we fine-tune or improve retrieval” is usually answered on cost and operability grounds, which are yours.
The same concerns, one new component
Read that table as a checklist. Every row on the right is a control you already operate; the work is extending it, not replacing it. That is why DevOps engineers tend to become good LLMOps platform engineers faster than data scientists do — the discipline transfers, and the discipline is the hard part.
One stack, three clouds
The six stages above are deliberately vendor-neutral, because the layers outlive the products. But you will be handed a cloud, and it helps to know what that cloud calls each layer:
| Layer | Azure | AWS | Google Cloud |
|---|---|---|---|
| Agent framework | Microsoft Agent Framework | Strands Agents | Agent Development Kit (ADK) |
| Model and agent service | Azure AI Foundry | Bedrock, AgentCore | Vertex AI, Gemini Enterprise |
| Gateway and control plane | API Management AI gateway | API Gateway and ALB | Apigee |
| Agent memory | Foundry agent threads | AgentCore Memory | Vertex AI Memory Bank |
| Execution | Functions, Container Apps | Lambda, Fargate, Step Functions | Cloud Run |
| Data and retrieval | Fabric, OneLake | S3, Redshift, Aurora pgvector | BigQuery, BigLake |
| Graph and knowledge | Fabric IQ | Neptune | BigQuery graph |
| Observability | Azure Monitor | CloudWatch, X-Ray | Cloud Trace |
| Identity and guardrails | Entra Agent ID, Purview | Bedrock Guardrails, IAM | VPC Service Controls |
Read that as a translation table rather than a shopping list. Two things follow from it. The rows are the same nine problems on every cloud, so if you know one column properly you can read the other two — the argument between them is about integration, pricing and how much they lock in, not about architecture. And the rows are the durable part: products in this table have been renamed or repositioned roughly twice a year for three years, while the layer each one occupies has not moved once.
In practice, few enterprises live in a single column. The common shape is a neutral framework (LangGraph or CrewAI), a neutral gateway (LiteLLM or Portkey), memory in Mem0 or Graphiti, tracing in Langfuse, data on Snowflake or Databricks — with the cloud column underneath supplying identity, execution and the model endpoint. That mixture is not a failure to standardise; it is what portability costs, and it is worth designing for deliberately rather than arriving at by accident.
What to have built by the end
| # | Project | Proves you can |
|---|---|---|
| 1 | A self-hosted inference endpoint with a throughput curve | Reason about GPU cost and latency from measurement |
| 2 | An AI gateway with enforced per-team budgets | Give the organisation one control point |
| 3 | A sandboxed agent runtime with default-deny egress | Bound the blast radius of an autonomous component |
| 4 | An eval-gated deploy pipeline with automatic rollback | Make quality a release gate, not a hope |
Four traps
- Treating the model API as just another HTTP dependency. It is unmetered, it can hang, its output is unbounded, and it costs money per call. Every one of those needs a control.
- Observability without cost attribution. Traces that do not carry tokens and spend tell you a system is slow but never that it is uneconomic.
- Sandboxing after the first incident. An agent with broad credentials and open egress is a lateral-movement path. Bound it before it ships, not after.
- No rollback story for prompts. If a prompt lives in application code and ships with a fortnightly release, quality regressions stay in production for a fortnight.
Where to go next
- Agentic AI Knowledge Base — the maintained reference behind most of the links above, and its AgentOps and production best practices sections in particular
- Rise of cloud native with the CNCF ecosystem
- Solution options for choosing a Kubernetes management strategy
- Java/Python Developer to AI Engineer — what runs on the platform you are building
- Data Engineer to AI Data Engineer — the data plane it serves