Emerging Multi-Agent Orchestrator System Design Patterns

Orchestrator-based SKILLs and DAG-driven pattern for enterprise agents

Multi-agent Systems (MAS) have evolved from building prototypes to production-grade applications, particularly in an enterprise context. Every enterprise organization has either started its journey to build Agentic Applications or is operating in production with initial use-cases such as Agentic AI Assistants, Content or Code Generation, Process Automation, etc.

Structuring the Multi-agent Systems as per enterprise-specific functional and non-functional requirements, constraints, and best practices (e.g., adopting Architecture Decision Records, ADRs, related to reliability, security, cost-effectiveness (tokenomics), performance, availability, and more) is both art and science.

While the design patterns are evolving with the continuous nature of innovation and emerging solutions in Agentic AI, there are two widely adopted system design patterns for multi-agent orchestration:

• Pattern A — Orchestrator with Dynamic Skills: assess the task, load the required skill on demand, and process in a loop to achieve the goal.
• Pattern B — Orchestrator with sub-agents with Directed Acyclic Graph (DAG) as workflow structure: plan as per goal, delegate to the required sub-agents wired as a DAG, and run in a loop (e.g., ReAct) to achieve the goal.

What problems are these patterns addressing?

As the business problem & use-cases in consideration, along with enterprise context, drive the architecture and design pattern selection, requirements around flexibility, cost-effectiveness, and dynamism are key factors driving the decision. Regardless of the pattern, an agent needs the right harness (infrastructure and runtime scaffolding), the appropriate context (enough as per the problem at hand), and the right set of tools (to be mapped to Skills or Subagents).

Which one fits depends on how well-defined your workflow already is, and how much governance your organization needs over what an agent is allowed to do at each step.

Pattern A — Orchestrator with Dynamic Skills

Key design considerations:

• An orchestration agent receiving the user task or query assesses & matches the skills (using patterns such as hybrid search with semantic or keyword-based for skill matching), loads & executes the required skill in a loop with guardrails in place for security, cost, latency, etc.
• Each Agent Skill, which follows an open format for extending AI agent capabilities with specialized knowledge and workflows, with an SOP (Standard Operating Procedure) like system instructions, scripts, and resources (MCP tools). Agents can discover and load these dynamically to perform tasks as required to achieve the desired business goals/objectives.
• Advanced patterns are emerging as research, such as Automated Skill Discovery and Skill Curation for self-evolving Agents, indicating the evolving nature of this pattern.

When to apply:

• When you are building internal tooling and prototyping for productivity solutions such as L1/L2 Troubleshooting Agent for Production Operations and the SRE team, where dynamic tools are needed along with evolving scripts.
• When there is a need for early-stage agent platforms to experiment with a growing catalog of Skills in an enterprise to explore Agentic capabilities.
• When you need centralized reasoning and cost-effectiveness with security and other guardrails (e.g., rate limits, model selection, skill security), this pattern can help, considering the orchestrator as a centralized component.

Trade-offs to consider:

• Skills management and skill sprawl: Effective use of the skill library as it expands, with maintainability and reliable skill selection.
• Non-determinism at scale: With the growing list of Skills, Skill selection will become more complicated, and the nature of non-determinism increases.
• Workflow token consumption and effectiveness are driven in a non-deterministic way as the orchestrator decides the path at runtime in a loop, which needs to be constrained by guardrails.

Pattern B — Orchestrator with sub-agents with Directed Acyclic Graph (DAG)

Key design considerations:

• An orchestrator agent with a suite of sub-agents as tools to delegate the tasks as per user request or query.
• The sub-agents can work in parallel or sequentially as per the business use case, and they are wired as a DAG such as: data ingestion -> analysis and reasoning -> context loading -> tools selection -> tools execution -> summarize and output.
• Each sub-agent can have its own set of MCP Tools, and can leverage patterns such as Agentic RAG or LLM Wiki or GraphRAG to build the context for the underlying LLM.
• Evolving patterns such as Unified Agentic Memory Management (an agentic memory framework that enables LLM-based agents to autonomously decide when, what, and how to manage both long-term and short-term memory) need to be considered as per the context.

When to apply:

• When you are building a structured workflow with agent execution order to be defined in various shapes and forms — you still have an optionality to control the dynamism.
• When you have complex business domains and regulated industries (e.g., financial services) where you have reasonable reasoning and explainability to be considered.
• Where workflows are already well understood from a business process perspective.

Trade-offs to consider:

• Considering the bespoke nature, the development and maintenance cost needs to be evaluated. Manageability of sub-agents along with its associated tools, mapping, and a separate surface for build, design, test, and deployment considerations.
• Flexibility for novel scenarios will require reconsideration or change as once a DAG workflow has been built, an injection of a sub-agent or another set of tools needs to be carefully considered.
• Latency, cost-effectiveness, and resiliency need to be designed with system design practices in consideration and require a strong engineering team to support.

Common Sub-patterns that apply to both patterns

While there is an evolving set of design practices and patterns, there is a common catalog of sub-patterns across enterprise multi-agent systems, which needs to be considered:

• GraphRAG for relationships: Incorporating graph-structured data, which organizes information as a network of nodes (entities like people or places), edges (relationships between those entities), and labels (attributes that define the category of a node and edge), as a knowledge graph.
• Context graph for broader reasoning: Building the entities the business already cares about (accounts, renewals, tickets, incidents, policies, approvers, agent runs) connected by decision events and “why” links (e.g., Slack conversations or ad hoc decisions).
• An LLM Wiki for knowledge and procedural memory: A pattern introduced by Andrej Karpathy to build and maintain a persistent, consolidated collection of knowledge (usually in a set of Markdown files instead of Vector Databases with RAG-based architecture).
• Short-term and long-term memory management: Leveraging human-memory analogy to manage agent memory across short-term for conversational memory, and long-term memory for reference (episodic, semantic, and procedural).
• Security guardrails, pre and post: Validate inputs before they reach a model and validate outputs before they reach a downstream system or a user. Post-response guardrails matter as much as pre-prompt ones, particularly for anything an agent is allowed to act on.
• An LLM gateway: A unified control point for routing, rate limiting, logging, and model swaps keeps governance centralized even as the number of agents and skills grows.

What does this mean for your architecture decisions (ADRs)?

These patterns are not mutually exclusive and need to be applied as per the enterprise context and business problems to be addressed. For example — if your enterprise is early in its Agentic AI journey, Pattern A with Dynamic Skills is a good starting point to experiment and evaluate multiple permutations and combinations of skills. But if your enterprise is aligned towards a defined and multi-step business process (e.g., claim processing, loan origination), Pattern B’s DAG structure is worth building the solution.

Additionally, apart from the common set of auxiliary design patterns shared above, many other aspects need to be considered and documented as part of the Architecture Decision Records (ADRs):

Architecture Decision Records (ADRs) for Multi-Agent Systems

Looking forward to engaging if you have built similar systems or are building multi-agent system architecture.

References

• Equipping agents for the real world with Agent Skills — Anthropic
• Agent Skills Specification
• AI’s trillion-dollar opportunity: Context graphs
• Context Graph Definition — IBM
• Automated Skill Discovery for Multi-Agent Systems
• Learning Skill Curation for Self-Evolving Agents
• Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents