Choosing an AI agent architecture is an operating decision, not just a framework decision. This guide compares single-agent, workflow-based, and multi-agent designs, then gives you a repeatable way to estimate reliability, latency, model usage, tool complexity, human approval needs, and cost before you build or change an AI automation system.
Overview
Most AI automation projects begin with a tempting question: which agent framework or model should we use? A more useful starting point is to define the work the system must perform and the level of control it requires. Architecture determines how many model decisions occur, how tools are called, where state is stored, how failures are recovered, and when a person must approve an action.
There are three practical patterns to compare:
- Single-agent design: one model receives a task, selects from available tools, and continues until it reaches an answer or stopping condition. This is suitable for bounded tasks with a manageable tool set.
- Workflow-based design: application code controls a defined sequence, such as classify, retrieve, draft, validate, and request approval. The model handles selected steps, while the workflow controls execution.
- Multi-agent design: several specialized agents coordinate through a supervisor, shared state, or explicit handoffs. This can separate responsibilities, but it adds communication paths and failure modes.
The simplest architecture that meets the requirement is usually the best starting point. A fixed workflow is easier to test than an open-ended agent loop. A single agent may be appropriate when tool selection is the main source of flexibility. Multi-agent orchestration deserves a clear justification, such as genuinely different responsibilities, access boundaries, or review stages.
For a deeper design checklist, see AI Agent Architecture Patterns: Choosing Tools, Memory, Workflows, and Guardrails. Architecture should also be evaluated alongside deployment and observability, rather than treated as an isolated prompt engineering choice.
How to estimate
Estimate an agent system in four dimensions: execution volume, model work, operational overhead, and business risk. You do not need exact numbers at the beginning. You need consistent inputs that can be replaced as your measurements improve.
1. Estimate execution volume
Start with the number of tasks completed during a period:
Tasks per period = users or events × tasks per user or event × completion rate
Separate successful, failed, retried, and human-escalated tasks. A system that appears to handle 1,000 tasks may actually perform 1,300 model runs if retries and review loops are included.
2. Estimate model work
For each task, record the expected number of model calls and the approximate input and output tokens per call. A useful first approximation is:
Model calls per period = tasks × calls per task × retry multiplier
Keep model selection flexible in the spreadsheet or configuration. The best LLM for developers is not necessarily the most capable model for every step. A smaller or faster model may handle classification, extraction, routing, or validation, while a more capable model is reserved for ambiguous reasoning.
3. Estimate latency
For sequential execution, approximate latency by adding the duration of each step:
Total latency ≈ model time + tool time + retrieval time + queue time + approval time
Parallel work can reduce wall-clock time, but only when the tasks are independent and the system can safely merge their results. Account for the slowest parallel branch, not the average branch, when setting user-facing expectations.
4. Estimate operational exposure
Count the tools, permissions, external systems, memory stores, and approval points involved in a task. More components generally create more places to observe, secure, test, and recover. This does not make a larger architecture wrong; it means the additional complexity should solve a visible problem.
Use an evaluation set before choosing a design. Include normal requests, ambiguous requests, malformed inputs, tool failures, permission errors, and cases that must be escalated. The Prompt Testing Framework provides a useful model for versioning and evaluating the instructions that govern these cases.
Inputs and assumptions
Create a small comparison table for each candidate architecture. Record assumptions explicitly so the estimate can be recalculated when traffic, model configuration, or workflow requirements change.
- Task volume: expected tasks per day or month, including seasonal variation where relevant.
- Calls per task: planning, tool selection, retrieval, drafting, validation, and final response calls.
- Retry rate: the proportion of calls or tasks that repeat after a timeout, invalid tool input, or failed validation.
- Token profile: average and high-percentile input and output tokens for each model step.
- Tool profile: tool count, average tool duration, failure rate, idempotency, and permission scope.
- Memory profile: short-term conversation state, durable user preferences, and retrieval-based context. Do not assume that storing more history improves results; test whether the added context is relevant.
- Approval profile: actions that require a person, expected approval delay, and the behavior when approval is rejected or unavailable.
- Quality target: the minimum acceptable success rate for each task type, with separate thresholds for informational answers and consequential actions.
- Cost inputs: provider pricing or internal infrastructure rates entered as variables, not hard-coded claims. Include storage, vector retrieval, logging, monitoring, and human review where applicable.
A simple variable-based cost model is:
Total operating cost = model cost + tool and infrastructure cost + storage cost + observability cost + human review cost
If model pricing changes, update the model-cost cells rather than rewriting the architecture analysis. If you are using retrieval, isolate embedding, indexing, retrieval, and generation inputs. The Production-Ready RAG Tutorial can help you separate those concerns.
For memory, start with the minimum state required to complete the task. Short-term state is useful for the current run; durable memory should be deliberately written, scoped, and removable; retrieval memory should have clear relevance and freshness rules. More memory can increase context size, latency, and the chance of irrelevant instructions being passed to the model.
Worked examples
Example 1: Support-ticket triage
Suppose a system receives support tickets, identifies a category, retrieves relevant internal material, drafts a response, and sends only low-risk replies automatically. A workflow-based design is a strong baseline:
- Classify the ticket.
- Retrieve approved context.
- Draft a response.
- Run validation for missing citations, unsupported claims, or prohibited actions.
- Send automatically or route to a reviewer.
This design makes each stage measurable. If classification is reliable but drafting is weak, you can improve one step without giving an autonomous agent broader permissions. A single agent could work, but its tool choices and stopping behavior would require careful testing. A multi-agent design is probably unnecessary unless separate teams, data boundaries, or review roles justify it.
Example 2: Internal research assistant
An internal research assistant may search several repositories, compare documents, and produce a cited summary. A single agent with retrieval tools can be appropriate when the tool set is small and the task is exploratory. A workflow becomes preferable when the output must always include required fields, source references, and a confidence or review status.
Estimate the system by measuring retrieval calls, documents passed into context, generation calls, and the rate of insufficient or conflicting evidence. Do not judge it only by response fluency. Test whether answers are grounded and whether the system declines to make unsupported conclusions. The Production RAG Evaluation Checklist offers relevant evaluation categories.
Example 3: Operations automation
Consider an assistant that changes records, opens tickets, or triggers deployments. Here, permissions and reversibility matter more than conversational flexibility. Use a deterministic workflow for known operations, require structured tool arguments, validate them in application code, and place human approval before irreversible or high-impact actions.
A multi-agent design might separate planning from execution, but that separation does not replace authorization checks. Every agent should receive only the tools and permissions it needs. Log the proposed action, validated arguments, approval decision, execution result, and rollback status.
When to recalculate
Revisit the estimate whenever an input changes materially. At minimum, recalculate after a model change, pricing change, traffic increase, new tool integration, memory redesign, or change in approval policy. Also recalculate after a meaningful shift in task mix: a larger share of complex requests can increase calls, tokens, latency, and escalation even when total volume is stable.
Use production observations rather than intuition. Track task completion, retry rate, tool errors, time to completion, token usage, escalation rate, approval delay, and the quality of final outcomes. Break these metrics down by workflow, model, tenant, and task type where appropriate. Aggregate averages can hide a failing path.
Keep a lightweight architecture record with the current assumptions, evaluation-set results, model configuration, tool permissions, memory policy, and known failure modes. When results decline, change one variable at a time where possible. You may discover that a prompt revision, narrower tool schema, better retrieval filter, or deterministic validation solves the problem without adding another agent.
The practical decision rule is simple: begin with a bounded workflow or single agent, measure its actual behavior, and introduce multi-agent coordination only when a specific requirement demands it. Recalculate the model and infrastructure estimate when pricing or traffic changes, and recalculate the risk estimate whenever permissions, tools, or automation scope expands. That discipline keeps AI app development focused on reliable outcomes rather than architectural novelty.
