One-sentence Summary

Researchers from Google Research, Google DeepMind, and MIT introduce a predictive model of agent system scaling that, across 260 configurations spanning six benchmarks, five architectures, and three LLM families, captures performance variation with coordination, model capability, and task factors, achieving a cross-validated $R^2 = 0.373$R2=0.373 (or $R^2 = 0.413$R2=0.413 with a task-grounded metric) and revealing that architecture-task alignment determines collaborative success, with relative performance changes from +80.8% to −70.0% versus single-agent baselines and the framework selecting the best architecture for 87% of held-out configurations.

Key Contributions

• A predictive scaling model captures coordination, model capability, and task factors, achieves cross-validated R² = 0.373 across six benchmarks (0.413 with a task-grounded capability metric), and correctly selects the best-performing architecture for 87% of held-out configurations.
• The model reveals a capability-saturation effect (coordination yields diminishing returns beyond ~45% single-agent performance), a tool-coordination trade-off, and architecture-dependent error amplification ranging from 4.4× (centralized) to 17.2× (independent).
• On unseen frontier models evaluated on BrowseComp-Plus, the framework shows consistent relative architecture preferences, with a multi-agent system performance prediction MAE of 0.061.

Introduction

The authors tackle the open question of when multi-agent coordination genuinely improves over single-agent systems in LLM-driven tasks that require sustained environmental interaction, partial observability, and adaptive refinement. Prior work largely evaluated on static, non-agentic benchmarks where adding agents showed monotonic improvement, but real-world agentic tasks introduce coordination overhead, error propagation, and fragmented context that can make collaboration counterproductive. Existing evaluations also conflated architecture effects with differences in prompts, tools, and compute budgets, and lacked process-level metrics. The authors contribute a controlled experimental framework that isolates coordination structure across 260 configurations spanning five architectures, three model families, and six agentic benchmarks. They derive a predictive model based on empirical coordination metrics that identifies quantifiable trade-offs, such as a tool-coordination penalty and capability ceiling, and achieves 87% accuracy in selecting the optimal architecture for held-out tasks, offering practitioners a principled replacement for heuristic team design.

Dataset

The authors construct a benchmark suite of agentic tasks, not a static corpus, to evaluate interactive reasoning. The composition and processing are defined by the following principles:

• Task definition and filtering A task is included only if it satisfies three necessary properties: sequential interdependence (later actions depend on earlier observations), partial observability (critical information must be acquired through tool use or active querying), and adaptive strategy formation (the policy must update beliefs from interaction). Benchmarks that lack these conditions, such as GSM8K or MMLU, are excluded; tasks that currently require multi-step interaction, like SWE-Bench, are retained. The definition is relative to current model capabilities: a task is agentic when the optimal interactive policy substantially outperforms any single-pass function.

• Sources and selection The suite extends the framework of Zhu et al. and builds on the Agentic Benchmark Checklist. The authors select benchmarks where base LLMs perform poorly in single-shot mode and non-trivial performance demands multi-step environment feedback. The final set is not enumerated in the text but is implicitly composed of tasks that enforce an action–observation loop length greater than three.

• Controlled interface and normalization All benchmarks share identical tool APIs and observation structures to avoid confounds from external feedback quality. Scores are normalized to the best single-agent baseline, so the reported metric measures coordination gain or loss relative to that baseline. The evaluation emphasizes adaptive reasoning over memorized facts, with cross-model comparisons accounting for inherent knowledge differences through baseline normalization.

• Usage in the paper The benchmark suite is used to compare agent architectures. Each model family is evaluated on the same tasks, and the resulting scores isolate architectural effects on agentic capability. The design ensures that any observed advantage comes from the model’s ability to gather information, plan, and revise hypotheses through interaction, rather than from static knowledge or tool-quality differences.

Method

The authors formalize an agent system as a tuple $S = (A, E, C, \Omega)$S=(A,E,C,Ω), where $A$A is a set of agents, $E$E is a shared environment, $C$C defines the communication topology, and $\Omega$Ω is an orchestration policy. When $|A| = 1$∣A∣=1, the system is a Single-Agent System (SAS); when $|A| > 1$∣A∣>1, it becomes a Multi-Agent System (MAS). Each agent $a_i$ai​ perceives, reasons, and acts within the environment through iterative feedback loops.

Every agent $a_i$ai​ is itself defined by a tuple $S_i = (\Phi_i, \mathcal{A}_i, M_i, \pi_i)$Si​=(Φi​,Ai​,Mi​,πi​). The reasoning policy $\Phi_i$Φi​ is typically instantiated by a large language model (LLM). The action space $\mathcal{A}_i$Ai​ consists of tool calls, formally $\mathcal{A}_i = \{ \text{ToolCall}(t, \theta) : t \in \mathcal{T}, \theta \in \Theta_t \}$Ai​={ToolCall(t,θ):t∈T,θ∈Θt​}, where $\mathcal{T}$T is the set of available tools (e.g., web search, code execution) and $\Theta_t$Θt​ denotes valid parameter configurations. The internal memory $M_i$Mi​ stores the agent’s observation history. The decision function $\pi_i : \mathcal{H} \to \mathcal{A}_i$πi​:H→Ai​ maps observation histories to actions, and is realized by the LLM: given a history $h_{i,t}$hi,t​, the model generates a reasoning trace and selects the next action.

At each timestep $t$t, agent $a_i$ai​ selects an action according to:

where $h_{i,0} = \{s_0\}$hi,0​={s0​} contains the initial task specification. The history update function $f_i$fi​ appends the new action-observation pair to the existing history, subject to context-window truncation when the token limit is exceeded. This update mechanism applies uniformly to both SAS and MAS configurations.

Communication between agents is handled through explicit message passing within the orchestration layer. The communication topology $C$C defines the information flow patterns. The authors consider four topologies:

• Independent: agent-to-aggregator only, with no peer communication.
• Centralized: orchestrator-to-agents only.
• Decentralized: all-to-all connectivity.
• Hybrid: a combination of centralized and limited peer-to-peer edges.

The orchestration policy $\Omega$Ω determines how sub-agent outputs are aggregated (e.g., majority voting, weighted synthesis), whether the orchestrator can override sub-agent decisions, whether memory persists across coordination rounds, and the termination conditions based on consensus or quality thresholds.

Based on these primitives, the authors instantiate four concrete MAS architectures to form a structural ablation of coordination mechanisms:

• Independent MAS: $n$n agents operate in parallel, and an aggregator concatenates their outputs without cross-validation or majority voting. This isolates the effect of parallelism without communication.
• Centralized MAS: a single orchestrator coordinates $n$n sub-agents over $r$r rounds, creating a hierarchical structure with a potential bottleneck at the orchestrator.
• Decentralized MAS: agents engage in $d$d sequential debate rounds with all-to-all communication, enabling consensus formation through peer discussion.
• Hybrid MAS: combines an orchestrator with limited peer communication, inheriting hierarchical control while allowing lateral information exchange.

The design distinguishes communication (message passing) from coordination (strategic direction). In centralized systems, the orchestrator handles task decomposition and progress monitoring; in decentralized systems, communication and coordination are intertwined through debate rounds.

To quantify error dynamics, the authors define a task-level error rate $E = 1 - P$E=1−P, where $P$P is the fraction of tasks successfully resolved. The task-level error amplification factor $A_e^{\text{task}} = E_{\text{MAS}} / E_{\text{SAS}}$Aetask​=EMAS​/ESAS​ captures whether multi-agent coordination suppresses or amplifies errors relative to a single-agent baseline. A trace-level amplification factor $A_e^{\text{trace}}$Aetrace​ further measures the extra computational work arising from inter-agent coordination failures, estimated from execution-trace token analysis. These metrics provide complementary views on how different architectures handle errors.

Experiment

The study evaluates multi-agent systems across 260 configurations, 6 benchmarks, and 9 LLMs, revealing that coordination benefits are highly domain-dependent: decomposable tasks like financial analysis gain up to 80.8% from centralized coordination, while sequential planning tasks degrade by up to 70%. A quantitative scaling principle shows that a single-agent baseline above roughly 45% saturates improvement, and that task decomposability, tool complexity, and coordination efficiency explain performance variance. The findings enable architecture selection based on measurable properties, with centralized structures suited for analysis, decentralized for tool-heavy tasks, and single-agent for sequential planning.

The evaluation uses six benchmarks spanning web browsing, finance, Minecraft planning, business tool use, software engineering, and command-line execution. The most robust finding is capability saturation, where a single-agent baseline metric plateaus around 45% success. An Agentic Capability Index is proposed as a more reliable alternative to static intelligence composites. The six benchmarks cover diverse agentic domains: web browsing, finance, Minecraft planning, business tool use, software engineering, and system administration. Capability saturation (single-agent baseline) is the only predictor that remains significant under both cluster-robust (p=0.004) and Holm–Bonferroni corrections. The Agentic Capability Index (ACI) — mean single-agent performance across all benchmarks — correlates only moderately with the static Intelligence Index (r=0.45) and improves cross-validated fit. A capability-saturation threshold of approximately 45% is supported by a 94% match rate across configurations on SWE-bench and Terminal-Bench. Within-domain cross-validation correctly identifies the optimal architecture for 87% of held-out configurations, even though absolute cross-domain prediction is limited.

Single-agent systems operate sequentially with zero communication overhead and minimal memory. Among multi-agent architectures, the independent variant achieves full parallelization while adding only a single synthesis exchange, keeping sequential depth unchanged. Decentralized debate and centralized orchestration introduce extra rounds that multiply LLM calls, communication overhead, and memory usage; the hybrid design further adds peer communication, yielding the highest coordination cost. Independent multi-agent setups preserve the same sequential depth as a single agent while enabling full parallelism and requiring only one communication exchange for vote synthesis. Decentralized debate and centralized orchestration scale LLM calls and inter-agent messages linearly with the number of debate or orchestrator rounds, increasing resource consumption compared to the independent approach. The hybrid architecture incurs the greatest communication overhead by combining orchestrator-driven rounds with additional peer-to-peer message exchanges.

Incrementally adding empirical coordination metrics and their interactions with task properties improves cross-validated prediction of agentic system performance. The full model, which combines base capability, task features, and coordination measures with nine interaction terms, achieves the best cross-validated fit and lowest AIC, indicating these metrics capture variance beyond simpler predictors. Capability saturation, where higher single-agent performance leaves less room for coordination gains, emerges as the most statistically robust finding across correction methods. The full model with interaction terms achieves the highest cross-validated R² (0.373) and lowest AIC (−236.3), outperforming specifications that only add coordination structure or the single-agent baseline. Replacing the static Intelligence Index with the Agentic Capability Index (ACI), which measures mean single-agent performance across benchmarks, further lifts cross-validated R² to 0.413 without reversing any findings.

A regression model across 260 configurations reveals that agentic system performance is driven most robustly by tool diversity, the single-agent baseline, and the interaction between coordination efficiency and tool complexity. Intelligence shows a linear effect without significant curvature, and agent count alone is not a reliable predictor. The model fit improves when measuring capability with a task-grounded Agentic Capability Index rather than a static composite. Log-transformed tool count is a strong positive predictor that survives correction for multiple comparisons. Higher single-agent baseline performance strongly predicts overall system performance, capturing the diminishing room for coordination gains. The interaction between efficiency and tool complexity is significant, indicating that efficiency penalties are amplified in tool-rich environments. Intelligence index has a linear effect but no significant quadratic term, suggesting no non-linear scaling of capability. Agent count alone is not significant, and its effect is better understood through interactions with baseline and task properties. Replacing the static Intelligence Index with the Agentic Capability Index raises cross-validated R² from 0.373 to 0.413, confirming that dynamic performance metrics align better with agentic outcomes.

Coordination architectures show a clear trade-off: single-agent systems are efficient and low-error, while multi-agent setups can improve success rate but at steep coordination costs. Efficiency drops monotonically from 0.466 for single agents to 0.074 for hybrid architectures, and independent agents exhibit the highest trace-level error amplification, which hurts their success rate. Overhead and turn counts escalate with coordination, with hybrid systems incurring the largest overhead and the most turns. Independent agents suffer from extreme error amplification (17.2), over 17 times higher than single-agent systems, leading to a success rate of just 0.370. Efficiency declines sharply as coordination complexity grows: single-agent systems achieve 0.466, whereas hybrid architectures drop to 0.074, a sixfold reduction.

The evaluation spans six diverse agentic benchmarks and finds that a single-agent baseline plateaus around 45% success, establishing a robust capability saturation effect that limits coordination gains. Multi-agent architectures can improve success but involve steep efficiency trade-offs, with coordination overhead and error amplification escalating sharply as complexity increases. A dynamic Agentic Capability Index outperforms static intelligence metrics as a predictor, and the interplay of tool diversity, baseline capability, and coordination efficiency further explains system performance.