One-sentence Summary

Researchers at Carnegie Mellon University propose a framework that leverages a meta agent to automatically discover harness adaptations—optimizing instructions, tools, and orchestration—which lift task difficulty from the model into the harness, enabling small language models to recover up to 89.7% of frontier LLM performance on routine business tasks at only 4% of the inference cost.

Key Contributions

• A framework mapping agent failure modes to harness adaptation strategies, linking capability gaps to tailored instructions, tools, or orchestration loops.
• A harness optimizer that automatically discovers effective adaptations from failure trajectories, using a meta-agent to diagnose SLM behaviors without relying on pre-assumed limitations.
• Experiments on seven business-oriented tasks and three SLM families demonstrate optimized harnesses significantly improve SLM performance, with seven task-model pairs closing the gap to frontier LLM agents and the best SLM agent recovering 89.7% of LLM performance at 4% of the inference cost; effectiveness is strongest for repetitive workflows and SLMs with sufficient base capabilities.

Introduction

Large language model (LLM) agents are increasingly deployed in business workflows, but their high inference cost, latency, and privacy concerns make them expensive to operate at scale. Small language models (SLMs) provide a cheaper alternative, yet they typically struggle with the planning, instruction-following, and tool-use demands of general agentic tasks. Prior work on adapting agents has either relied on handcrafted heuristics that do not generalize well across models or focused on enhancing already powerful LLMs, leaving SLMs underexplored for cost-sensitive automation. The authors introduce a systematic approach that automatically diagnoses SLM failures using a capability-based taxonomy and then optimizes the agent’s harness (instructions, tools, and orchestration) through a meta-agent. Their empirical study across seven routine business tasks demonstrates that harness adaptation allows SLMs to recover up to 89% of an LLM agent’s performance while cutting costs by over 90%, making specialized SLM agents practical for repetitive business processes.

Method

Harness Adaptation Strategies

The authors address agent failures by offloading task demands from the model to the harness. When model capabilities fall short of task demands, this mismatch manifests as failures in tool-use, instruction-following, knowledge, long-context handling, or planning. To bridge this gap, the authors group harness adaptation strategies into three categories: context adaptations, tool adaptations, and agent-loop adaptations.

Context adaptations shape the information presented to the model, such as adding system prompts or managing long contexts through pruning and summarization. Tool adaptations reshape the action space by creating custom tools or filtering irrelevant ones to simplify selection. Agent-loop adaptations modify the control logic, such as instrumenting code with deterministic checks or orchestrating multi-agent systems.

Automating SLM Harness Adaptation

While frontier LLMs often perform well with general-purpose harnesses, Small Language Models (SLMs) frequently exhibit capability gaps that require specialized adaptations. Naively swapping an SLM into a harness designed for an LLM often results in poor performance. The authors demonstrate that an adapted harness can provide detailed plans, reduced tool sets, and monitoring hooks to make an SLM on par with an LLM at a fraction of the cost.

Finding the right adaptation manually is challenging due to the large design space and intertwined failure modes. The authors argue that harness design should be automated as a search problem. They create a meta-agent powered harness optimizer that takes an SLM, a target task, and an initial harness as input to output an optimized harness.

Search Space and Optimization Loop

The search space is instantiated with concrete editable components from an open-source framework, including contexts, tools, hooks, context management, and sub-agents. This structured space allows the optimizer to cover relevant failures reliably.

The optimization loop operates in three phases: sampling and evaluating, diagnosing failures, and validating new harnesses.

1. Sample and Evaluate: The system maintains a pool of previously tried harnesses. Using a genetic search procedure, it samples a harness from the Pareto front and evaluates it on a batch of training instances, logging full trajectories and scores.
2. Diagnose and Propose: A meta-agent inspects the trajectories and the current harness code to identify failures. It receives context from task trajectories, the current harness, search memory of past proposals, and design-space documentation. The meta-agent then edits the harness components, such as adding a tool or inserting a skill. A sanity check is run to catch invalid code before proceeding.
3. Validate and Save: The new harness is evaluated on the same training batch. If it shows improvement, it undergoes full validation and is added to the pool if it outperforms prior candidates.

Example of an Optimized Harness

In a budget-approval task, the default harness exposed a broad set of tools and required end-to-end planning, which worked for a frontier LLM but failed for an SLM. The harness optimizer identified necessary changes over several iterations. It narrowed the action space, rewrote the system prompt into an explicit step-by-step procedure, and introduced a custom hook to prevent looping.

By externalizing fragile workflow parts into prompts, filters, and runtime checks, the SLM recovered the frontier model performance significantly more cheaply.

Experiment

This study evaluates an automated harness optimizer that adapts agent configurations—such as system prompts and tools—for small language models (SLMs) across seven routine business tasks, using a held-out test set to compare optimized SLM agents against a frontier LLM with a generic harness. The experiments reveal that optimized harnesses can bring SLMs to near-frontier performance at dramatically lower cost, especially for repetitive tasks and more capable models. The most effective adaptations address instruction-following and knowledge failures by adding context and managing tools, and the optimizer’s success depends more on faithful failure analysis and exploration quality than on the number of search iterations.

The benchmark consists of seven diverse business tasks, each derived from existing evaluation suites and designed to produce verifiable outcomes. The tasks cover domains such as auditing, budget approval, stock monitoring, anomaly detection, end-to-end testing, website administration, and code refactoring, all with clearly defined workflows and evaluation criteria. Task instances are reused or generated from prior benchmarks, ensuring reproducibility through ground-truth checks, test execution, or AST comparisons. The seven tasks span human resources, finance, operations monitoring, software testing, web management, and software engineering. All tasks are grounded in established benchmarks including TheAgentCompany, LOCA-Bench, WebGenBench, WebArena, and RefactorBench. Evaluation relies on verifiable methods such as direct ground-truth comparison, executing generated tests, or checking code abstract syntax trees. Most tasks contain 100 instances; website management is limited to 50 instances curated from the WebArena CMS subset. Task creation leverages existing pipelines and, for static websites, a frontier language model to generate complete HTML sites.

The large model achieves high average accuracy (89.7%) but at a substantial cost ($1.735/instance). The small model is orders of magnitude cheaper ($1.735/instance).Thesmallmodelisordersofmagnitudecheaper(0.043/instance) and competitive on select tasks, yet its average accuracy falls to 31.4%, failing completely on several complex tasks. On the Attention task, the small model reaches 91.7% accuracy (vs. 96.5% for the large model) with a per-instance cost of only $0.025, a 35x cost reduction. The small model's average cost per instance is about 40x lower than the large model's, but its average accuracy drops from 89.7% to 31.4%, with near-zero scores on Stock, Anomaly, Playwright, and Web tasks.

The benchmark comprises seven business tasks adapted from existing evaluation suites, and experiments compare a large model against a much cheaper small model. The large model attains high accuracy across all tasks but at a high per-instance cost, while the small model reduces cost by roughly 40 times yet achieves only modest accuracy, failing entirely on several complex tasks. Overall, the small model is competitive only on simpler tasks, illustrating a sharp trade-off between cost savings and reliability for these enterprise scenarios.