Command Palette
Search for a command to run...
Weekly Paper Digest | 8 Papers Focus on New Advances in Agent Harnesses, Covering Coding Agents, Embodied Control, and Image/Video Generation
As AI Agents evolve from simple Q&A to complex task execution, the Agent Harness is gradually becoming a crucial direction in agent research. It is a runtime framework that connects large language models, tools, and execution environments, responsible for organizing task workflows, managing context and memory, coordinating tool calls, and handling errors and result verification during execution. In simple terms, the model determines what an Agent can think, while the Harness largely influences whether these capabilities can be translated into stable and reliable actual actions.
Why is the Harness attracting increasing attention? As model reasoning and generation capabilities continue to improve, the challenges faced by agents are no longer just about generating correct answers, but also about how to sequentially complete multi-step tasks, reduce erroneous operations, adapt to different execution environments, and control risks during autonomous action. The Harness is gradually developing from a foundational framework supporting model operation into key infrastructure that impacts Agent execution effectiveness, application boundaries, and security.
Around these issues, an increasing number of recent studies have begun to explore the design, evaluation, and optimization of Harnesses from various perspectives: some studies analyze the roles of planning, tools, and context management through controlled experiments; others explore more reliable execution mechanisms in terminal operations, robot control, and visual generation; and some focus on security constraints, automated construction, and recursive self-improvement, attempting to enable the Harness not only to run but also to continuously optimize based on task feedback.
In this issue, we have curated 8 studies related to Agent Harnesses, featuring explorations from institutions such as NVIDIA, Google Cloud AI Research, Northwestern University, and the University of Science and Technology of China. These cover directions including framework design, execution control, embodied intelligence, visual generation, security governance, engineering capability evaluation, recursive optimization, and self-evolution. Let's take a look!
Additionally, the "Papers" section on the HyperAI official website will continue to track cutting-edge achievements in the field of artificial intelligence, helping everyone stay up-to-date with relevant research trends ⬇️
View the latest papers:
This Week's Paper Recommendations
1. Harness Design for Coding Agents
Paper Title:
An Empirical Study of Harness Design for Coding Agents
Paper and Detailed Analysis:
This study conducts a systematic empirical analysis of Harness design for coding agents, focusing on how different operational mechanisms impact task performance while keeping the underlying model constant. Researchers conducted controlled experiments centered on three core components: planning mechanisms, action spaces, and context management, attempting to clarify the specific impact of each Harness component on task success rates, execution efficiency, and resource costs, thereby avoiding judging the effectiveness of a design solely based on the final performance of the complete system.
The research team conducted multiple matched experiments on benchmarks such as SWE-Bench Verified and Terminal-Bench 2.1. The results show that context management is particularly important for long-horizon tasks, as properly handling historical information can reduce failures caused by context overflow; the benefits of planning mechanisms vary depending on model capabilities, and the choice of tool interfaces also needs to be adjusted according to model characteristics. The study indicates that more components in the Harness are not necessarily better; instead, it should be designed according to task requirements, model capabilities, and resource budgets, providing an empirical basis for subsequent quantitative Harness optimization research.
2. Mid-Harness
Paper Title:
Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
Paper and Detailed Analysis:
Researchers from institutions including NVIDIA and the Korea Advanced Institute of Science and Technology (KAIST) proposed Mid-Harness, exploring how to introduce action sampling and verification mechanisms between the large language model and the execution framework to improve the execution reliability of terminal agents. Traditional terminal Agents typically execute commands generated by the model directly, but erroneous operations can alter the runtime environment and affect subsequent tasks. Even if the model itself has the capability to generate better commands, it may not necessarily provide the optimal choice on the first attempt. Mid-Harness generates multiple candidate actions before actual execution, then compares and filters them through a verifier to reduce the impact of improper operations on task progress.
The research team conducted experiments focusing on the sampling methods of candidate actions, the selection of verifiers, and different execution strategies, analyzing the actual value of allocating additional computational resources between the model and the Harness. The study shows that increasing the number of candidate actions does not necessarily yield benefits; its effectiveness depends on whether the verifier can identify better operations. Appropriate comparison and filtering mechanisms can improve action quality and enhance the success rate of the entire task trajectory. This work shifts the optimization focus from merely improving model generation capabilities to action selection prior to execution, providing new insights for building more reliable terminal agents.
3. Show-Harness
Paper Title:
Show-Harness: Just a VLM Agent Can Play Robots
Paper and Detailed Interpretation:
Researchers propose Show-Harness, aiming to address how Vision-Language Models (VLMs) can translate visual understanding and language reasoning capabilities into actual robotic manipulation. Traditional robotic systems typically require specialized training of vision-language-action models or rely on independent controllers to execute instructions generated by high-level models. Show-Harness designs a semantic action interface that represents robotic manipulation as discrete action units that the model can understand and reason about; these actions are then converted into actual control commands by interpreters designed for specific robots. Additionally, the research team developed GUMI (GUI Manipulation Interface), which supports collecting robot manipulation demonstrations through a graphical interface, reducing dependence on specialized teleoperation hardware.
The research team conducted evaluations across multiple tasks, robot morphologies, and environments, exploring the framework's performance in zero-shot control and lightweight fine-tuning scenarios. Results indicate that Show-Harness enables state-of-the-art VLMs to perform various robotic manipulation tasks without specialized fine-tuning, and can also adapt smaller open-source models through relatively lightweight fine-tuning. The study also shows that mechanisms such as contextual planning, action history logging, and failure recovery improve execution stability. This work demonstrates that designing appropriate interaction interfaces between models and environments can unlock the embodied capabilities already present in foundational VLMs, driving Harness from a software runtime framework to an action control layer in the physical world.
4. MaLiang-Harness
Paper Title:
MaLiang-Harness: A Programmable Path to Image and Video Generation
Paper and Detailed Interpretation:
Researchers propose MaLiang-Harness, exploring how to extend the Harness design philosophy to the domains of image and video generation. While traditional procedural generation approaches can provide explicit control over composition and animation, a program running successfully does not guarantee that the final visual result meets user expectations. To address this issue, the framework organizes multimodal large language model-driven visual generation into a process of continuous construction, inspection, and revision. By preserving executable programs and task contexts, tracking editing history and rendering evidence, and validating against the current version, the visual generation process becomes more traceable, recoverable, and iterative.
The research team constructed MaLiang-IBench and MaLiang-VBench to evaluate image and video generation capabilities respectively, testing several state-of-the-art multimodal large language models. Results show that GPT-6-Astra achieved a 100% generation success rate on both benchmarks, but image tasks meeting all visual quality thresholds accounted for 96.0%, while video tasks accounted for 76.9%. This result indicates that successful program execution and meeting final visual quality standards remain distinct evaluation dimensions; similar general model capability scores do not imply identical performance in visual generation tasks. This study provides a systematic runtime framework for programmable visual generation and expands the application scope of Harness in generative AI scenarios.
5. EvoSafeHarness
Paper Title:
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
Paper and Detailed Interpretation:
Researchers propose EvoSafeHarness, aiming to address security protection issues for agents facing direct harmful requests and indirect prompt injections. Traditional security constraints are typically pre-designed by experts and applied across different models and task domains. However, behavioral characteristics of different models and risk requirements of different domains vary, meaning fixed rules may overly restrict normal tasks or miss security risks in specific scenarios. EvoSafeHarness transforms the design of security constraints into an optimizable process, jointly tuning natural language security policies and executable code logic. By incorporating behavioral feedback from the target model, domain specifications, and adversarial review, it generates constraint suites better suited to specific deployment environments.
The research team conducted evaluations on multiple agent safety benchmarks. In the relevant test settings of DecodingTrust-Agent, this method reduced the average attack success rate from 45.6% to 10.0%, with a utility loss of approximately 3.3 percentage points; under specific evaluation settings in AgentDojo, the attack success rate dropped to 0.0%, while normal task utility reached 82.8%. These results indicate that customizing safety mechanisms for specific models and domains can improve the balance between security and task utility. However, safety performance still depends on the attack type, task environment, and evaluation scope. This work expands the role of the Harness from task execution to risk control, exploring how agents can adhere to appropriate safety boundaries while completing tasks.
6. Doc 2 Harness
Paper Title:
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Paper and Detailed Analysis:
Researchers proposed Doc2Harness to directly evaluate the engineering capabilities of large language models in building complete Agent Harnesses. Previous studies typically tested agent task performance within pre-designed execution frameworks, but rarely examined whether models could independently construct truly functional execution frameworks based on development documentation and interface specifications. Doc2Harness designs the evaluation in two stages: first, the construction model generates candidate Harnesses based on interface contracts and development documentation; subsequently, the research team runs the framework in an isolated environment, checking whether its actual behavior meets requirements through observable information such as model requests, disk files, process exit codes, and command outputs.
The research team used this benchmark to evaluate 19 frontier models, with each model performing 3 independent constructions for 328 test cases. Results showed that the best-performing model, GPT-6 Astra, achieved a pass rate of 53.05%, while the weakest model achieved 27.44%; even the best-performing model failed to pass nearly half of the test cases. Failure issues were mainly concentrated in features requiring implementation across multiple modules and the division of responsibilities between different modules. The study demonstrates that a model's ability to generate code does not mean it possesses the complete engineering capability to build complex execution infrastructure. Doc2Harness provides a deterministic benchmark for measuring long-horizon Harness engineering capabilities and offers a reference for future improvements in the automatic construction of agent frameworks.
7. Harness²
Paper Title:
Harness²: Recursive Agent Harnessing for an Open World
Paper and Detailed Analysis:
When agents face real-world tasks that are constantly changing and difficult to define in advance, traditional Harnesses relying on manual design and fixed configurations often struggle to adapt in a timely manner. To address this issue, the research proposes Harness², attempting to enable agents not only to complete tasks but also to continuously improve their own execution frameworks during the process. The method constructs two interconnected recursive levels: task-level recursion uses a Propose–Probe–Compose process to propose candidate framework modifications for the current task, conduct comparative tests, and decide whether to adopt them based on actual execution traces and task outputs; framework-level recursion extracts reusable experiences from these modifications to progressively update the approach for subsequent framework optimization. Even in the absence of ground-truth answers that can directly determine task correctness, the system can provide a basis for framework adjustments by comparing execution performance under different configurations.
The research team conducted evaluations on three specialized benchmarks and four model-framework configurations. Experiments showed that single-step framework improvements could enhance the performance of GLM-5 and Gemini 3.5 Flash on multiple benchmarks: an improvement of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB; as the task-level recursion budget increases, the magnitude of performance improvement can further expand. The significance of Harness² lies in its attempt to bridge the gap between "completing the current task" and "accumulating experience to improve subsequent tasks," transforming the Harness from a one-time configuration into an execution system capable of continuously adapting to new tasks, and offering a new perspective for agent self-improvement in open environments.
8. Self-Evolving Agent Harness
Paper Title:
Self-Evolving Agent Harness
Paper and Detailed Analysis:
If previous research focused on how to design the Harness, control actions, or recursively optimize task performance, this study poses a more holistic question: When an agent needs to operate over the long term, can the Harness develop a mechanism for continuous self-updating? The study defines “self-evolution” as a feedback-driven process of continuous revision: without altering the underlying model weights, the system can adjust prompts, memory, workflows, and executable components based on execution experience, retaining accepted changes for subsequent runs. The adjustable elements include not only individual modules but also tools and code, repository-level control logic, and the combinatorial relationships between different plugins and components. Meanwhile, the study emphasizes that proposing modifications, judging their effectiveness, and deciding whether to formally adopt them are three distinct stages that must be handled separately; one cannot assume a change has long-term value simply because it improved performance on a single task.
On a theoretical level, the paper combines evolutionary optimization with continual PAC learning to analyze the generalization and performance regression issues that may arise during continuous Harness updates. Under specific assumptions, it discusses expected generalization bounds, retention bounds related to historical regression budgets, and candidate modification admission rules based on fresh, independent validation data. Rather than merely comparing scores on a specific benchmark, this study focuses more on how the Harness retains effective experience through iterative cycles while reducing overfitting, forgetting, and degradation in performance on older tasks. The paper also addresses deployment issues such as shifts in task distribution, evaluator bias, scientific reliability, physical safety, and organizational accountability, noting that continuous verification and independent safety measures remain indispensable. It outlines a direction where the Harness is no longer just a static software structure built around a model, but potentially a system capable of accumulating feedback, prudently modifying itself, and undergoing continuous scrutiny.
