One-sentence Summary

Researchers from Meituan, MBZUAI, Huazhong University of Science and Technology, and other institutions propose AutoDesign, a framework that aligns with human design priors by using a meta-harness optimizer to guide a code agent in recursively improving the harness based on rollout feedback; in the paper-to-poster PosterBench evaluation, AutoDesign achieves a score of $78.32$78.32, surpassing Claude Design by $7.45$7.45 points, and it raises the average PosterBench Score from $54.99$54.99 to $67.39$67.39.

Key Contributions

• AutoDesign is a meta-harness optimization framework that recursively updates one DesignHarness component at a time from aggregated rollout trajectories, source and rendering diagnostics, evaluator feedback, and reference posters while keeping model weights fixed.
• PosterBench is introduced as a unified evaluation protocol for academic paper-to-poster generation, with a 100-paper Main Track spanning five disciplines and a shared 10-paper PosterBench-mini subset for controlled evaluation.
• AutoDesign achieves the highest PosterBench Main Track score of 78.32, surpassing Claude Design by 7.45 points. Across seven controlled code agent-model configurations, the learned DesignHarness improves average PosterBench Score from 54.99 to 67.39 (+12.4%), and a system-blind human study rates AutoDesign highest in preference.

Introduction

Multimodal design requires turning heterogeneous sources into editable, human-facing artifacts such as posters, slides, and webpages, making it a challenging long-horizon task because systems must extract evidence, reason across mixed inputs, plan revisions, and respond to feedback. Prior systems often rely on generation-critique-revision cycles and may retain reflections or skills, but they typically treat human-aligned feedback as a transient correction to one output rather than as reusable knowledge for improving the production system itself. The authors propose AutoDesign, a meta-harness optimization framework that recursively improves a design harness across tasks by grounding human preferences through an evaluator initialized from annotated references and by accepting harness updates only when they improve training performance without degrading a development set. They instantiate this approach as DesignHarness for academic paper-to-poster generation and introduce PosterBench, a benchmark and evaluation protocol that jointly measures source fidelity, scientific communication, layout, readability, and visual quality.

Method

The authors define a design harness $H$H as the system surrounding a fixed model $\pi_\theta$πθ​ that transforms a multimodal source $x$x into a human-facing artifact $y$y given context $c$c:

$$y \sim H(\pi_\theta, x, c)$$y∼H(πθ​,x,c)

The harness produces the artifact through an execution trajectory $\tau$τ, which records the sequence of intermediate actions and revisions. To enable systematic meta-harness optimization and facilitate credit assignment, the design harness is decomposed into five functional components: Context and Memory, Tools and Specifications, Execution Runtime, Orchestration, and Evaluation and Feedback.

The meta-harness operates on this design harness to improve its concrete implementation. Given a user specification and an initial harness $H_0$H0​, the meta-harness yields an optimized harness $H_T$HT​. The optimization target is the expected quality of artifacts produced by the design harness:

$$J(H) = \mathbb{E}_{(x, c) \sim p_{\text{task}}, y \sim H(\pi_\theta, x, c)} [R_{\text{meta}}(y, x, c)]$$J(H)=E(x,c)∼ptask​,y∼H(πθ​,x,c)​[Rmeta​(y,x,c)]

where $R_{\text{meta}}$Rmeta​ is the evaluator used to assess artifact quality. The objective is to find $H^\star = \arg\max_H J(H)$H⋆=argmaxH​J(H). Throughout this process, the model parameters $\theta$θ remain fixed, meaning the optimization acts on the system surrounding the model rather than the model weights themselves.

AutoDesign organizes artifact generation and harness optimization as two nested feedback loops.

The inner loop improves a single artifact under a fixed design harness $H$H. It initializes a scaffold with a designer $M_{\text{design}}$Mdesign​ and a critic $M_{\text{critic}}$Mcritic​. At refinement step $k$k, the designer generates the artifact $y_k$yk​ based on the previous state and feedback, while the critic evaluates it:

$$y_k = M_{\text{design}}(y_{k-1}, f_{k-1}; x, c)$$yk​=Mdesign​(yk−1​,fk−1​;x,c) $$f_k = M_{\text{critic}}(y_k; x, c)$$fk​=Mcritic​(yk​;x,c)

This interaction produces an execution trajectory $\tau$τ under the current design harness.

The outer loop improves the design harness $H$H across tasks based on evidence collected from multiple generation runs.

Each outer-loop iteration proceeds through four stages: rollout, evaluation, update proposal, and acceptance.

1. Rollout: The current harness $H_t$Ht​ is executed on a training set to produce artifacts and corresponding trajectories.
2. Evaluation: An evaluator $R_{\text{meta}}$Rmeta​ assesses artifacts across seven quality dimensions, including Faithfulness, Coverage, Density, Visual Evidence, Layout, Readability, and Aesthetics. This evaluator combines rule-based checks for measurable properties with VLM-based judgments for perceptual properties.
3. Update Proposal: An optimizer $P$P, instantiated as a coding agent acting as a planner and code editor, analyzes trajectories and scores to propose a candidate harness $H'_{t+1}$Ht+1′​: $H'_{t+1} = P(H_t, \tau_t, s_t, \mathcal{L})$Ht+1′​=P(Ht​,τt​,st​,L) Updates are restricted to exactly one of the five harness components per iteration to keep credit assignment interpretable.
4. Acceptance Gate: The candidate is accepted only if performance improves on the training set and does not decline on an independent development set: $\text{Accept}(H'_{t+1}) \iff J_{\text{train}}(H'_{t+1}) > J_{\text{train}}(H_t) \land J_{\text{dev}}(H'_{t+1}) \geq J_{\text{dev}}(H_t)$Accept(Ht+1′​)⟺Jtrain​(Ht+1′​)>Jtrain​(Ht​)∧Jdev​(Ht+1′​)≥Jdev​(Ht​)

The system also supports an optional human-in-the-loop mode where users can provide directional guidance to the planner or revise the evaluator if systematic bias is detected.

The meta-harness optimization yields a final DesignHarness architecture that supports multiple output media, such as academic posters, slides, and web pages.

The resulting architecture consists of four main stages:

1. Paper Ingestion: This stage transforms the input source and design context into a structured, provenance-aware context. It extracts document metadata, section outlines, key passages, and visual evidence, organizing them into a content brief and artifact plan. Every element retains a reference to its source location for traceability.
2. Artifact Generation and Revision: The designer module, implemented as a coding agent, generates or revises the artifact as editable HTML files. This allows revisions to be implemented as localized code edits without requiring full regeneration.
3. Validation and Finalization: At each refinement step, a rule-based validator applies deterministic blocking checks for issues like missing assets, broken provenance links, or layout violations. If the candidate passes, the loop terminates. If it fails, the artifact is rendered and inspected by a Critic VLM for perceptual properties like aesthetics and readability. The combined feedback guides the next revision. The loop allows up to 12 attempts.
4. Finalization: The best valid candidate undergoes post-processing, including rendering adjustments and asset inlining, to produce a self-contained output. If the attempt budget is exhausted without passing all checks, fallback mechanisms identify a deliverable candidate.

Experiment

The experiments benchmark poster generation using PosterBench, a frozen evaluator spanning 100 papers across five disciplines, and find that AutoDesign achieves the highest overall performance among compared systems. Ablations isolate the design harness and show that it consistently improves results across multiple model and coding-agent configurations, while visual preview feedback helps correct layout and clipping failures. A blind human evaluation ranks AutoDesign first and finds moderate but meaningful agreement between automatic scores and human preference, with larger benchmark score gaps predicting more consistent human judgments. Qualitative trajectory analysis confirms that iterative diagnostic feedback produces localized repairs and preserves valid content.

AutoDesign leads the evaluated systems on overall poster quality, with its Claude 4.8 configuration achieving the strongest benchmark result and the highest blind human preference estimate. The benchmark tracks human judgment with a positive but moderate association, and its predictions become more consistent with reviewers when score differences are larger. A representative revision trace shows critic feedback producing localized layout repairs while preserving valid content. AutoDesign attains the highest overall benchmark performance, ahead of Claude Design and OpenDesign configurations. Blind reviewers prefer AutoDesign over other systems, and agreement between benchmark-preferred posters and human choices increases with larger score gaps.

AutoDesign configurations lead the PosterBench Main Track, with the Claude Code and Claude 4.8 setup scoring highest and the Codex setup nearly matching through stronger aesthetics. Claude Design and OpenDesign systems trail overall despite some competitive dimension scores. Human preference judgments also favor AutoDesign, and agreement with human judges improves as benchmark score gaps widen. AutoDesign with Claude Code achieves the strongest overall score, supported by high density, layout, and readability. AutoDesign with Codex nearly matches the top score but trades visual evidence and layout for stronger aesthetics. Claude Design posts the highest coverage and visual evidence among listed systems, yet lower density and readability limit its overall score. OpenDesign variants rank lowest overall, with the Codex configuration showing comparatively weak visual evidence and readability. Human preference places AutoDesign ahead of Claude Code, OpenDesign, and Claude Design, and benchmark-preferred posters align more often with human choices when score margins are larger.

The leading configuration is AutoDesign paired with Codex and GPT 5.5, which records the highest overall PosterBench Score and the top faithfulness, density, readability, and aesthetics scores among the listed systems. AutoDesign also ranks second overall when paired with Claude Code and Claude 4.8, while OpenDesign configurations and Claude Design score lower overall. Claude Design achieves the strongest coverage score but is held back by weaker density and readability. AutoDesign with Codex and GPT 5.5 leads overall and performs best on faithfulness, density, readability, and aesthetics. Within the same coding agent and model, AutoDesign outperforms OpenDesign and Claude Design on overall score. Claude Design obtains the top coverage score but falls behind on density and readability, resulting in a lower overall rank. The weakest overall configuration is OpenDesign with Codex and GPT 5.5, creating a wide gap from the leading system.

On the fixed 10-paper subset, AutoDesign leads the design harness track with the highest overall weighted rubric score, ahead of OpenDesign and Claude Design. Its advantage is driven by layout and readability, though it records the lowest visual evidence among the three design variants. In the coding harness track with AutoDesign and GLM 5.2 fixed, Kimi Code outperforms ZCode by a wide overall margin, with broad advantages across most dimensions. AutoDesign leads the design harness track overall, but has weaker visual evidence relative to OpenDesign and Claude Design. Kimi Code outperforms ZCode across most scoring dimensions, with especially large advantages in layout and readability.

Attaching DesignHarness improves the PosterBench Score in every completed configuration, with gains spanning roughly five to twenty points. Improvements occur across multiple model and coding-agent pairings, and lower-baseline configurations tend to show larger gains. The largest single improvement is for DeepSeek V4 Pro with Claude Code. DesignHarness increases PosterBench Score for all seven completed configurations, with gains between about five and twenty points. The largest gain appears for DeepSeek V4 Pro with Claude Code, while higher-scoring baseline setups such as GPT-5.5 with Codex and Claude 4.8 with Claude Code improve more modestly.

AutoDesign configurations consistently achieve the highest overall poster quality and human preference across evaluations, outperforming Claude Design and OpenDesign, with leading results driven by layout, density, and readability even though some variants trail in visual evidence or coverage. Benchmark preference aligns more closely with human judgment when score differences are larger, and critic feedback supports localized layout repairs without disturbing valid content. In fixed-subset comparisons, AutoDesign also leads the design harness track, while Kimi Code outperforms ZCode in the coding harness. Attaching DesignHarness improves PosterBench scores across all completed configurations, with larger gains for lower-baseline setups.