One-sentence Summary

The authors introduce JarvisHub, a canvas-native creative agent harness that transforms an editable canvas into a shared project state, external memory, and action space, enabling long-horizon multimodal creation through typed nodes and links, a three-layer architecture, and human-steerable automation, thereby moving beyond isolated tool use and linear interactions.

Key Contributions

• JarvisHub is a canvas-native agent harness that models a creative project as an editable graph of typed nodes and links, serving as a shared workspace, external memory, and action space for long-horizon multimodal creation.
• The open harness enables project-state benchmarks, process-level evaluation protocols measuring context preservation, tool-use appropriateness, and repair success, and structured trajectory data for training future creative agents.
• Qualitative demonstrations on narrative media generation, interactive web development, and presentation deck generation show that JarvisHub supports inspectable and reusable agentic workflows, and recorded trajectories capture canvas states and agent actions for evaluation and model improvement.

Introduction

Multimodal generation models now produce high-quality images, video, audio, and UI assets, but real creative work involves evolving project states filled with references, drafts, revisions, and feedback that are poorly captured by single-prompt tools, linear chat agents, or manually wired node editors. Prior systems either hide this project context, constrain it to a chat thread, or lock it inside closed commercial products, making it difficult to study how agents maintain and reason over a long-horizon creative workspace. The authors address this gap with JarvisHub, a canvas-native creative agent harness that treats the canvas as a shared, editable project graph. It provides a layered architecture—canvas state storage, a protocol bridge for validated agent actions, and an agent runtime for tool use and trajectory recording—so that agents can inspect, revise, and extend a project state while preserving visibility into every decision.

Method

The authors present JarvisHub, a canvas-native agent harness designed for long-horizon multimodal creation. Instead of relying on chat history, the system treats the canvas as the primary project memory, storing artifacts, dependencies, revisions, feedback, and intermediate results. As shown in the figure below, each turn follows a structured loop where the runtime observes the canvas, interprets user input, selects a permitted action via a protocol bridge, invokes the necessary capability, and writes the observation back to the canvas.

The canvas state serves as the shared project object that users edit and agents read or update. At turn $t$t, a creative project is represented as $\mathcal{C}_t = (\mathcal{G}_t, \mathbf{X}_t, \mathbf{M}_t, \mathbf{U}_t, \mathbf{L}_t)$Ct​=(Gt​,Xt​,Mt​,Ut​,Lt​), where $\mathcal{G}_t = (\mathcal{V}_t, \mathcal{E}_t)$Gt​=(Vt​,Et​) is a typed artifact graph. Here, $\mathcal{V}_t$Vt​ denotes the set of canvas nodes and $\mathcal{E}_t$Et​ represents directed typed relations such as reference use, version lineage, or generation dependency. The terms $\mathbf{X}_t$Xt​, $\mathbf{M}_t$Mt​, $\mathbf{U}_t$Ut​, and $\mathbf{L}_t$Lt​ store editable contents, provenance and metadata, user interactions, and spatial layouts, respectively. Each node $v_i$vi​ is instantiated as $v_i = (\mathrm{id}_i, k_i, \mathbf{p}_i, \mathbf{x}_i, \mathbf{y}_i, \mathbf{m}_i, s_i)$vi​=(idi​,ki​,pi​,xi​,yi​,mi​,si​), capturing its identifier, kind, position, inputs, outputs, metadata, and runtime status. This separation ensures artifacts are addressable, reusable, and have inspectable dependencies.

To ensure canvas updates are explicit and recoverable, the authors enforce a protocol bridge between the agent and the canvas. Given a user query $q_t$qt​ and the current state $\mathcal{C}_t$Ct​, the bridge provides a capability manifest $\Gamma_t$Γt​ and derives an execution grant $\Omega_t$Ωt​ that limits permissible actions. The agent proposes an action $a_t$at​, which is executed only if it can be encoded as a checked tool call or canvas mutation. The resulting observation $o_t$ot​ and any feedback signal $f_t$ft​ drive the state transition:

$$\mathcal{C}_{t+1} = \mathcal{F}(\mathcal{C}_t, a_t, o_t, f_t, r_t)$$Ct+1​=F(Ct​,at​,ot​,ft​,rt​)

where $\mathcal{F}$F is the state-transition operator that writes accepted actions and evidence to the canvas, making interactions auditable rather than hidden inside language generation.

The agent runtime orchestrates creative tasks by converting decisions into valid canvas updates. As illustrated in the figure below, the runtime interprets the request against the canvas, filters operations by the execution grant, and invokes capabilities.

This process defines a granted action space:

$$\mathcal{A}_t = \mathcal{A}(\Omega_t, \Gamma_t, \mathcal{C}_t, q_t), \quad a_t \in \mathcal{A}_t$$At​=A(Ωt​,Γt​,Ct​,qt​),at​∈At​

The runtime groups capabilities into tool families, including canvas tools, generation tools, and native tools. It further leverages skills for reusable creative procedures, memory for cross-turn consistency, and subagents for parallel subtask exploration. Together, these components allow the workflow to explore alternatives while keeping the state anchored in the canvas.

Long-horizon creation requires actionable feedback. Signals $f_t$ft​ from users, evaluators, or subagents determine the next state transition, such as continuing from an accepted node, repairing a failed node, or asking for clarification. The harness records these signals alongside the execution context in a formal trajectory:

$$\tau = \{(q_t, \mathcal{C}_t, \Gamma_t, \Omega_t, a_t, o_t, f_t, r_t, \mathcal{C}_{t+1})\}_{t=1}^T$$τ={(qt​,Ct​,Γt​,Ωt​,at​,ot​,ft​,rt​,Ct+1​)}t=1T​

This ensures that every mutation and artifact usage is traceable for recovery and future analysis.

Experiment

The evaluation of JarvisHub on three long-horizon creative tasks (narrative media generation, interactive web development, and presentation deck generation) demonstrates its canvas-native harness supports persistent project context and iterative refinement. Using GPT-5.5 as the agent backend, the experiments produce workspace traces and final artifacts that show how plans, references, and progress remain inspectable throughout the workflow. The qualitative results confirm that the canvas preserves narrative continuity, visual direction, and structural consistency across complex, multi-step deliverables.

JarvisHub's canvas-native harness supports persistent project state, controlled actions, tool execution, feedback-guided revision, and traceability. Qualitative experiments on narrative media generation, interactive web development, and presentation deck creation demonstrate that these capabilities enable inspectable, coherent long-horizon workflows where the canvas preserves context, dependencies, and revision history. Persistent project state keeps prompts, references, drafts, and feedback in an editable canvas, allowing agents and users to return to earlier work without losing context. Feedback-guided revision uses human or model input to continue from accepted results, locally repair failed nodes, ask for clarification, or stop, supporting iterative refinement. Traceability records requests, actions, observations, feedback, and checkpoints, making the entire creative process inspectable and restorable as shown by workspace traces across all tasks.

The agent runtime groups its capabilities into five tool families—canvas, generation, native, recovery, and MCP—each with a distinct role such as updating project state, producing artifacts, or using external execution. All families operate under a unified manifest-and-grant contract, where actions are granted per turn, invoked by the agent, and committed back to the shared canvas through the same protocol bridge, ensuring no hidden state accumulates outside the canvas. Canvas tools manage project state by reading, creating, updating, and connecting nodes, while generation tools produce image, video, audio, and composed-media artifacts directly on the canvas. Every tool family, including native and MCP tools, adheres to the same manifest-and-grant contract and commits results through the protocol bridge, keeping the canvas as the sole project state.

The evaluation covers three long-horizon creative tasks—narrative media generation, interactive web development, and presentation deck generation—each requiring planning, reference organization, intermediate artifacts, feedback, and local revision. Qualitative results indicate that the canvas-native environment preserves inspectable project context and supports coherent deliverables by maintaining narrative continuity, visual consistency, and structural organization throughout the creative workflow. Narrative media generation tasks demand identity preservation, style consistency, and cross-shot continuity across visual or audiovisual sequences. Interactive web development involves coordinating layout design, interaction logic, and frontend code with iterative preview inspection and revision. Presentation deck generation tests content selection, narrative organization, slide layout, and cross-slide visual consistency. The canvas preserves narrative and visual continuity across shots in media generation, visual direction and interface consistency in web development, and topic structure and visual style across slides in presentation decks.

JarvisHub's canvas-native harness combines persistent project state, feedback-guided revision, and a unified manifest-and-grant contract across all tool families to enable inspectable, coherent long-horizon workflows. Qualitative experiments on narrative media generation, interactive web development, and presentation deck creation show that the canvas preserves context, dependencies, and revision history, maintaining narrative continuity, visual consistency, and structural organization throughout each task. By committing all actions and artifacts to a shared canvas through a protocol bridge, the environment supports iterative refinement and full traceability without hidden state.