One-sentence Summary

Researchers from University of Science and Technology of China and FrameX.AI propose FrameForge, a framework that harnesses video foundation model priors for unified conditional image generation by modeling input-adaptive virtual temporal relations through Mixture of Temporal Experts and Chain of Temporal Reasoning, and achieves superior subject-driven generation, canny-to-image generation, and image editing on DreamBench, COCO2017, and GEdit-Bench.

Key Contributions

• FrameForge is a unified adaptation framework that leverages the temporal priors of frozen pretrained video models for heterogeneous conditional image generation by modeling input-adaptive virtual temporal relations between condition and target.
• It introduces Mixture of Temporal Experts (MoTE), which dynamically selects temporal RoPE interval experts based on textual and visual context to model how far the target lies from the condition, and Chain of Temporal Reasoning (CoTR), which uses a latent intermediate state and a reasoning-bottleneck attention mask to model the condition-to-target transformation.
• Experiments on DreamBench, COCO2017, and GEdit show superior performance in subject-driven generation, canny-to-image generation, and image editing over compared methods, while ablation studies verify the effectiveness of adaptive temporal relation modeling and implicit temporal reasoning.

Introduction

Real-world text-to-image generation often requires conditioning on heterogeneous inputs such as reference images, text, and structural maps, which impose different semantic and spatial constraints. Building a unified model is challenging because prior image-centric methods either train large multi-task models or attach control branches and lightweight adapters to pretrained generators, treating conditions as auxiliary signals without explicitly modeling how far the target differs from the condition or how the condition should be transformed into the target. Video foundation models offer a useful temporal prior, but existing adaptations assign condition and target frames to fixed temporal positions, ignoring that image editing, subject-driven generation, and canny-to-image generation require different transformation magnitudes and that samples within the same task also vary. The authors formulate these heterogeneous condition-target relations as an input-adaptive virtual temporal interval and propose FrameForge, which uses Mixture of Temporal Experts (MoTE) to select temporal positional experts and Chain of Temporal Reasoning (CoTR) to learn an implicit condition-to-target transformation pathway, all while keeping the video backbone frozen and using a single shared LoRA.

Method

The authors adapt a pretrained video diffusion model to conditional image generation by formulating the task as target-frame prediction guided by textual and visual conditions. The framework leverages a 3D VAE and a multimodal diffusion transformer to model visual content and temporal dynamics in a compressed latent space. The target image is encoded and perturbed, while visual conditions are encoded into a latent representation using a frozen VAE encoder. To facilitate implicit condition-to-target transformation, the authors introduce a noise-initialized reasoning latent. Together with text representations, these inputs are jointly processed by the backbone, while a specialized module determines their temporal positional configurations. The overall conditioning is expressed as:

Refer to the framework diagram:

To provide a flexible temporal prior for cross-frame interaction, the authors introduce the Mixture of Temporal Experts. Instead of using a fixed temporal gap, this module constructs multiple temporal rotary positional embedding experts corresponding to different virtual intervals. Each expert is associated with a candidate latent span, which translates to an equivalent video-frame span. For a given expert, rotary embeddings are constructed over a virtual latent grid, and the embeddings at specific temporal coordinates are retained for the condition, target, and reasoning latents.

A lightweight router selects the appropriate expert based on the concatenated text, condition, and noisy target representations. The router produces token-level routing logits, which are aggregated via masked average pooling to obtain a sample-level logit vector. The routing probabilities and the Top-1 expert assignment are computed using a softmax function with a temperature parameter. To enable end-to-end optimization with discrete routing, a straight-through estimator is adopted, allowing the forward pass to use a single selected expert while propagating gradients through the soft routing probabilities. The routed rotary embeddings are then computed as a weighted sum and applied to the visual queries and keys in self-attention. To prevent routing collapse and ensure balanced utilization of the temporal experts, a batch-level load-balancing loss is introduced, which is added to the standard diffusion loss.

To model the implicit transformation from condition to target, the authors propose the Chain of Temporal Reasoning. This module inserts an intermediate reasoning latent state between the condition and target. Sampled from a standard Gaussian distribution, this latent is progressively contextualized through the transformer blocks by interacting with the text, condition, and target representations. It serves as a latent pathway for aggregating conditional information before target prediction, trained solely with the target-frame objective without requiring intermediate supervision.

To ensure the model effectively utilizes this intermediate pathway, the authors apply a stochastic reasoning-bottleneck attention mask during the intermediate training stage. This mask removes direct attention between the target tokens and the text or condition tokens, forcing the target tokens to receive conditional information through the evolving reasoning representation rather than through a direct shortcut. The mask modifies the attention connectivity by adding negative infinity to the corresponding self-attention logits, while preserving the original full-attention pattern for samples without masking. This mechanism does not affect the temporal positional conditioning introduced by the temporal experts.

The training strategy involves freezing the pretrained video-backbone weights and the VAE encoder, while inserting a single Low-Rank Adaptation adapter into the attention and feed-forward projections of the transformer. The optimization proceeds in three stages. In the first stage, the authors optimize only the router with the condition-target formulation, keeping the adapter parameters fixed and omitting the reasoning latent to learn temporal routing. In the second stage, the reasoning latent is added, and the router and adapter are jointly optimized while applying the stochastic reasoning-bottleneck mask. In the final stage, both modules are retained, and joint optimization continues with full attention. This final configuration matches the inference setup, where the router selects a temporal expert at each denoising step, and the shared adapter is utilized for every task.

Experiment

FrameForge is evaluated on subject-driven generation, canny-to-image generation, and instruction-guided image editing using a LoRA-fine-tuned HunyuanVideo backbone. The experiments show that the method preserves subject identity while following text prompts, maintains structural fidelity to edge conditions, and improves instruction-guided edits, especially for motion and spatial changes. Routing analysis reveals that MoTE dynamically selects temporal experts according to both task and denoising stage, while ablations confirm that MoTE and CoTR are complementary and that the chosen configurations offer the best overall trade-off. Additional results indicate that these benefits generalize across video backbones.

For subject-driven image generation on DreamBench, the proposed method achieves the strongest overall result among compared approaches, improving over ChronoEdit and outperforming OminiControl, ACE++, and EasyControl. Its subject consistency is second only to ChronoEdit, while methods with higher perceptual quality such as EasyControl and IP-Adapter show substantially lower subject consistency. The results indicate a better balance between preserving subject identity and maintaining image realism. The proposed method ranks first in overall score, ahead of OminiControl, ACE++, and EasyControl. Subject consistency is competitive, ranking second only to ChronoEdit, while IP-Adapter and EasyControl sacrifice subject consistency for higher perceptual quality.

On COCO2017 canny-to-image generation, FrameForge achieves the highest structural similarity among the compared methods, with a 7.2% relative SSIM improvement over the next best model, while its text-image alignment score stays comparable to ControlNet and RealGeneral. Its FID is comparable to video-based baselines such as DRA-Ctrl and RealGeneral. The lowest-FID method still shows weaker structural fidelity, so the evaluated metrics reflect complementary properties. FrameForge attains the best SSIM, a 7.2% relative improvement over the second-best EasyControl, while its CLIP-T score remains comparable to ControlNet and RealGeneral. FrameForge's FID is in the same range as DRA-Ctrl and RealGeneral, and ControlNet's lower FID does not correspond to stronger edge fidelity, highlighting the complementary nature of FID and SSIM.

FrameForge leads open-source methods on motion change and overall GEdit, improving over ChronoEdit by clear margins. Its motion change performance surpasses closed-source Gemini 2.5 and Seedream 4.0 and approaches GPT-4o. Image-backbone adaptation methods lag on motion change despite competitive G-PQ, underscoring the value of video foundation model priors. FrameForge achieves the best open-source scores on motion change and overall GEdit, with clear gains over ChronoEdit. Its motion change score surpasses Gemini 2.5 and Seedream 4.0 and approaches GPT-4o. ACE++ and similar image-backbone adaptation methods struggle on motion change and overall editing despite competitive G-PQ. Closed-source native models still lead overall, but FrameForge narrows the gap on motion-focused editing.

Shared LoRA joint training improves subject-driven generation over task-specific LoRA but reduces editing performance. Adding either MoTE or CoTR mainly recovers editing, while leaving the other tasks relatively stable. Combining both modules yields the best overall results, with the highest subject-driven and editing scores and the lowest canny-to-image FID. Shared LoRA joint training improves subject-driven generation but degrades editing relative to task-specific adaptation. MoTE and CoTR are complementary: individually they mainly improve editing, while the full combination achieves the highest subject-driven and editing scores and the lowest canny-to-image FID.

With CoTR enabled and other settings fixed, varying the number of temporal RoPE experts changes the trade-off across tasks. Three experts achieve the best overall balance, improving editing and canny-to-image FID over one expert while keeping subject-driven generation unchanged. Five or seven experts slightly improve subject-driven generation but degrade editing and canny-to-image generation. Increasing the expert count from one to three improves editing and canny-to-image FID while maintaining the subject-driven score. Five or seven experts slightly improve subject-driven generation but degrade editing and canny-to-image generation. Three temporal RoPE experts give the best overall trade-off among the tested settings.

The experiments cover subject-driven generation on DreamBench, canny-to-image generation on COCO2017, and motion-focused generative editing on GEdit. FrameForge achieves the best overall subject-driven balance between identity preservation and realism, with subject consistency competitive with ChronoEdit, and attains the strongest structural fidelity in canny-to-image while keeping text alignment and FID comparable to strong baselines. On GEdit it leads open-source methods in motion change and overall editing, outperforming several closed-source models on motion and approaching GPT-4o. Ablations indicate that combining shared LoRA joint training with MoTE and CoTR, and using three temporal RoPE experts, provides the best overall trade-off across subject-driven generation, editing, and canny-to-image quality.