One-sentence Summary

Researchers from CUHK, HKUST, HKU, and ByteDance Inc. propose DreamOmni2, a framework for multimodal instruction-based editing and generation that integrates a synthetic data pipeline with feature mixing and joint VLM training, along with index encoding and position encoding shift to handle multi-image inputs, thereby enabling the editing and creation of both concrete and abstract concepts beyond prior text-only or subject-driven methods and providing new benchmarks for these tasks.

Key Contributions

• Two new tasks, multimodal instruction-based editing and multimodal instruction-based generation, are introduced that accept both text and image instructions and can reference concrete objects or abstract attributes, extending the scope of existing editing and subject-driven generation.
• A three-stage data synthesis pipeline creates training data: a feature mixing method builds extraction data for abstract and concrete concepts, editing and extraction models generate multimodal editing data, and the extraction model produces multimodal generation data. A DreamOmni2 benchmark built from real images is also proposed for these tasks.
• The DreamOmni2 framework uses an index encoding and position encoding shift scheme to distinguish multiple input images and avoid pixel confusion, and employs joint training of the generation/editing model with a vision-language model to improve understanding of complex instructions.

Introduction

Unified image generation and editing models are gaining traction because they let users handle diverse design tasks in one system, reducing friction and deployment overhead while advancing toward more general visual intelligence. However, current approaches remain limited: instruction-based editing depends on text prompts that cannot fully describe complex patterns, materials, or artistic styles, and subject-driven generation typically focuses on concrete objects from a single reference image, ignoring abstract attributes. The authors tackle these gaps by introducing DreamOmni2, which supports multimodal instruction-based editing and generation guided by any concrete or abstract concept through multiple reference images. Their main contribution is a three-stage data pipeline that uses feature mixing to create paired training images and an extraction model to generate references, combined with a model architecture that employs index encoding and position encoding shifts to handle multi-image inputs. They further propose joint training with a vision-language model to better interpret complex real-world instructions, and they build a real-image benchmark to validate the framework’s practical effectiveness.

Dataset

The authors create a synthetic dataset for multimodal instruction-based editing and generation using a three-stage pipeline. The dataset consists of training tuples for two tasks:

• Editing tuples – each contains a source image, an editing instruction, a reference image, and a target image.
• Generation tuples – each contains multiple reference images (one to five), a generation instruction, and a target image.

Data sources and composition

• Target images come from two pools:
• Synthetic (T2I-generated): Diverse object and attribute keywords are combined into prompts by an LLM, then fed to a text-to-image model.
• Real images: A vision-language model extracts keywords directly from real photos.
• Both types are mixed to cover abstract attributes, concrete objects, and natural distributions.
• Reference images are extracted from a target image (or a source image) by a specialized extraction model, using a chosen keyword.

Key details for each subset

• Editing data: The target image is the desired output. The source image is created by applying an instruction-based editing model to alter the selected keyword in the target. The instruction is generated by an LLM. The extraction model provides the reference image.
• Generation data: References are obtained by extracting keywords from stage-2 source images and merging them with stage-2 references. The tuple then includes multiple reference images, an instruction, and the target image.
• No dataset size, exact filtering rules, or mixture ratios are specified in the paper.

How the data is used in the model

• Stage 1 produces paired source-target images via feature mixing. These pairs are used to train the extraction models that later produce reference images.
• The full editing and generation tuples (stages 2 and 3) serve as supervision for training the multimodal instruction-based editing/generation model. No explicit training split or data proportions are detailed.

Processing and pipeline highlights

• Stage 1 (Feature Mixing): A dual-branch diffusion approach generates source and target images simultaneously without halving resolution. This avoids the blending and alignment issues of prior diptych methods and preserves image quality.
• Stage 2 (Editing Data Construction): Target images are collected from T2I and real sources. The trained extraction model isolates a reference according to a keyword. An editing model transforms the target into a source by modifying that keyword, and an LLM writes the corresponding instruction.
• Stage 3 (Generation Data Construction): The extraction model pulls references from the stage-2 sources, complementing existing references to form multi-reference tuples.
• No cropping strategy is mentioned; generation runs at full resolution throughout.

Method

To address the lack of training data formultimodal instruction-based editing and generation, the authors propose a comprehensive synthetic data pipeline consisting of three stages.

In the first stage, they introduce a feature mixing scheme to leverage the base model's text-to-image capabilities. A dual-branch structure is employed to simultaneously generate source and target images. The attention mechanism for the target branch is formulated as: $\mathrm{Attn}_{tar}(\boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V}) = \operatorname{softmax}\left(\frac{\boldsymbol{Q}\boldsymbol{K}^\top}{\sqrt{d}}\right)\boldsymbol{V}$Attntar​(Q,K,V)=softmax(d​QK⊤​)V where $Q = [Q_{tar}^n; Q_{tar}^t]$Q=[Qtarn​;Qtart​], $K = [K_{tar}^n; K_{tar}^t; K_{src}^n]$K=[Ktarn​;Ktart​;Ksrcn​], and $V = [V_{tar}^n; V_{tar}^t; V_{src}^n]$V=[Vtarn​;Vtart​;Vsrcn​]. This scheme avoids the resolution reduction and content blending issues of previous diptych methods, producing high-quality paired data for training extraction models.

In the second stage, the authors create multimodal instruction-based editing data. They generate target images using both text-to-image models and real images. An extraction model, trained in the first stage, is used to extract an object or attribute from the target image based on a selected keyword to create a reference image. An instruction-based editing model then alters the selected keyword in the target image to obtain the source image. Finally, an LLM generates the editing instructions, forming training tuples of source image, instruction, reference image, and target image.

In the third stage, multimodal instruction-based generation data is created. The extraction model extracts keywords from the source images generated in stage 2 to create additional reference images. Combining these with reference images from stage 2 yields training tuples consisting of multiple reference images, an instruction, and a target image. The resulting dataset covers a wide range of object categories and abstract attributions, as shown in the figure below.

To handle multiple input images in multimodal instruction-based tasks, the authors propose the DreamOmni2 framework. Since positional encoding alone cannot accurately distinguish the index of reference images in DiT architectures, they add an index encoding to positional channels. Furthermore, they apply an offset to the position encoding based on the size of previously input reference images, which reduces copy-and-paste artifacts and pixel confusion.

To bridge the gap between well-structured training instructions and irregular real-world user instructions, the authors propose joint training of a VLM and the generation models. The VLM interprets complex user instructions and outputs them in a predefined structured format. For editing, the output combines user instructions with refined image descriptions; for generation, the VLM directly outputs a refined image description.

During training, the authors fine-tune Qwen2.5-VL 7B to learn the predefined standard output format. They then train the editing and generation models using LoRA on Flux Kontext. This approach retains the original instruction-editing capabilities of Kontext while seamlessly integrating multimodal instruction-based editing and generation. Separate LoRA models are trained for generation and editing to allow users to choose whether to preserve source image consistency.

Experiment

The proposed DreamOmni2 benchmark addresses the absence of evaluation suites for multimodal instruction-based editing and generation, covering both abstract attributes and concrete objects. Experiments demonstrate that DreamOmni2 outperforms open-source and commercial models in editing accuracy and generation consistency, with human and VLM evaluations showing fewer unintended alterations. Qualitative visualizations confirm its superior instruction adherence and object fidelity, while joint training of the VLM and editing/generation model, together with index and shifted position encoding, significantly boosts performance.

DreamOmni2 is the only benchmark that supports both multimodal instruction-based editing and generation with multiple reference images, while also covering concrete object and abstract attribute targets. Prior benchmarks are limited to generation only and ignore abstract attributes or multimodal editing. DreamOmni2 expands the task scope to multimodal instruction-based editing and generation, whereas DreamBooth and OmniContext only assess image generation. Unlike existing benchmarks, DreamOmni2 includes editing targets for abstract attributes such as global and local concepts, not just concrete objects.

GPT-4o consistently achieves the highest success rates for both concrete object and abstract attribute editing, leading all automated and human evaluations. Open-source models such as DreamO and UNO show near-zero success in human assessments, while Nano Banana lags noticeably behind GPT-4o, especially on abstract attributions. DreamOmni2 attains the best human evaluation scores overall, significantly outperforming open-source competitors and closely matching GPT-4o’s results. GPT-4o leads all metrics, but its human-judged success rates remain moderate across both editing categories. DreamO and UNO register almost no successful edits in human evaluations for concrete objects or abstract attributions. Nano Banana performs closer to GPT-4o on concrete objects, but its human-rated success drops sharply for abstract attributions. DreamOmni2 achieves the highest human evaluation results, surpassing all other models and rivaling GPT-4o.

In multimodal instruction-based generation, the proposed DreamOmni2 outperforms the commercial Nano Banana and matches GPT-4o's level, while open-source models such as DreamO and Qwen-Image-Edit achieve very low success rates, especially failing almost completely on abstract attributes. Open-source models (DreamO, Qwen-Image-Edit, UNO) obtain near-zero human success rates on abstract attribution generation. DreamOmni2 surpasses Nano Banana and reaches performance comparable to GPT-4o across both concrete and abstract editing tasks.

Joint training of the generation/editing model and a vision-language model yields the highest success rates across all editing and generation tasks. Training only the generation/editing model significantly lifts performance over an untuned baseline, while adding VLM training alone provides mixed but notable gains. The complete joint training scheme nearly doubles the scores for abstract attribute generation and editing compared to any single training component. Jointly training the generation/editing model and VLM together achieves the best results, with success rates above 0.65 for concrete object editing and generation, and over 0.62 for abstract attributes. Adding only generation/editing training greatly improves over the baseline (e.g., from 1.2% to 31.7% for abstract editing), while VLM-only training further helps abstract generation; the combined scheme outperforms both.

Adding index encoding alone improves performance over the baseline, but the best results come from combining index encodings with position encoding shifts. The full scheme achieves the highest editing and generation scores across concrete objects and abstract attributes, demonstrating that both techniques together help the model understand image references and avoid copy-paste effects. Index encoding without position shifts raises editing and generation scores compared to using no encoding at all. Applying position encoding shifts without index encoding yields moderate gains, but still trails the combined approach. Simultaneous use of index encoding and position shifts leads to the top scores, with editing reaching 0.66 on concrete objects and generation reaching 0.67. The joint scheme prevents the copy-and-paste effect while enabling the model to correctly associate 'Image 1' and similar references with the right inputs.

The DreamOmni2 benchmark expands evaluation to multimodal instruction-based editing and generation with multiple reference images, covering both concrete objects and abstract attributes, unlike prior benchmarks limited to generation only. In comparisons, DreamOmni2 achieves human evaluation scores that substantially surpass other open-source models and closely approach GPT-4o, with open-source alternatives failing almost entirely on abstract concepts. Ablation experiments reveal that jointly training the generation model with a vision-language model delivers the highest performance across all tasks, and that combining index encoding with position encoding shifts is essential for correctly interpreting image references while avoiding copy-paste failures.