One-sentence Summary

DreamX Team at Alibaba Group presents DreamX-Creator 1.0, a compact 7B native joint audio-video generation system that jointly denoises modality-specialized audio and video streams coupled through Gated Cross-Modal Attention, and that combines progressive joint training, audio-video reinforcement learning, and an autoregressive 1-step 2K refinement pipeline to produce synchronized audio-video generation competitive with state-of-the-art open-source systems while democratizing native audio-video generation.

Key Contributions

• The paper introduces DreamX-Creator 1.0, a 7B native joint audio-video generator conditioned on a first frame and text prompt that jointly denoises modality-specialized audio and video streams and couples them through Gated Cross-Modal Attention with token- and head-wise output gates, achieving synchronized audio-video generation competitive with state-of-the-art open-source systems.
• The paper presents a unified Audio-Video Data System that filters temporally coherent clips, produces structured multimodal annotations, and organizes capability-oriented data pools. It also introduces Progressive Joint Training with two audio-video pre-training stages followed by High-Quality Finetuning and Audio-Video Reinforcement Learning using Modality-Aware Multimodal Feedback routed to video, audio, and cross-modal components.
• The paper contributes an Autoregressive 1-Step 2K Refinement pipeline that adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk, and releases the compact 7B generator and 2K Refiner for open research.

Introduction

Generative video has improved rapidly in visual fidelity, motion, and duration, but audio is often omitted or synthesized separately, while directional video-to-audio or audio-to-video pipelines treat one modality as a fixed condition and limit reciprocal interaction when visual and acoustic events are jointly determined. Prior joint audio-video models enable synchronized sampling but still face unresolved challenges in data filtering and annotation, adaptive cross-modal interaction, perceptual and synchronization objectives, and the differing computational needs of joint latent generation versus high-resolution refinement. Many existing systems are also too large, hosted-only, or difficult to reproduce. The authors introduce DreamX-Creator 1.0, a compact open-weight native joint audio-video generator centered on a 7B backbone with gated cross-modal attention, progressive joint training, modality-aware reinforcement learning, and an autoregressive one-step 2K refiner.

Dataset

The authors construct a native audio-video generation corpus from heterogeneous public and internal sources, then process it through filtering, structured annotation, and capability-based organization.

Data sources

• Combines multiple public datasets with internally collected data.
• Public sources include Koala-36M, VGGSound, AudioSet, OpenHumanVid, SpeakerVid-5M, Action-100M, and Talker-T2AV.
• These sources provide complementary coverage across different content domains and audio-visual interactions.

Filtering and preprocessing

• Raw videos are segmented at scene boundaries using PySceneDetect.
• Near-silent segments are removed based on audio RMS energy.
• Invalid or low-resolution samples are discarded.
• Three frames are trimmed from both ends of each segment to reduce boundary artifacts and cross-shot contamination.
• Visual quality is assessed with Q-Align.
• Motion magnitude is estimated with UniMatch-based optical flow.
• Audio quality is assessed with Audiobox Aesthetics.
• General audio-visual synchronization is measured with Synchformer.
• Clips with visible speech are additionally filtered with SyncNet for fine-grained lip-audio alignment.

Structured annotation

• The authors use Qwen3-Omni-30B-A3B-Instruct for joint video and audio analysis.
• Joint perception allows visual and acoustic evidence to constrain each other, reducing hallucinated or cross-modally inconsistent descriptions.
• Annotations capture subjects, actions, scene context, camera view, speech, music, sound events, ambience, temporal event order, and visually grounded versus off-screen audio.
• Qwen3-ASR-1.7B transcribes spoken content.
• Qwen3.6-27B consolidates the multimodal annotation and ASR transcript into a coherent caption.
• The resulting caption is used as the conditioning signal for audio-visual generation.

Capability taxonomy and composition

• Annotated clips are organized into four capability-oriented pools.
• Qwen3.6-27B infers the dominant cross-modal supervision pattern from each structured annotation.
• Pools group clips by dominant supervision signals to support targeted learning of different generation capabilities.
• The organization allows data sampling according to the supervision required by different generation objectives, rather than uniform sampling from a heterogeneous corpus.
• Content distribution: speech accounts for 45.0% of the data, event sounds for 33.4%, and the remainder covers music, natural sounds, and mixed content.
• The excerpt does not specify exact training split sizes or mixture ratios per pool.

Method

The DreamX-Creator 1.0 method is organized as a pipeline that spans data curation, native joint audio-video generation, reinforcement-learning post-training, and efficient high-resolution refinement. The authors first construct an audio-video dataset with multi-dimensional quality filtering and joint audio-visual annotation. They then train a dual-stream diffusion transformer that jointly denoises video and audio latents, using bidirectional cross-modal attention and gated conditioning. After pretraining, they apply a modality-aware reinforcement-learning stage that optimizes decomposed video, audio, and synchronization rewards. Finally, they distill an autoregressive one-step 2K video refiner to increase spatial resolution without modifying motion or audio timing.

Data Filtering and Annotation

Collected videos are first segmented into valid audio-visual clips at scene boundaries using PySceneDetect. Segments with near-silent audio are removed based on audio RMS energy, and invalid or low-resolution samples are discarded. Because scene-boundary localization can be imperfect, the authors trim three frames from both ends of each segment to reduce boundary artifacts and cross-shot contamination. A staged filtering pipeline then evaluates visual quality with Q-Align, motion magnitude with UniMatch-based optical flow, and audio quality with Audiobox Aesthetics.

For cross-modal alignment, Synchformer scores general synchronization between temporally correlated audio and visual events, while SyncNet provides fine-grained lip-audio assessment for clips with visible speech. General clips are filtered by Synchformer, and visible-speech clips must additionally satisfy the SyncNet criterion.

After filtering, the authors construct structured annotations that jointly capture visual content, acoustic events, cross-modal relationships, and spoken text. Qwen3-Omni-30B-A3B-Instruct jointly analyzes the video and audio streams rather than captioning each modality independently. Joint perception allows visual and acoustic evidence to constrain each other and reduce hallucinated or cross-modally inconsistent descriptions. The resulting annotation covers subjects, actions, scene context, camera view, speech, music, sound events, and ambience, while preserving temporal event order and distinguishing visually grounded sounds from off-screen audio. Speech is transcribed separately by Qwen3-ASR-1.7B, and the multimodal annotation and ASR transcript are consolidated by Qwen3.6-27B into a coherent caption that retains temporal structure and spoken content. This caption serves as conditioning for generation.

Native Joint Audio-Video Generation

The native generator takes a first frame and a text prompt and jointly denoises video and audio latent streams. Each stream retains its own token rate, positional encoding, and transformer backbone. Text conditioning is supplied by a shared text encoder through a modality-specific conditioning path. The first half of the network processes the two streams independently. The latter half adds two cross-modal attention paths: audio-to-video, or A2V, uses video queries with audio keys and values, and video-to-audio, or V2A, reverses these roles. When both paths are active, they read the same pre-fusion states and compute their residual updates in parallel. A direction mask selects one path in directional mode or both paths in Joint mode.

Because the two streams have different token rates, their positions are mapped to a shared temporal coordinate system. Temporal rotary position encoding is then applied to cross-modal queries and keys, providing time-aware attention without resampling either latent sequence or replacing the modality-specific positional encoding.

Each training sample is assigned to an A2V, V2A, or Joint mode. Let $m_{a \to v}, m_{v \to a} \in \{0,1\}$ma→v​,mv→a​∈{0,1} denote the direction masks, and let larger $\sigma$σ indicate stronger corruption:

$$\begin{array}{rl} \mathrm{A2V:} & \sigma_v > \sigma_a, \quad (m_{a \to v}, m_{v \to a}) = (1,0), \\ \mathrm{V2A:} & \sigma_a > \sigma_v, \quad (m_{a \to v}, m_{v \to a}) = (0,1), \\ \mathrm{Joint:} & \sigma_v = \sigma_a, \quad (m_{a \to v}, m_{v \to a}) = (1,1). \end{array}$$A2V:V2A:Joint:​σv​>σa​,(ma→v​,mv→a​)=(1,0),σa​>σv​,(ma→v​,mv→a​)=(0,1),σv​=σa​,(ma→v​,mv→a​)=(1,1).​

Thus A2V and V2A use a noisier target stream and a cleaner conditioning stream. Both streams retain their flow-matching losses. Let $h^v$hv and $h^a$ha denote the pre-fusion hidden states. The cross-modal updates are

$$\begin{array}{l} \Delta h^v = m_{a \to v} \operatorname{Attn}\left(Q(h^v), K(\widehat h^a), V(\widehat h^a)\right), \\ \Delta h^a = m_{v \to a} \operatorname{Attn}\left(Q(h^a), K(\widehat h^v), V(\widehat h^v)\right). \end{array}$$Δhv=ma→v​Attn(Q(hv),K(ha),V(ha)),Δha=mv→a​Attn(Q(ha),K(hv),V(hv)).​

For A2V, $\widehat h^a = \operatorname{sg}(h^a)$ha=sg(ha); for V2A, $\widehat h^v = \operatorname{sg}(h^v)$hv=sg(hv); Joint mode uses the original hidden states in both paths. Stop-gradient is applied before key and value projections, preventing the target-stream loss from updating the conditioning backbone through cross-modal attention while leaving attention parameters trainable.

Gated Cross-Modal Attention

Inside each latter-half block, cross-modal attention follows modality-specific self-attention and text cross-attention. For a generic path from source stream $Y$Y to target stream $X$X, the two streams are normalized and projected into a common multi-head attention space:

$$\begin{array}{ll} \bar X = \operatorname{LN}_x(X), & \bar Y = \operatorname{LN}_y(Y), \\ Q = \operatorname{RMSNorm}(W_Q \bar X), & K = \operatorname{RMSNorm}(W_K \bar Y), \quad V = W_V \bar Y. \end{array}$$Xˉ=LNx​(X),Q=RMSNorm(WQ​Xˉ),​Yˉ=LNy​(Y),K=RMSNorm(WK​Yˉ),V=WV​Yˉ.​

After splitting into attention heads, temporal rotary position encoding is applied using coordinates $\tau_x$τx​ and $\tau_y$τy​ in the shared temporal coordinate system. The attention calculation also masks padded positions beyond the valid source-stream length.

The authors use a token-wise and head-wise sigmoid output gate after scaled dot-product attention and before the output projection. Unlike a query-only gate, this gate depends on both the target token $x_i$xi​ and its cross-modal attention output $C_{i,h}$Ci,h​:

$$\begin{array}{l} g_{i,h} = \operatorname{sigmoid}\left([W_x \operatorname{LN}(x_i)]_h + \mathbf w_c^\top \operatorname{LN}(C_{i,h}) + b_h\right), \\ \Delta x_i = W_o\left(\operatorname{Concat}_h [g_{i,h} \odot C_{i,h}]\right), \\ x_i' = x_i + \Delta x_i. \end{array}$$gi,h​=sigmoid([Wx​LN(xi​)]h​+wc⊤​LN(Ci,h​)+bh​),Δxi​=Wo​(Concath​[gi,h​⊙Ci,h​]),xi′​=xi​+Δxi​.​

Here $W_x$Wx​ maps the target hidden state to one logit per head, $\mathbf w_c$wc​ maps the normalized head-wise attention output to a scalar, and $b_h$bh​ is a head-specific bias. The gate is an output modulation mechanism rather than token routing or path selection.

Progressive Joint Training

Training consists of two audio-video pretraining stages followed by high-quality fine-tuning. Across all stages, the authors mix the internal A2V, V2A, and Joint conditioning configurations. Pretraining uses a fixed mixture, while high-quality fine-tuning adjusts their relative emphasis according to the predominant cross-modal dependency of the training content.

In Stage 1, the model is initialized from modality-specific video and audio backbones. The first half of both backbones remains frozen, rank-256 LoRA adapters are applied to the latter half, and the cross-modal attention modules and output gates are optimized directly.

In Stage 2, the LoRA weights are merged into their respective backbones. Training continues on broad-coverage paired data with both diffusion-transformer backbones and all cross-modal modules jointly optimized.

In Stage 3, a curated fine-tuning subset is constructed by filtering candidate clips using audio-video synchronization metrics, modality-specific aesthetic scores for video and audio, spatial resolution, and clip duration. OmniShotCut performs fine-grained shot-boundary detection and transition labeling, and candidate segments containing or spanning a detected transition are discarded. The full-parameter model is then fine-tuned on the retained pairs. The first fine-tuning run inherits group-wise learning rates from Stage 2, and subsequent runs progressively reduce them.

Optimization Details

Flow matching is used for both streams. For modality $m \in \{v,a\}$m∈{v,a}, a clean latent $z_0^m$z0m​ and independent Gaussian noise $\epsilon^m$ϵm are interpolated according to the modality-specific noise level $\sigma_m$σm​:

$$z_{\sigma_m}^m = (1-\sigma_m)z_0^m + \sigma_m \epsilon^m, \qquad u_m^* = \epsilon^m - z_0^m.$$zσm​m​=(1−σm​)z0m​+σm​ϵm,um∗​=ϵm−z0m​.

The token- and feature-normalized flow-matching objective is

$$\mathcal L_{\mathrm{FM}}^m = \frac{1}{N_m d_m} \sum_{j=1}^{N_m} \left\| \widehat u_{\theta,j}^m - u_{m,j}^* \right\|_2^2,$$LFMm​=Nm​dm​1​j=1∑Nm​​​uθ,jm​−um,j∗​​22​,

and the total loss is

$$\mathcal L_{\mathrm{AV}} = \lambda_v \mathcal L_{\mathrm{FM}}^v + \lambda_a \mathcal L_{\mathrm{FM}}^a.$$LAV​=λv​LFMv​+λa​LFMa​.

The two streams use independently sampled Gaussian noise tensors. A base timestep is drawn from a discrete 1,000-timestep flow schedule with a shift factor of 5.0, and modality-specific noise levels follow the directional ordering above. The two pretraining stages use $\lambda_v = \lambda_a = 0.5$λv​=λa​=0.5, while high-quality fine-tuning uses $\lambda_v = 0.5$λv​=0.5 and $\lambda_a = 0.1$λa​=0.1.

The optimizer is AdamW with $(\beta_1,\beta_2)=(0.9,0.99)$(β1​,β2​)=(0.9,0.99) and $\epsilon=10^{-10}$ϵ=10−10. In Stage 1, LoRA parameters use a learning rate of $1 \times 10^{-4}$1×10−4, while cross-modal attention and output-gate parameters use $2 \times 10^{-5}$2×10−5. After merging the adapters, Stage 2 updates backbone parameters with $1 \times 10^{-5}$1×10−5 and cross-modal modules with $2 \times 10^{-5}$2×10−5. Training uses bfloat16 mixed precision and clips the gradient norm at 1.0.

Audio-Visual Reinforcement Learning

Flow-matching objectives do not directly optimize perceptual quality, prompt adherence, cross-modal semantic consistency, or precise temporal correspondence. The authors therefore introduce an audio-visual reinforcement-learning stage that post-trains DreamX-Creator 1.0 with multimodal feedback while keeping both streams inside the joint generator.

A single global reward is insufficient because video quality, audio quality, and synchronization may not improve together. The feedback is therefore decomposed into video-specific, audio-specific, and cross-modal components. Let $A_v$Av​, $A_a$Aa​, and $A_{av}$Aav​ denote normalized advantages. The supervision routed to the two generation streams is

$$\widetilde A_v = A_v + A_{av}, \qquad \widetilde A_a = A_a + A_{av}.$$Av​=Av​+Aav​,Aa​=Aa​+Aav​.

The modality-specific terms improve their corresponding streams, while the shared cross-modal term jointly supervises both streams and their bidirectional interaction modules.

Audio-visual synchronization is mainly determined by a small number of regions, such as visible mouths and sound-producing objects. The authors use video-to-audio responses from selected interaction blocks to estimate the relevance of each video token. Scores are normalized within each frame and converted into token weights:

$$w_i^{(e)} = 1 + \alpha_e \operatorname{sigmoid}\left(\frac{s_i - \mu_{f(i)}}{\sigma_{f(i)} + \epsilon}\right).$$wi(e)​=1+αe​sigmoid(σf(i)​+ϵsi​−μf(i)​​).

Here $s_i$si​ is the cross-modal relevance score of video token $i$i, and $f(i)$f(i) denotes the frame containing that token. The mean and standard deviation are computed within the same frame. The coefficient $\alpha_e$αe​ is gradually increased from zero during warmup, so training begins with near-uniform token weights and progressively focuses on synchronization-relevant regions. The authors also apply depth-dependent gradient scaling to the audio-to-video pathway. Gradients entering the audio stream are attenuated more strongly in shallow blocks and progressively restored in deeper interaction blocks.

For reinforcement learning, the policy is initialized from the pretrained base model. For each first-frame and prompt condition, the current policy generates a group of $G$G joint audio-video candidates. Each candidate is evaluated using video-quality, audio-quality, prompt-consistency, and audio-visual synchronization rewards. The rewards are normalized within the group to obtain relative video, audio, and cross-modal advantages. The training objective is

$$\mathcal L_{\mathrm{RL}} = \lambda_v \mathcal L_v(A_v + \lambda_{av}^v A_{av}) + \lambda_a \mathcal L_a(A_a + \lambda_{av}^a A_{av}) + \lambda_{\mathrm{reg}} \mathcal L_{\mathrm{reg}}(\pi_\theta, \pi_{\mathrm{base}}).$$LRL​=λv​Lv​(Av​+λavv​Aav​)+λa​La​(Aa​+λava​Aav​)+λreg​Lreg​(πθ​,πbase​).

The regularization term constrains the updated policy toward the pretrained base model to preserve diversity and prevent reward over-optimization. Training alternates between grouped candidate generation, multimodal reward evaluation, and policy optimization. The updated policy periodically refreshes the rollout model, forming an online generation-evaluation-update loop. The post-trained model retains the same first-frame-conditioned joint generation interface.

Autoregressive One-Step 2K Refinement

High-resolution video generation requires coherent motion and rich spatial detail. Directly generating 2K videos with the joint audio-visual model is expensive. The authors therefore introduce 2K Refiner, an autoregressive one-step 2K refinement stage. The joint generator first produces a coherent low-resolution video, and 2K Refiner enhances it to 2K resolution while preserving generated content, motion, and audio-synchronized timing.

At inference, 2K Refiner refines the video sequentially over temporal chunks. For each chunk, a single denoising evaluation is conditioned on the low-resolution video and the previously refined high-resolution chunks. This avoids multi-step diffusion sampling at 2K resolution and makes refinement scalable to long videos. The audio stream is not modified.

The refinement training pipeline has three stages. First, a bidirectional multi-step diffusion teacher is trained for high-quality video refinement. It operates in latent space and uses bidirectional spatiotemporal attention to exploit both past and future frames. Flow matching trains the teacher to predict a velocity field conditioned on the low-resolution video. Training pairs are synthesized by degrading high-resolution videos with a curriculum that progresses from mild degradations to stronger blur, noise, compression, resampling, geometric distortion, and temporally correlated corruptions. The temporal corruptions are important because the inputs are generated videos, which can contain flickering textures, local motion jitter, and unstable fine structures.

Second, the bidirectional teacher is adapted into an autoregressive multi-step refiner by teacher forcing. The current chunk is denoised conditioned on the low-resolution video and ground-truth past high-resolution chunks. This causal factorization reduces high-resolution computation while preserving temporal continuity through the autoregressive context.

Third, the autoregressive multi-step refiner is distilled into a one-step student using Distribution Matching Distillation, following the DMD2 formulation. The student predicts the current high-resolution chunk from Gaussian noise, the low-resolution video, and its own previously refined chunks. To avoid exposure bias, distillation uses self-rollout: complete videos are generated by the student itself, and the distribution-matching objective is applied to these generated rollouts. The distilled objective combines distribution matching with pixel-space supervision. Generated latents are decoded with a frozen VAE decoder, and a DISTS perceptual loss plus an $\ell_2$ℓ2​ reconstruction loss are applied against the high-resolution ground truth. This design yields efficient one-step 2K refinement without changing the motion or audio-aligned timing produced by the joint generator.

Experiment

The model is evaluated on Verse-Bench, which covers general audio-visual events and speech-centric scenarios, using unified prompts derived from separate video and audio descriptions. Quantitative experiments assess video quality, audio quality, speech generation, and audio-visual alignment against several research baselines and larger open-weight systems. The refiner improves perceptual video quality while retaining semantic consistency and audio-visual alignment, and user studies show competitive video quality with favorable win rates over several baselines. However, the 7B model still trails larger systems in audio aesthetics, cross-modal semantics, and lip synchronization, especially on more challenging cases.

The data system organizes annotated audio-video clips into four capability-oriented pools based on their dominant cross-modal supervision signals. Each pool supports targeted training for audio-to-video, video-to-audio, or joint generation, while preserving diversity across content types. Speech and event sounds form the largest content categories, with the remainder covering music, natural sounds, and mixed content. Clips are grouped by dominant supervision signal rather than sampled uniformly, enabling targeted learning for A2V, V2A, and joint audio-video generation. Speech and dialogue provide fine-grained facial and articulatory cues for A2V and joint tasks, while actions and Foley provide visible event cues for temporally aligned sound generation.

This preliminary comparison shows the 7B model is competitive but not uniformly dominant across quantitative metrics. It records the best lip-sync score and lowest audiovisual desynchronization, plus relatively strong video quality and speech recognition accuracy, while scoring highest on content usefulness. In contrast, several audio aesthetics dimensions and cross-modal semantics trail the strongest baselines, indicating a trade-off rather than overall superiority. The proposed model achieves the best LSE-C and the lowest DeSync, indicating strong lip synchronization and audiovisual alignment among the compared systems. It also attains the highest CU and competitive VQ and WER, while CE and IB are weaker than the top baselines, pointing to remaining gaps in audio aesthetics and cross-modal semantics.

The 7B model and its variants achieve the best video quality and audiovisual desynchronization scores among the compared systems despite having far fewer parameters. They trail the larger open-weight baselines on most audio aesthetics dimensions and cross-modal semantics, and they remain behind MiniMax-H3 in lip synchronization. The refiner further improves video quality while keeping audio and speech metrics close to the RL variant. The 7B model records higher VQ and lower DeSync than the 22B and 33B baselines, indicating better video quality and audiovisual alignment on these metrics. AudioBox Aesthetics dimensions CE, CU, and PQ favor the larger baselines over the 7B variants, while PC is lower and therefore better for the 7B model. Lip synchronization measured by LSE-C is highest for MiniMax-H3, with the 7B variants scoring below it. The refiner achieves the highest VQ among the listed models and improves DeSync over the base model, while audio aesthetics remain similar to the RL variant.

All evaluated refinement methods improve perceptual quality metrics over the baseline, particularly aesthetic, MUSIQ, and MANIQA, while reducing LSE-C and IB relative to the baseline. Among the refinement methods, the proposed refiner achieves the best MUSIQ, MANIQA, LSE-C, and IB and keeps DeSync low, though DeSync remains higher than the baseline. This points to a more balanced trade-off between perceptual enhancement and preservation of audio-visual and semantic consistency. All refinement methods improve aesthetic, MUSIQ, and MANIQA over the baseline. Every refinement method degrades LSE-C and IB relative to the baseline, reflecting reduced lip-sync and semantic alignment. The proposed refiner ranks best among refiners on MUSIQ, MANIQA, LSE-C, and IB. FlashVSR has the lowest DeSync among refiners, while the proposed refiner has the second-lowest DeSync.

The evaluation uses capability-oriented pools organized by dominant cross-modal supervision signals, covering speech, events, music, natural sounds, and mixed content for audio-to-video, video-to-audio, and joint generation. The 7B model is competitive but not uniformly dominant, with strong lip synchronization, audiovisual alignment, video quality, and content usefulness, while trailing larger baselines on several audio aesthetics and cross-modal semantics metrics. Refinement methods improve perceptual quality metrics such as aesthetic, MUSIQ, and MANIQA but reduce lip-sync and semantic alignment, with the proposed refiner offering the best balance among refiners on MUSIQ, MANIQA, LSE-C, and IB while keeping desynchronization relatively low. Overall, the results highlight efficient video and alignment strengths in the smaller model and a trade-off between visual enhancement and audiovisual or semantic consistency during refinement.