One-sentence Summary

DynamiCrafter, introduced by researchers from The Chinese University of Hong Kong, Tencent AI Lab, and Peking University, animates open-domain images by leveraging text-to-video diffusion priors, projecting the input into a text-aligned context representation via a query transformer and concatenating the full image with initial noises for detail preservation, thus producing visually convincing motions with high fidelity to the original image.

Key Contributions

• DynamiCrafter leverages the motion prior of pre-trained text-to-video diffusion models to animate open-domain images, moving beyond the constrained domains of prior methods.
• A dual-stream image injection mechanism is introduced: a query transformer projects the still image into a text-aligned context representation, and the full image is concatenated with the initial noise to preserve fine visual details.
• Experimental results show that the generated videos exhibit more logical and natural motion, higher conformity to the input image, and notable superiority over existing competitors.

Introduction

The authors tackle the problem of animating still images into videos with natural motion while preserving the original visual details, a task with applications in storytelling, content creation, and visual effects. Prior methods are largely restricted to narrow object categories or simple oscillating movements, and recent text-to-video diffusion models, when conditioned on an image, either lose fine details or produce temporally incoherent motion due to inadequate image injection strategies. The main contribution is DynamiCrafter, a framework that introduces a dual-stream image conditioning mechanism. It combines text-aligned context features for semantic understanding with a direct visual detail pathway, enabling a pre-trained video diffusion model to generate vivid, detail-consistent animations for open-domain images, and it further explores text-based dynamic control.

Dataset

The authors construct a dataset specifically for decoupled training of motion control using text. The dataset is derived from the WebVid10M dataset through filtering and re-annotation.

• Source: WebVid10M video-caption dataset.
• Filtering & Re-annotation: The original captions are filtered and rewritten to contain purer descriptions of motion and dynamics, rather than heavy scene descriptions. For example, a caption might be simplified to “Man doing push-ups.”
• Composition: The resulting dataset includes captions with a clear focus on actions and motions, along with category labels such as “human.” No explicit size or split details are provided, but the dataset is a processed subset of WebVid10M.
• Purpose & Usage: The dataset is used to train a variant called DynamiCrafterDCP in a decoupled manner. In this training, the image condition already provides scene context, so the text input is specifically designed to convey motion/action information. This decoupling gives the model stronger text-based control over generated dynamics.
• Processing: The main processing is the re-annotation step that strips scene descriptions and emphasizes dynamic wording, with no mention of additional cropping or metadata construction beyond the caption refinement.

Method

The authors aim to animate a still image to produce a short video clip that inherits the visual content and exhibits natural dynamics. They formulate this as an image-conditioned video generation task and tackle it by utilizing the generative priors of pre-trained video diffusion models.

Preliminary: Video Diffusion Models Diffusion models define a forward process to convert data into Gaussian noises and learn to reverse this process by denoising. The denoising network is supervised by the objective:

For video generation, Latent Diffusion Models (LDMs) are used to reduce computational complexity. The authors base their study on the open-source video LDM VideoCrafter. A video is first encoded into a latent representation frame-by-frame. Both the forward diffusion and backward denoising processes are performed in this latent space, and the generated videos are obtained through a decoder.

Image Dynamics from Video Diffusion Priors To animate a still image with text-to-video (T2V) generative priors, the visual information must be injected into the video generation process comprehensively. The image should be digested for context understanding, and visual details should be preserved. The authors propose a dual-stream conditional image injection paradigm, consisting of text-aligned context representation and visual detail guidance. Refer to the framework diagram:

Text-aligned context representation. To guide video generation with image context, the authors project the image into a text-aligned embedding space. They employ the CLIP image encoder to extract image features. Instead of using only the global semantic token, they use the full visual tokens from the last layer of the CLIP image ViT to capture more complete information. A learnable lightweight model $\mathcal{P}$P, utilizing a query transformer architecture with stacked cross-attention and feed-forward networks, translates these visual tokens into the final context representation $\mathbf{F}_{\mathrm{ctx}} = \mathcal{P}(\mathbf{F}_{\mathrm{vis}})$Fctx​=P(Fvis​).

The text embedding $\mathbf{F}_{\mathrm{txt}}$Ftxt​ and context embedding $\mathbf{F}_{\mathrm{ctx}}$Fctx​ interact with the U-Net intermediate features $\mathbf{F}_{\mathrm{in}}$Fin​ through dual cross-attention layers:

where $\lambda$λ is a coefficient that fuses text-conditioned and image-conditioned features, achieved through tanh gating and adaptively learnable for each layer.

The learned coefficients across different layers indicate that image information has a more significant impact on the two-end layers compared to the intermediate layers. Increasing $\lambda$λ in intermediate layers suppresses cross-frame movements, while decreasing it challenges shape preservation. As shown in the figure below:

Visual detail guidance (VDG). While the rich context representation enables the model to produce videos resembling the input image, minor discrepancies may occur due to the CLIP encoder's limited capability to fully preserve input details. To enhance visual conformity, the authors concatenate the conditional image with per-frame initial noise and feed them to the denoising U-Net as guidance. This allows the model to integrate both global context and local details complementarily.

Text prompts remain necessary even with a rich context representation. The context representation carries more extensive information than text embeddings, which may overburden the T2V model and cause shape distortion. Additional text prompts offer a native global context to help the model utilize image information efficiently. As shown in the figure below:

Training Paradigm The conditional image is integrated through two complementary streams. To modulate them cooperatively, the authors devise a dedicated training strategy consisting of three stages:

1. Training the image context representation network $\mathcal{P}$P based on a lightweight text-to-image (T2I) model to focus on image context learning.
2. Adapting $\mathcal{P}$P to the T2V model by jointly training $\mathcal{P}$P and the spatial layers of the T2V model.
3. Joint fine-tuning with VDG by concatenating the input image with per-frame noise. Only $\mathcal{P}$P and the spatial layers are fine-tuned to avoid disrupting the pre-trained temporal prior knowledge. During training, a video frame is randomly selected as the image condition to prevent shortcut learning and ensure flexible context representation.

Motion Control using Text Since images are associated with multiple potential dynamics, text can guide the generation of dynamic content. However, existing datasets often mix scene descriptions with motion descriptions. For image animation, the scene description is included in the image condition, so the motion description should be treated as the text condition for decoupled training.

To enable this, the authors construct a dataset by filtering and re-annotating the WebVid10M dataset. The process involves filtering out camera movements, ensuring caption-video alignment, and removing graphics or CGI, followed by using GPT-4 to extract dynamic wording and categories. As shown in the figure below:

The resulting dataset contains captions with purer dynamic wording. Training a model on this dataset improves text-based control over dynamics, allowing the model to reflect specific motion prompts effectively compared to other methods. As shown in the figure below:

Experiment

The evaluation setup uses a dual-stream injection into a video diffusion prior, trained on WebVid-10M and tested on UCF-101 and MSR-VTT with FVD, KVD, and a new perceptual conformity metric. Qualitative comparisons with open-source and commercial methods show that DynamiCrafter achieves better temporal coherence, motion quality, and visual fidelity to the input image, while user studies confirm its superiority in motion and conformity. Ablation studies validate that both context and detail streams, adaptive gating, and a staged training paradigm are critical for coherent animation, and the framework can be extended to storytelling, looping video, and frame interpolation.

The proposed method outperforms VideoComposer and I2VGen-XL on zero-shot FVD and PIC across UCF-101 and MSR-VTT, with large gains in video quality and perceptual conformity. The dual-stream image injection design drives the improvement, but on UCF-101 the KVD is slightly higher than I2VGen-XL. On MSR-VTT, the method achieves the best KVD as well. On UCF-101, FVD is reduced by over 140 points and PIC increases by over 0.076 compared to the next best competitor; improvements on MSR-VTT are also substantial. On UCF-101, KVD is slightly higher than I2VGen-XL, but on MSR-VTT the method achieves the lowest KVD, outperforming all competitors.

The proposed method outperforms all open-source baselines by a wide margin in motion quality, temporal coherence, and input conformity. It achieves temporal coherence and input conformity on par with commercial products, while delivering noticeably better motion quality than any compared system. The user study confirms a clear advantage in motion quality over both proprietary and open-source alternatives. Preference for motion quality is highest for the proposed method (38.84%), surpassing commercial products PikaLabs (28.60%) and Gen-2 (22.91%). Input conformity is on par with PikaLabs (79.88% vs. 79.07%) and far exceeds open-source methods, which score below 19%.

The full method balances temporal coherence and visual fidelity. Removing the context injection stream severely degrades both FVD and PIC, while excluding the visual detail guidance stream improves FVD and PIC at the cost of shape distortions and limited motion. Training on the entire T2V model or conditioning only on the first frame further harms stability and consistency. Ablating the context stream causes a large FVD increase and a PIC drop, reflecting severe temporal inconsistency. Removing the visual detail guidance stream yields the best FVD and PIC but leads to distorted shapes and weak motion. Omitting adaptive gating slightly worsens FVD and PIC, producing shaky and unnatural motion. Replacing full visual tokens with a single global CLIP token raises FVD substantially while keeping PIC similar, giving only semantic similarity. Fine-tuning the entire T2V model makes FVD surge to 364.11 and destabilizes the training. Conditioning solely on the first frame degrades both FVD and PIC, causing a content sudden change effect.

The method is evaluated through zero-shot video generation on UCF-101 and MSR-VTT, a user study comparing against open-source and commercial systems, and an ablation of its components. It achieves clear gains in video quality and perceptual conformity over competitors, with motion quality surpassing even commercial products and input fidelity matching them. The dual-stream image injection design is critical, as removing context injection harms temporal consistency while omitting visual detail guidance causes shape distortions despite improved metrics, confirming that the full model best balances temporal coherence and visual fidelity.