One-sentence Summary

NVIDIA's SANA-Video 2.0, a video diffusion transformer that combines hybrid linear–softmax attention (with periodic softmax anchors at a $3:1$3:1 ratio) and block attention residuals to restore full-rank interactions, is trained from scratch for efficient high-resolution generation, and its optimized pipeline runs $3.2\times$3.2× faster than a softmax baseline while achieving a $120\times$120× speedup over Wan 2.2-A14B and matching DiT quality.

Key Contributions

• The paper presents a hybrid video diffusion transformer that combines 75% gated linear attention with 25% periodic gated-softmax anchors, avoiding quadratic cost while recovering the full-rank token interactions missing from pure linear attention.
• It introduces Block Attention Residuals (AttnRes), which route completed attention-block summaries into later linear layers, enabling anchor-feature reuse across depth and boosting deep-layer effective rank by roughly 12%.
• From-scratch training with reduced-resolution proxy studies identifies 25% softmax as the optimal quality–efficiency trade-off, and the resulting 5B model achieves a VBench score of 84.30 at 480p in 13.2 seconds on a single H100, with a 3.2× faster DiT forward pass than a matched full-softmax baseline at 720p/60s and a 120× speedup over Wan 2.2-A14B after full-stack Sol-Engine optimization.

Introduction

Video generation relies on large diffusion transformers with full softmax attention, but their quadratic cost severely limits speed for long, high-resolution clips. Prior work either sticks with expensive softmax for accuracy or switches to pure linear attention for efficiency, sacrificing expressiveness in spatiotemporal detail. Recent language models have shown that a mostly linear stack with periodic softmax anchors and cross-layer residual routing can restore rank capacity, but adapting this hybrid design to the unique demands of video diffusion remains unexplored. The authors present SANA-Video 2.0, a hybrid-attention video diffusion transformer that replaces 75% of self-attention layers with gated bilinear linear attention, inserts 25% softmax anchors at regular depths, and routes refreshed representations across depth via block attention residuals, all trained from scratch. This design matches strong full-softmax baselines in quality while achieving large speedups that widen with video duration.

Dataset

The authors curate a video dataset from a large raw video corpus through a uniform processing pipeline, then partition it into training subsets with progressively stricter quality and motion criteria.

• Source and processing pipeline: The raw video corpus undergoes a six-stage pipeline that includes shot segmentation, removal of black bars and subtitles, and filtering of clips with low resolution, blur, static frames, or exposure defects. Quality and motion are scored along multiple axes rather than aggregated into a single metric; this prevents the selection from collapsing to only visually pristine but nearly static clips while still including high-quality low-motion content.

• Progressive quality/motion funnel: The processed clips are stratified into a funnel of subsets according to quality and motion thresholds:

  • Pre-training uses a broad pool with permissive thresholds to maximize coverage.
  • Continual training raises the quality and motion requirements, discarding lower-tier clips.
  • SFT (supervised fine-tuning) concentrates on a small trusted subset with the strictest quality and motion floors.

• Caption and metadata construction: Captions evolve alongside the subsets, moving from mixed short and long descriptions in earlier stages to dense, structured captions later. When motion descriptors are available, they are quantized and appended to the text, turning the motion selection signal into a conditioning cue.

• Usage in model training: The same quality and motion axes inform the pre-training timestep policy, which uses broad coverage initially and then focuses later training stages on fidelity and prompt adherence. The subsets are used sequentially: pre-training on the large permissive pool, continual training on the medium tiers, and SFT on the small high-quality subset, with captions adjusted at each stage.

Method

The authors present SANA-Video 2.0, a video Diffusion Transformer (DiT) that integrates hybrid sequence attention with block residual attention across depth. The model operates on latents from LTX-VAE 2.3 and incorporates text features from Gemma-2-2B-IT through cross-attention at every layer. The local processing path remains convolution-free throughout.

Hybrid Attention and Block Residuals The architecture employs a hybrid attention design, interleaving bidirectional gated linear attention layers with gated softmax anchors at a 3:1 ratio. This configuration maintains linear complexity for token mixing while using softmax anchors at every fourth layer to refresh interactions that the compressed linear state cannot fully represent. To further enhance information flow, the authors introduce Block Attention Residuals (AttnRes). The transformer layers are grouped into consecutive blocks, typically of 8 layers. As the forward pass progresses, sublayer outputs are accumulated into a running partial sum. Upon block completion, this sum is frozen as a block summary. A router at each layer aggregates the initial token embedding, completed block summaries, and the current partial sum to compute the representation.

To optimize memory, the model utilizes shared routing queries, specifically a single attention query and a single FFN query reused at every depth, rather than learning separate queries per layer.

Training Recipe and Data Curation The training process follows a comprehensive pipeline involving data curation, a resolution and duration curriculum, and preference-based post-training. The data curation applies a uniform processing pipeline including shot segmentation and cleanup, followed by multi-axis scoring for quality and motion. This creates a progressive quality and motion funnel where pre-training uses broad pools, and subsequent stages such as continual training and SFT focus on higher quality and motion thresholds.

Training utilizes flow matching with a logit-normal timestep density. The authors introduce Content-aware flow-shift (TQD) to address the motion-quality dilemma. TQD routes motion-rich clips to high-noise regimes, which govern global structure, and aesthetically clean, low-motion clips to low-noise regimes, which govern fine detail. In later stages, a token-count-aware flow shift is used, interpolating shift values based on latent token count to handle varying resolutions and durations continuously.

Post-Training and Alignment The post-training stage employs Diffusion-DPO for visual-preference alignment. Using rankings from a judge model, the authors form preferred-rejected pairs and optimize a loss that encourages the policy model to reduce the flow-matching error gap between preferred and rejected samples. Additionally, Online Reinforcement Learning via Reward Feedback Learning (ReFL) is utilized. This involves rolling out denoising trajectories and using frozen image reward models to provide frame-level feedback, optimizing the model without backpropagating through the full sampling trajectory.

System Design and Deployment For deployment, the authors utilize a Sol-Engine stack to optimize inference. The backbone is designed to be kernel-friendly, avoiding temporal-convolution FFNs to allow efficient mapping to standard attention kernels. The optimization stack includes kernel fusion, residual reuse which evaluates a subset of denoising steps, and sparse attention on softmax anchors.

The model also demonstrates capabilities in Physical AI scenarios. By fine-tuning on real-robot and egocentric videos, the model generates physically plausible manipulation videos, showing competitive performance against larger models in robotics tasks.

Experiment

Experiments validate the design choices and efficiency of SANA-Video 2.0. Timestep-stratified validation shows that a 25% softmax attention ratio provides a strong quality–speed trade-off, while a timestep-independent AttnRes router recovers representation rank and enables cross-depth reuse. Main VBench results confirm competitive quality with large speed advantages over similarly sized models, and online RL post-training improves all reward signals simultaneously. Deployment optimizations, including caching and low-precision inference, further reduce latency while maintaining generation quality.

The training curriculum progresses through three stages: pre-training on roughly 30 million clips at 480p, continual training on 10 million clips with ramping resolution and duration, and a compact SFT stage on only 10 thousand clips at 720p. Flow-shift adapts from a constant base with a content-aware modulation (TQD) in pre-training to a token-count-aware schedule in later stages, shifting probability mass toward high noise as clip length increases. Objectives shrink from TQD plus Self-Flow in early stages to standard flow matching in SFT, with a reduced learning rate at the final stage. Pre-training uses a fixed flow-shift of 1 with the TQD add-on, which offsets the timestep density to route high-motion clips to high noise and aesthetically clean clips to low noise. Continual training increases resolution from 480p to 720p and clip duration from 5 to 8 seconds, while flow-shift becomes token-count-aware and scales from 3 to 6, giving longer clips more high-noise probability. SFT operates on a drastically smaller set of only 10k clips at 720p, uses standard flow matching without TQD or Self-Flow, and applies a halved learning rate of 5×10^{-5}. Clip count falls by three orders of magnitude across stages, from ~30M in pre-training to ~10M in continual and ~10k in SFT, reflecting a move from broad structure learning to fine-quality refinement.

SANA-Video 2.0 (5B, 81 frames) achieves a VBench total of 84.30 with the highest quality score (85.61) among all compared models, surpassing Wan 2.1 14B and others. Only the much larger Bernini-R 14B edges ahead on total and semantic metrics, but SANA-Video 2.0 is over 30× faster on the same hardware. Extending to 121 frames lifts the total score to 85.29 while remaining far more efficient than competitors. SANA-Video 2.0's quality score of 85.61 leads all models in the VBench comparison, narrowly ahead of Wan 2.1 14B (85.59) despite using far fewer parameters. The 5B model generates an 81-frame video in 13.2 seconds on one H100, making it 31.8× faster than Bernini-R 14B which only slightly outperforms it on total and semantic scores.

Design probes for residual attention show that removing the timestep input from the router improves early short-run loss, and shared query projections match per-layer loss while slashing memory overhead. After continued training, keeping residual attention wins a majority of noise buckets by a narrow margin, and a block span of eight is selected as a balanced engineering default. Omitting the explicit timestep input from the router drops early training loss from 0.962 to 0.920. Shared attention queries achieve nearly identical loss to per-layer queries (0.495 vs. 0.496) while reducing memory overhead from +16% to +4%.

Applying the Sol-Engine deployment stack to SANA-Video 2.0 progressively reduces generation latency on both B200 and H100 GPUs. Kernel optimizations first halve the time, then diffusion caching with residual reuse cuts the number of denoising steps, and sparse attention on softmax anchors yields a final 3.58× speedup on B200 and 2.84× on H100 relative to each GPU’s own unoptimized baseline. Kernel optimization alone gives a 2.04× speedup on B200 and 1.60× on H100, the single largest latency reduction in the stack. Diffusion caching reduces the number of function evaluations from 50 to 33 and lifts the combined speedup to 3.00× on B200 and 2.37× on H100. Adding sparse attention further accelerates the pipeline, reaching a final latency of 17.52 s on B200 (3.58× baseline) and 33.43 s on H100 (2.84× baseline). The three stages are orthogonal: kernel optimization preserves exact computation, while caching and sparse attention introduce approximation for additional gains.

Quantization-aware training (QAT) with MXFP4 weights and MXFP8 activations preserves the BF16 baseline's overall quality, while post-training quantization (PTQ) introduces a minor drop. Both low-precision configurations reduce static model storage by about 68% and peak allocated memory by about 57%, but inference latency decreases by only 5.8% because only linear GEMM layers are quantized, leaving attention operators in full precision. QAT matches the BF16 baseline VBench Total score, whereas PTQ's score is slightly lower. Low-precision inference reduces static model size by 67.9% and peak memory by 56.9%. Latency improves by only 5.8% because linear GEMMs account for a small fraction of runtime, while attention operations stay in BF16.

The training curriculum for SANA-Video 2.0 employs a three-stage progression that shifts from broad structure learning with content-aware timestep modulation to fine quality refinement on a small high-resolution set, demonstrating that adaptive flow-shift and objective reduction effectively scale generation. Design probes on residual attention confirm that removing the timestep input from routers and sharing query projections reduces memory overhead without harming loss, leading to a practical block-span configuration. Benchmarking on VBench shows the 5B model achieves leading quality metrics and matches or surpasses much larger models, while generating videos over 30 times faster on the same hardware. The Sol-Engine deployment stack combines kernel optimizations, diffusion caching, and sparse attention to deliver up to a 3.58× cumulative speedup, and quantization-aware training with mixed precision preserves full-quality scores while cutting model storage and memory usage, though latency gains remain limited because attention operators stay in full precision.