One-sentence Summary

Researchers from Joy Future Academy, JD, introduce Self Gradient Forcing (SGF), a two-pass training strategy that closes the historical context-gradient gap in autoregressive video diffusion models by backpropagating future-frame losses to supervise context memory writing without full rollout gradients, yielding significantly improved long-video extrapolation to several minutes, subject identity consistency, background/layout consistency, and temporal stability from only 5-second training windows.

Key Contributions

• The paper identifies a historical context-gradient gap in Self Forcing, where future losses cannot supervise how self-generated video latents are encoded into the key-value cache for later frames.
• Self Gradient Forcing (SGF), a two-pass training strategy, decouples autoregressive rollout from parallel context-gradient reconstruction, allowing future losses to optimize the memory-writing of self-generated history without full rollout backpropagation.
• Experiments across frame-wise and chunk-wise generation, multiple initializations, and long horizons show that SGF, trained on a 5-second window, extrapolates to minutes-long videos and improves identity, layout consistency, and temporal stability over Self Forcing.

Introduction

Autoregressive video diffusion models generate long videos by conditioning each new chunk on previously generated content, but the standard teacher-forcing training uses ground-truth contexts, creating an exposure bias when the model must condition on its own outputs at inference. Self Forcing closes this mismatch by training on self-generated histories, yet it truncates gradient flow through the historical KV cache: future losses can supervise how later chunks read the cache, but cannot propagate back to the clean-timestep computation that wrote the self-generated frames into KV memory. This historical context-gradient gap leaves memory writing unsupervised, causing identity, layout, and temporal consistency to gradually degrade during long extrapolation. The authors propose Self Gradient Forcing (SGF), a two-pass training strategy that records the no-gradient serial rollout and then reconstructs the same computation in parallel, allowing gradients to flow through the clean-history KV representations and future-to-context attention. SGF restores the missing memory-writing supervision without requiring full backpropagation through the rollout, and it substantially improves native long-video extrapolation while matching short-horizon quality.

Method

The authors address a critical limitation in autoregressive video diffusion known as the historical context-gradient gap. In standard autoregressive generation, latent blocks are produced sequentially. At block $j$j, the causal generator denoises $z_j^t$zjt​ by attending to a historical K/V cache rather than raw past latents. After a block $i < j$i<j is generated, its predicted clean latent $\tilde{x}_i$x~i​ is processed at the clean context timestep $t_{\mathrm{ctx}} = 0$tctx​=0, and the resulting K/V entries are appended to the cache. The cache-writing computation is recurrent:

In frozen-cache Self Forcing, these historical K/V entries are treated as detached rollout state. Consequently, future losses do not supervise the $t_{\mathrm{ctx}} = 0$tctx​=0 computation that produced them, creating a missing supervision path for how self-generated latents are encoded into K/V memory. While keeping the serial historical K/V cache fully differentiable would theoretically close this gap, it is impractical for long-horizon self-rollout because the backward graph grows with rollout length, causing severe memory bottlenecks.

To resolve this, the authors introduce Self Gradient Forcing (SGF), which restores the missing memory-writing supervision without opening a recurrent autograd graph through the full self-rollout. SGF can be understood through an equivalent two-pass view of Self Forcing.

In the first pass, the model performs an ordinary no-gradient self-rollout and records a sampled exit state. In the second pass, SGF reconstructs the corresponding causal computation in parallel. Unlike frozen-cache Self Forcing, which treats the reconstructed context K/V path as detached memory, SGF removes the stop-gradient boundary on this path. The context latents $\tilde{X}_{\mathrm{ctx}}$X~ctx​ themselves remain stop-gradient inputs, ensuring the sampled rollout trajectory is not optimized. However, the model re-encodes these fixed self-generated latents at $t_{\mathrm{ctx}} = 0$tctx​=0, and the resulting K/V entries remain differentiable when future target tokens attend to them. This allows future Distribution Matching Distillation losses to supervise both target-side denoising and clean-context K/V writing:

The Pass-2 reconstruction is designed to recover the sampled exit computation from Pass 1. With deterministic layers, matched positional indices, and the same causal reconstruction geometry, the reconstructed target $\hat{X}_{\mathrm{tar}}$X^tar​ is theoretically identical to the recorded predicted context latents. This two-pass design avoids the memory blow-up of a direct differentiable cache, as Pass 1 is serial but no-gradient, while Pass 2 is gradient-enabled but fixed-window and parallel.

For frame-wise streaming generation, the authors employ a sink-plus-FIFO context policy. They maintain a fixed sink prefix and a FIFO window of recent latents to isolate the effect of SGF rather than context selection. For the video VAE, specific sink latents are used to preserve the temporal boundary prefix induced by its asymmetric grouping pattern.

Experiment

The evaluation trains all models on a 5-second window and tests native extrapolation to 60s and 240s, comparing SGF against matched Self Forcing baselines under identical prompts, seeds, and inference geometry. SGF consistently improves long-horizon quality and consistency, reducing drift artifacts such as identity shifts and background changes, and is clearly preferred by human raters. The training overhead is modest, confirming that learning to reconstruct self-generated context with gradients yields more stable long-video memory without sacrificing short-horizon performance.

Across 60-second and 240-second frame-wise generation, SGF consistently improves quality and consistency metrics over matched Self Forcing baselines, with especially clear gains in aesthetic quality, background consistency, imaging quality, motion smoothness, and subject consistency. Dynamic degree is the main exception, where Self Forcing can obtain a higher score, but this is linked to incoherent motion artifacts such as scene jumps, broken camera geometry, and object deformation rather than better motion quality. SGF yields higher aesthetic quality, background consistency, imaging quality, motion smoothness, and subject consistency across all initializations and both horizon lengths. Self Forcing often achieves higher dynamic degree, but the increase is attributed to disruptive artifacts like identity drift, crop drift, and layout changes that inflate the metric without improving perceived motion quality.

For chunk-wise 60-second video generation, SGF consistently improves over Self Forcing on most quality and consistency metrics, including aesthetic quality, background consistency, and subject consistency. The only exception is dynamic degree, where Self Forcing scores higher, but qualitative analysis attributes this to incoherent motion artifacts rather than genuine motion quality. SGF yields higher aesthetic quality, background consistency, and subject consistency than Self Forcing under TF initialization, with subject consistency rising from 0.951 to 0.982. Self Forcing achieves a higher dynamic degree (0.909 vs 0.634 under TF initialization), but this advantage is driven by scene jumps, camera breakage, and object deformation that inflate apparent motion.

A blind preference study shows that raters consistently prefer SGF over Self Forcing for long-horizon video generation, with all GSB scores positive. The advantage is most pronounced under TF initialization, where frame-wise generation at 240 seconds reaches the highest preference. Longer horizons generally strengthen the preference for SGF, reflecting its improved temporal stability. All GSB scores are positive, indicating a consistent preference for SGF across every setting and horizon tested. TF initialization yields the strongest preference for SGF, especially in frame-wise generation at the longer horizon. Longer horizons (240 seconds) tend to increase the preference margin compared to 60-second generations. Chunk-wise generation shows slightly lower preference scores than frame-wise generation for the same initializations, but the preference remains clearly positive.

Training with context-gradient recovery via SGF is feasible, unlike directly differentiating through the KV cache, which runs out of memory. SGF incurs a modest increase in peak memory and a slight runtime overhead, while actually reducing the stable memory footprint. The two-pass design isolates gradient computation, preventing the recurrent memory explosion of a differentiable cache. Enabling gradients through the entire historical KV cache exhausts memory, while SGF completes training successfully. Relative to frozen-cache Self Forcing, SGF raises peak memory by roughly 10% but reduces stable memory by about 19%. Wall-clock time per five training steps increases by approximately 13%, a modest overhead. SGF confines gradient computation to a parallel Pass-2 reconstruction after a no-gradient Pass-1 rollout, avoiding the serial cache-formation graph that causes out-of-memory in direct differentiable-cache training.

The evaluation compares SGF against Self Forcing across frame-wise and chunk-wise video generation at short and long horizons, using both automated quality metrics and blind human preference. SGF consistently yields higher aesthetic quality, background consistency, and subject consistency, and is preferred by raters, with the advantage growing at longer horizons. The only metric where Self Forcing sometimes scores higher, dynamic degree, is driven by incoherent artifacts like scene jumps and object deformation rather than genuine motion quality, while SGF’s training approach enables gradient-based context recovery with modest memory and time overhead, avoiding the memory explosion of a fully differentiable cache.