One-sentence Summary

LongE2V, proposed by researchers at National Yang Ming Chiao Tung University, jointly addresses event-based video reconstruction, prediction, and frame interpolation by fine-tuning pre-trained video diffusion priors, and introduces Autoregressive Unrolling, Adaptive Context Switching, Reencoding Alignment with Cross Residual Correction, and Event Voxel Density Augmentation to mitigate temporal drift and ensure bidirectional consistency, thereby achieving state-of-the-art performance and zero-shot generalization on real-world benchmarks.

Key Contributions

• LongE2V fine-tunes a pre-trained video diffusion model to jointly handle event-based video reconstruction, prediction, and frame interpolation, achieving high perceptual quality and data efficiency on real-world benchmarks.
• Autoregressive Unrolling and Adaptive Context Switching are introduced to mitigate temporal drift in extremely long sequences by dynamically updating the generation context instead of relying on fixed schedules.
• For frame interpolation, Reencoding Alignment with Cross Residual Correction resolves temporal misalignment in the 3D VAE, ensuring precise bidirectional consistency between generated frames.

Introduction

Event cameras capture high-speed dynamics with microsecond resolution and high dynamic range, but their sparse, intensity-free output makes video reconstruction an ill-posed problem. Prior methods often produce blurry textures, drift during long-term prediction, or ghosting artifacts in interpolation, and they typically require separate architectures for each task. The authors introduce LongE2V, a unified framework that leverages pre-trained video diffusion models to perform event-based reconstruction, prediction, and frame interpolation. To ensure stability, they propose autoregressive unrolling with adaptive context switching; for interpolation, reencoding alignment and cross residual correction resolve temporal misalignments. Event voxel density augmentation further enables robust generalization across sensor resolutions, outperforming state-of-the-art methods in perceptual quality and zero-shot transfer.

Method

The authors leverage CogVideoX I2V as the base video diffusion model, which encodes video into latents $Z_0$Z0​ using a 3D VAE with 4x temporal and 8x spatial compression. The Diffusion Transformer (DiT) is trained to minimize the denoising objective: $\mathcal{L} = \mathbb{E}_{Z_0, t, C, \epsilon} \left[ \|\epsilon - \epsilon_\theta(Z_t, t, C)\|_2^2 \right]$L=EZ0​,t,C,ϵ​[∥ϵ−ϵθ​(Zt​,t,C)∥22​] where $Z_t$Zt​ are the noisy latents and $C$C denotes conditioning signals. During inference, the model iteratively denoises random Gaussian noise to synthesize clean latents, which are decoded back to pixel space. For event representation, event cameras capture asynchronous streams $\{e_i\}_{i=1}^N${ei​}i=1N​ where $e_i = (x_i, y_i, t_i, p_i)$ei​=(xi​,yi​,ti​,pi​). To enable frame-based processing, these events are discretized into a voxel grid $V \in \mathbb{R}^{B \times H \times W}$V∈RB×H×W by accumulating polarity $p_i \in \{\pm 1\}$pi​∈{±1} into $B$B temporal bins via linear interpolation: $V(t, y, x) = \sum_{i} p_i \max(0, 1 - |t - t_i^*|) \delta(x - x_i, y - y_i)$V(t,y,x)=∑i​pi​max(0,1−∣t−ti∗​∣)δ(x−xi​,y−yi​) where $\delta$δ is the Kronecker delta and $t_i^* \in [0, B-1]$ti∗​∈[0,B−1] represents the normalized timestamp relative to duration $\Delta t$Δt. In experiments, $B=3$B=3.

To align asynchronous events with discrete frames $X = \{x_0, \ldots, x_{F-1}\}$X={x0​,…,xF−1​}, the event stream is partitioned into time windows. The event voxel $v_k$vk​ corresponds to frame $x_k$xk​ at timestamp $t_k$tk​ by aggregating events within the interval $[t_{k-1}, t_k)$[tk−1​,tk​). This yields a synchronized event sequence $V = \{v_1, \dots, v_{F-1}\}$V={v1​,…,vF−1​}.

To mitigate error accumulation and artifact propagation in long video generation, the authors employ context frames and context event voxels as history conditions. However, conditioning on history introduces a domain gap between training and inference. To bridge this gap, an Autoregressive Unrolling training strategy is proposed. Initially, the model is trained with Ground Truth context frames until convergence. Subsequently, the unrolling mechanism is activated where predictions generated on the training set are substituted as context frames for subsequent finetuning. This iterative unrolling forces the model to adapt to its own prediction errors and effectively aligns the training distribution with inference behavior.

To enhance robustness against sensor variations, an Event Voxel Density Augmentation strategy is introduced. Event voxels are randomly resized while preserving the aspect ratio, bounded within a dynamic range $[S_{min}, S_{max}]$[Smin​,Smax​]. The lower bound $S_{min}$Smin​ is set slightly larger than the network input to facilitate random cropping, while $S_{max}$Smax​ is capped at 2x the original resolution. Normalization is based on the statistics of non-zero values to preserve sparsity. Identical geometric transformations are synchronously applied to the first frame, context frames, and current video frames to maintain spatial alignment.

Following spatial and temporal alignment, all inputs are encoded via a frozen 3D VAE. The final input latents are constructed by concatenating three temporally-aligned sequences along the channel dimension: (1) context latents with the zero-padded first-frame latent $Z_{x_0}$Zx0​​, (2) context latents with the noise latents $Z_t$Zt​, and (3) context event latents with the current event latents. To accommodate the additional event condition, the first projection layer within the patchify module is augmented, extending the weights $\mathbf{W}_{in} \in \mathbb{R}^{D \times 2C_{vae} \times K \times K}$Win​∈RD×2Cvae​×K×K to $\mathbf{W}_{in}^* \in \mathbb{R}^{D \times 3C_{vae} \times K \times K}$Win∗​∈RD×3Cvae​×K×K. During training, this expanded layer is fully fine-tuned, and LoRA is employed for the DiT blocks. The loss is calculated exclusively on the $Z_t$Zt​ component.

The authors formulate event-based video reconstruction, prediction, and frame interpolation as tasks of event-based video generation by adjusting input conditions. For Video Reconstruction, the model recovers photometric details solely from events. For Video Prediction, given the start frame, event stream, and context, the model generates subsequent video frames. For Video Frame Interpolation, given the start and end frames as boundary conditions, the model synthesizes intermediate frames by leveraging both forward and backward event streams.

To mitigate error accumulation during autoregressive generation, an Adaptive Context Switch is introduced. The denoised latents $\hat{Z}_0$Z^0​ are re-input into the DiT to extract the attention map $A$A and compute the average attention weight $\mu_{attn}$μattn​ of current tokens attending to context tokens: $\mu_{attn} = \frac{1}{L \times H \times N_{curr}} \sum_{l=1}^{L} \sum_{h=1}^{H} \sum_{i \in \text{Current}} \sum_{j \in \text{Context}} A_{i,j}^{(l,h)}$μattn​=L×H×Ncurr​1​∑l=1L​∑h=1H​∑i∈Current​∑j∈Context​Ai,j(l,h)​ where $L$L is the number of layers, $H$H is the number of Attention Heads, and $N_{curr}$Ncurr​ is the number of Current Tokens. If $\mu_{attn} \ge \tau$μattn​≥τ, the existing context is retained. If $\mu_{attn} < \tau$μattn​<τ, indicating low relevance, a Context Switch is triggered: the current generation is discarded, the context is updated to the immediate predecessor, and the chunk is regenerated.

For zero-shot interpolation, Reencoding Alignment is proposed to rectify temporal misalignment caused by the non-commutativity of latent and pixel space operations in 3D VAEs. The predicted clean latents $\hat{Z}_0$Z^0​ are decoded, flipped in pixel space, and re-encoded: $\tilde{Z}_0^{fwd} = \mathcal{E}(\text{Flip}_{pix}(\mathcal{D}(\hat{Z}_0^{fwd}))), \quad \tilde{Z}_0^{bwd} = \mathcal{E}(\text{Flip}_{pix}(\mathcal{D}(\hat{Z}_0^{bwd})))$Z~0fwd​=E(Flippix​(D(Z^0fwd​))),Z~0bwd​=E(Flippix​(D(Z^0bwd​))) This ensures precise temporal alignment between the bidirectional branches. To compensate for reconstruction loss induced by the extra Decode-Encode loop, Cross Residual Correction is introduced. The residual between the original $\hat{Z}_0$Z^0​ and re-encoded $\bar{Z}_0 = \mathcal{E}(\mathcal{D}(\hat{Z}_0))$Zˉ0​=E(D(Z^0​)) latents is computed: $\Delta \hat{Z}_0^{fwd} = \hat{Z}_0^{fwd} - \bar{Z}_0^{fwd}, \quad \Delta \hat{Z}_0^{bwd} = \hat{Z}_0^{bwd} - \bar{Z}_0^{bwd}$ΔZ^0fwd​=Z^0fwd​−Zˉ0fwd​,ΔZ^0bwd​=Z^0bwd​−Zˉ0bwd​ The opposing residual is then injected into the aligned latents: $Z_0^{\prime bwd} = \tilde{Z}_0^{bwd} + \Delta \hat{Z}_0^{fwd}, \quad Z_0^{\prime fwd} = \tilde{Z}_0^{fwd} + \Delta \hat{Z}_0^{bwd}$Z0′bwd​=Z~0bwd​+ΔZ^0fwd​,Z0′fwd​=Z~0fwd​+ΔZ^0bwd​ This recovers high-frequency details and promotes Temporal Consensus. Finally, the corrected latents are fused via alpha blending and re-noised for the subsequent loop.

Experiment

The model is trained on the BS-ERGB dataset and evaluated on real-world event sequences from ECD, MVSEC, and HQF for reconstruction and prediction, and on BS-ERGB and HQF for frame interpolation. Comparisons show that it consistently outperforms prior state-of-the-art methods in both short- and long-term reconstruction and prediction, and even surpasses dedicated interpolation models in a zero-shot setting, producing sharper and more coherent outputs. Ablation studies confirm the importance of the pretrained diffusion prior, context mechanisms, and autoregressive training for stability, while for interpolation, reencoding alignment, cross residual correction, and event voxel augmentation are essential to avoid artifacts. Additionally, the model can leverage its text-guided generation backbone to perform event video colorization from textual prompts.

The proposed method achieves the best perceptual quality in reconstruction, attaining the lowest LPIPS scores across all three datasets and outperforming regression-based baselines. For prediction, it significantly surpasses VDM-EVFI on every metric and dataset, demonstrating robust long-term generation capabilities. These findings are consistent on ECD, MVSEC, and HQF. The method consistently yields the lowest LPIPS in reconstruction, indicating superior perceptual quality compared to regression-based approaches. It significantly outperforms VDM-EVFI across all metrics and datasets in the prediction task, validating its long-term generation strength.

A zero-shot generative method for event-based video frame interpolation surpasses dedicated supervised baselines in perceptual quality and generalization. On the HQF dataset it achieves the best PSNR, SSIM, and LPIPS, while on ERGB it delivers comparable fidelity metrics with substantially sharper textures. Unlike regression approaches that sacrifice detail for higher PSNR, this method preserves high-frequency structures and avoids ghosting or misalignment. The zero-shot model achieves the lowest LPIPS on both datasets, and on HQF it also attains the highest PSNR and SSIM, demonstrating strong generalization without fine-tuning. Qualitative comparisons show the method reconstructs clear, readable text and maintains structural integrity, whereas supervised baselines produce blur, ghosting, and color artifacts.

Omitting the pretrained prior prevents convergence and leads to extremely low reconstruction quality. Adding context, autoregressive unrolling, and an adaptive context switch progressively reduces error accumulation, with the full configuration attaining the highest fidelity across all metrics. Without a pretrained prior, the model fails to converge and produces near-noise outputs. Incorporating context and autoregressive unrolling substantially improves PSNR, SSIM, and LPIPS. The adaptive context switch provides a final refinement, yielding the best scores on all metrics.

Removing Reencoding Alignment causes severe blur, confirming its role in structural fidelity. Cross Residual Correction markedly improves perceptual quality, lowering LPIPS by 0.037, while Event Voxel Density Augmentation increases robustness to density mismatch. The full combination of all three components delivers the highest PSNR, SSIM, and lowest LPIPS. Enabling Cross Residual Correction alongside Reencoding Alignment substantially increases PSNR and reduces LPIPS, mitigating ghosting and semi-transparent artifacts. Adding Event Voxel Density Augmentation yields the lowest LPIPS (0.124) and prevents color deviations and black stripe artifacts caused by density mismatch.

The evaluation covers event-based video reconstruction and prediction across multiple datasets, comparing the proposed method against regression-based and dedicated supervised baselines. The method consistently achieves superior perceptual quality, with its zero-shot generative variant generalizing strongly to produce sharp, artifact-free outputs. Ablation studies confirm that a pretrained prior is essential for convergence, while autoregressive unrolling, adaptive context switching, and alignment corrections progressively reduce error accumulation and structural distortions, yielding the highest fidelity in long-term generation.