Command Palette
Search for a command to run...
minWM: 실시간 상호작용 비디오 월드 모델을 위한 풀스택 오픈소스 프레임워크
minWM: 실시간 상호작용 비디오 월드 모델을 위한 풀스택 오픈소스 프레임워크
초록
최근 비디오 디퓨전 파운데이션 모델은 고품질 비디오 생성 분야에서 눈부신 진전을 이루었으나, 이를 실시간 상호작용형 비디오 월드 모델로 전환하는 것은 여전히 과제로 남아있다. 상호작용형 월드 모델은 제어 가능하고 인과적이며 저지연의 롤아웃을 필요로 하며, 이는 실제로 데이터 구축, 제어 가능한 파인튜닝, 자기회귀 학습, 소수 단계 증류, 스트리밍 추론을 아우르는 전체 파이프라인을 요구한다. 본 연구에서는 실시간 상호작용형 비디오 월드 모델을 구축하기 위한 풀스택 오픈소스 프레임워크인 minWM을 제시한다. minWM은 기존 양방향 T2V/TI2V 비디오 파운데이션 모델을 카메라 제어 가능한 소수 단계 자기회귀 월드 모델로 변환하는 엔드투엔드 파이프라인을 제공한다. 구체적으로, minWM은 먼저 카메라 제어가 가능한 양방향 비디오 디퓨전 모델을 파인튜닝한 후, AR 디퓨전 학습, 인과적 ODE 또는 인과적 일관성 증류, 비대칭 DMD를 포함한 Causal Forcing / Causal Forcing++ 파이프라인을 적용하여 저지연 롤아웃을 위한 소수 단계 자기회귀 생성기로 증류한다. 본 프레임워크는 모듈식이며 아키텍처 확장성이 뛰어나다: 우리는 Wan2.1-T2V-1.3B 및 HY1.5-TI2V-8B를 포함한 대표적인 오픈 백본에서 이를 구현하며, 크로스 어텐션 기반 조건 주입과 MMDiT 스타일 아키텍처를 모두 포괄한다. minWM은 HY-WorldPlay와 같은 기존 비디오 월드 모델을 새로운 데이터 분포, 학습 레시피, 지연 시간 목표에 맞게 적응시키는 것도 지원한다. 실행 가능한 스크립트, 체크포인트, 문서화, 추론 코드의 공개를 넘어, 우리는 카메라 궤적 품질, 제어 가능성 학습 단계, 최소 배치 크기 요구 사항에 대한 실용적인 아블레이션 연구를 제공한다. minWM이 실시간 상호작용형 비디오 월드 모델을 구축하고 적응시키는 데 재현 가능하고 확장 가능한 레시피로 활용되기를 바란다. 프로젝트 페이지: https://github.com/shengshu-ai/minWM
One-sentence Summary
minWM is a modular full-stack open-source framework that converts bidirectional video diffusion models into camera-controllable, few-step autoregressive world models via a pipeline combining camera-conditioned fine-tuning, causal forcing, causal consistency distillation, and asymmetric DMD to enable low-latency interactive generation on architectures such as Wan2.1-T2V-1.3B and HY1.5-TI2V-8B.
Key Contributions
- This work introduces minWM, a full-stack open-source framework that provides a modular, end-to-end pipeline for converting existing bidirectional text-to-video and text-and-image-to-video foundation models into camera-controllable autoregressive world models. The framework unifies data construction, controllable fine-tuning, autoregressive training, few-step distillation, and streaming inference into a reproducible workflow.
- The framework executes a two-phase conversion recipe that first fine-tunes a bidirectional diffusion backbone on camera-annotated data to enable trajectory control. It subsequently applies Causal Forcing or Causal Forcing++ pipelines, combining autoregressive diffusion training, causal ODE or consistency distillation, and asymmetric DMD post-training to distill the model into a few-step autoregressive generator for low-latency rollout.
- The pipeline is instantiated on Wan2.1-T2V-1.3B and HY1.5-TI2V-8B backbones to demonstrate real-time interactive video generation across cross-attention and MMDiT architectures. By releasing intermediate checkpoints for each training stage and providing ablation studies on camera trajectory quality and training configurations, the framework offers actionable guidance for reproducible world model development.
Introduction
High-quality diffusion-based video foundation models have significantly advanced visual generation, yet they function as offline generators rather than interactive world models. Real-time interactive applications demand causal rollout, responsive camera control, and low-latency frame synthesis, but existing conversion techniques remain scattered across disconnected pipelines that require extensive manual effort across data preparation, fine-tuning, autoregressive training, and distillation. To bridge this gap, the authors introduce minWM, a full-stack open-source framework that unifies the entire workflow into a single reproducible pipeline. The authors leverage a two-phase strategy that first fine-tunes bidirectional video backbones for camera controllability and then applies causal forcing alongside asymmetric distillation to convert them into few-step autoregressive generators. This modular architecture enables researchers to seamlessly adapt existing foundation models into real-time, camera-controlled video world models while supporting mid-pipeline checkpointing and customizable training configurations.
Method
The authors present minWM, a full-stack framework for constructing real-time interactive video world models, which operates through a two-phase pipeline. The overall architecture begins with data preparation, proceeds through a training phase that includes bidirectional diffusion fine-tuning and autoregressive distillation, and culminates in low-latency inference. The process starts with inputs comprising an image, text, and an action signal, which are processed through data filtering and rebalancing, followed by structured annotation to generate a dataset suitable for training. This dataset is then used to fine-tune a bidirectional diffusion model with camera controllability, leveraging PROPE as the injection method for camera parameters. The framework supports multiple backbones, including Wan2.1-T2V-1.3B and HY1.5-TI2V-8B, enabling the conversion of existing video foundation models into camera-controllable few-step autoregressive generators.
As shown in the figure below, the training phase is divided into two stages. Phase 1 involves bidirectional diffusion training on the fine-tuned model, where camera control is integrated via PROPE. This phase equips the model with the ability to condition on camera trajectories while maintaining the original self-attention structure. Phase 2, AR Diffusion Distillation, applies the Causal Forcing or Causal Forcing++ pipeline to transform the bidirectional model into a real-time interactive autoregressive model. This includes AR diffusion training, causal ODE or causal consistency distillation initialization, and asymmetric DMD post-training. The final stage, inference, enables streaming VAE decoding and prompt engineering, resulting in real-time interactive video outputs. The framework is modular and extensible, supporting the adaptation of existing models to new data distributions and latency targets.
Experiment
The evaluation trains two base video generation models using an autoregressive framework with few-step distillation to assess inference efficiency and camera control. Qualitative results demonstrate that the approach significantly reduces first-frame latency, enabling seamless playback during generation, while successfully preserving camera-controllable capabilities. Ablation studies further reveal that robust camera control depends on high-quality ground-truth trajectory data, sufficient training iterations, and a minimum batch size to ensure stable optimization.
The authors compare the first-frame latency and speedup of multi-step bidirectional and few-step autoregressive models based on two different base models. Results show that the few-step AR model significantly reduces first-frame latency and achieves substantial speedup over the multi-step bidirectional baseline, while maintaining camera-controllable generation capabilities. Few-step AR models achieve substantial reductions in first-frame latency compared to multi-step bidirectional models. The few-step AR model provides significant speedup over the multi-step bidirectional baseline for both base models. Camera-controllable generation is preserved in the few-step AR models despite the latency improvements.
The evaluation compares the initial generation latency and processing speed of multi-step bidirectional models against few-step autoregressive architectures across two distinct base models. Results demonstrate that the few-step autoregressive approach substantially accelerates first-frame generation and overall inference efficiency. Importantly, these performance improvements are achieved while successfully preserving the models' camera-controllable generation capabilities.