One-sentence Summary

Researchers from KAIST AI, Inha University, and FLIPTION propose OmniShowcase, a multi-view subject-to-video framework combining a fully automated 3D-asset-based data synthesis pipeline for diverse camera trajectories, including full 360-degree orbits, with surface-normal geometry conditioning reconstructed from reference views, and introduce SHOWCASE-4K and SHOWCASEBENCH while achieving state-of-the-art subject identity and 3D consistency.

Key Contributions

• OmniShowcase includes a fully automated data synthesis pipeline that renders training videos from 3D assets along diverse camera trajectories, including full 360-degree orbits. This pipeline yields SHOWCASE-4K, a corpus covering twelve camera motions and 88 trajectory configurations with reconstruction-derived surface-normal conditioning.
• The framework conditions a pretrained image-to-video diffusion transformer on a turntable of surface-normal maps reconstructed from the reference views, supplying explicit 3D geometry without temporal alignment with generated frames.
• SHOWCASEBENCH provides a held-out evaluation set with source meshes and camera parameters; OmniShowcase achieves state-of-the-art subject identity preservation and 3D consistency while maintaining competitive video quality and motion on SHOWCASEBENCH and NAVI.

Introduction

Subject-to-video generation aims to create videos of a specified subject from reference images while preserving its appearance across time and viewpoints. Recent video diffusion models can produce photorealistic clips, but most subject-to-video methods condition on a single reference image, which constrains only visible surfaces and leaves unseen regions undefined. As the camera moves, this often causes inconsistent geometry, changing silhouettes, or unstable surface details. The core challenge is geometric: the model must infer 3D structure that is absent from the input. Existing multi-view approaches either use generated multi-view videos or per-subject optimization with static images, but they do not explicitly encode 3D geometry or avoid test-time adaptation. The authors introduce OmniShowcase, which combines an automated pipeline for rendering multi-view videos from 3D assets with geometry-aware conditioning. It reconstructs surface-normal sequences from sparse reference views and uses them as in-context cues during generation, improving subject identity and cross-view 3D consistency without requiring frame-wise alignment or per-subject optimization.

Dataset

Dataset description:

The authors construct SHOWCASE-4K as a synthetic, 3D-consistent video dataset with paired geometry conditions. The data pipeline has two branches: one for target video generation and one for geometry-condition generation. The key design is to render from known 3D meshes so that all views share one underlying shape and appearance, then composite realistic backgrounds.

Sources and filtering:

• 3D meshes are sourced from Objaverse, Digital Twin Catalog, and Google Scanned Objects.
• A heuristic pre-screen removes degenerate meshes.
• A vision-language model inspects four rendered views per asset and keeps only assets with valid geometry, plausible textures, and category consistency.
• Exact per-source counts are not reported in this section.

Target video construction:

• Blender renders an 81-frame target video and N=4 reference views per mesh at 90-degree yaw intervals.
• The synthesis uses 88 trajectory configurations: four orbit-family motions with four rotation amounts each, plus eight additional motions such as arc, truck, dolly, and zoom variants with nine framing transitions each.
• To reduce the domain gap from clean-background renders, subject masks from the render alpha channel drive a video inpainting model that synthesizes a coherent background while preserving rendered subject pixels.
• Scene prompts combine a location from a language-model-written scene bank with the sample camera motion and lighting.
• A vision-language gate discards videos where the subject floats, deforms, or departs from the references.
• The target is trimmed to 80 frames to satisfy the video VAE temporal constraint and sampled at 16 fps.

Geometry condition construction:

• The geometry condition is reconstructed from the four reference views rather than taken from the source mesh, so training and inference conditioning signals match.
• The pipeline masks the subject on a white background, predicts per-view pointmaps with a feedforward multi-view estimator, and fuses the views into 3D Gaussians with a multi-view reconstruction module.
• The Gaussians are rendered at 25 uniformly spaced yaw poses at 15 degrees elevation, and each rendered depth map is converted to surface normals.
• A vision-language gate discards reconstructions that depart from the reference or break apart.

Sample schema and usage:

• Each training sample contains a 25-frame normal turntable condition, an 80-frame target video, and four reference images.
• A video-language model, Tarsier2-7B, captions the normal turntable and target video separately.
• The normal and target segments are encoded separately and then concatenated along the temporal axis in latent space; each reference image is encoded independently.
• The condition and video segments are paired with explicit [CONDITION] and [VIDEO] markers.
• The text does not specify a train/validation split or mixture ratios across source meshes.

Method

The authors propose a method to generate camera-controllable videos of a subject from a small set of reference views while preserving its 3D structure and appearance. Given a set of reference views $\mathcal{R} = \{\mathbf{I}_i\}_{i=1}^N$R={Ii​}i=1N​ (where $N=4$N=4), the system reconstructs the subject in 3D and renders the geometry as a turntable of surface-normal maps. These normal maps, along with the reference images, are provided to a pretrained video diffusion model to generate the target video.

The backbone architecture builds upon a pretrained image-to-video diffusion transformer adapted with low-rank adaptation tuning, keeping the base weights frozen. The model operates in the latent space of a causal video VAE and is trained with a rectified-flow objective. To ensure the training data satisfies subject identity preservation, geometrically consistent viewpoint changes, and explicit geometry for conditioning, the authors design an automated 3D-consistent data synthesis pipeline. Unlike standard camera-controllable generators that synthesize unseen surfaces from priors, often leading to shape changes or subject duplication, the proposed approach renders the subject from a 3D asset and generates only the background.

The data synthesis pipeline constructs the target video and geometry conditions. From a 3D mesh, the system renders an 81-frame target video and $N=4$N=4 reference views at $90^\circ$90∘ yaw intervals. To close the domain gap from clean-background renders, the subject is composited into a prompt-driven scene using a video inpainting model, while a vision-language gate discards videos where the subject deforms or departs from the references. Crucially, the geometry condition is reconstructed from the four reference views rather than using the source mesh directly, ensuring the conditioning signal matches at inference. The system masks the subject, predicts aligned per-view pointmaps, and fuses them into 3D Gaussians. These are rendered along a yaw orbit to produce a 25-frame normal turntable $\{\mathbf{N}_t\}_{t=1}^{T_c}${Nt​}t=1Tc​​.

For geometry-aware conditioning, the sequence concatenates the $N=4$N=4 reference images, the $T_c=25$Tc​=25-frame normal turntable, and the target video frames along the temporal axis. The reference images are encoded independently, while the normal maps form a temporal sequence encoded using the video VAE. The concatenated latent sequence is defined as:

$$\mathbf{z} = [\mathbf{z}^{\mathcal{R}}; \mathbf{z}^{\mathcal{N}}; \mathbf{z}^{\mathcal{V}}] \in \mathbb{R}^{L \times C \times h \times w}, \quad L = N + T_n + T_v$$z=[zR;zN;zV]∈RL×C×h×w,L=N+Tn​+Tv​

Encoding the 25-frame normal condition and 80-frame target separately yields $T_n=7$Tn​=7 and $T_v=20$Tv​=20 latent frames, resulting in a sequence length of $L=31$L=31. A binary mask $\mathbf{m} \in \{0, 1\}^L$m∈{0,1}L marks the first $N + T_n$N+Tn​ positions as given. The transformer input stacks three streams along the channel axis:

$$\widetilde{\mathbf{z}}_\tau = \mathbf{m} \odot \mathbf{z} + (1 - \mathbf{m}) \odot (\tau \mathbf{z} + (1 - \tau)\boldsymbol{\epsilon}), \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$zτ​=m⊙z+(1−m)⊙(τz+(1−τ)ϵ),ϵ∼N(0,I) $$\mathbf{x}_\tau = \text{Concat}_{\text{channel}}(\widetilde{\mathbf{z}}_\tau, \mathbf{M}, \mathbf{m} \odot \mathbf{z})$$xτ​=Concatchannel​(zτ​,M,m⊙z)

This ensures the reference and normal latents remain clean, while only the target-video latents are interpolated with noise. No frame-wise correspondence is imposed between the normal and target sequences, allowing the turntable to serve as a trajectory-independent geometry cue.

During training, the authors optimize only the LoRA parameters $\theta$θ using the rectified-flow objective. Let $\mathbf{c}$c denote the text condition and $\mathcal{F}_\theta^\nu$Fθν​ the target-video segment of the transformer output. The target velocity under the backbone flow convention is $\mathbf{z}^\nu - \boldsymbol{\epsilon}^\nu$zν−ϵν. The loss is minimized as:

$$\mathcal{L} = \mathbb{E} \left[ \frac{1}{D_v} \left\| \widehat{\mathbf{u}}_\theta^\nu - (\mathbf{z}^\nu - \boldsymbol{\epsilon}^\nu) \right\|_2^2 \right]$$L=E[Dv​1​∥uθν​−(zν−ϵν)∥22​]

where $D_v$Dv​ is the number of scalar elements in $\mathbf{z}^\nu$zν, and the squared error is averaged over target-video latent elements only. At inference, the system reconstructs the geometry from a new subject reference views, renders the yaw orbit to derive the surface-normal turntable, and conditions the generator to produce the target latents. The normal-condition and generated target latents are concatenated and jointly decoded by the causal VAE to produce the final video frames.

Experiment

The evaluation uses held-out SHOWCASEBENCH and NAVI benchmarks to compare multi-view subject-to-video generation against Phantom-MV, MAGREF-MV, MV-S2V, and 3DreamBooth on identity, 3D consistency, text alignment, and video quality metrics. Qualitative and quantitative results show that baselines struggle to preserve camera trajectory, identity, and 3D consistency, while OmniShowcase achieves the best identity and geometry scores with competitive text and video quality. Ablations confirm that geometry conditioning is essential, since removing it or replacing it with textured renders weakens identity and 3D consistency, and thinning the turntable hurts coverage and reconstruction reliability.

OmniShowcase is the only compared subject-to-video method that combines generalization without optimization for inference with explicit geometric conditioning. It uses rendered 3D asset supervision and multi-view references, whereas prior methods rely on real video, generated video, or static multi-view images and lack geometric conditioning. The method also operates with substantially less training data than prior multi-view baselines. OmniShowcase is the only listed approach with both generalization without inference-time optimization and explicit geometric conditioning. Its training signal comes from rendered 3D assets, unlike other methods trained on real video, generated video, or static multi-view captures. It uses fewer training samples and fewer reference views than the static multi-view baseline while remaining generalizable.

OmniShowcase achieves the strongest identity preservation and 3D consistency among compared methods, with the best DINO, CLIP, MEt3R, and GT-CD results on both reported benchmarks. Phantom-MV and MAGREF-MV trail on identity and camera trajectory coverage, while MV-S2V and 3DreamBooth trade advantages across other axes. OmniShowcase keeps text alignment and video quality competitive even where it does not lead. OmniShowcase records the highest DINO and CLIP identity scores and the lowest MEt3R and GT-CD errors on the held-out benchmark. Phantom-MV and MAGREF-MV show weaker identity preservation and worse RotCoverage than stronger baselines, consistent with qualitative observations that they barely orbit the subject. MV-S2V and 3DreamBooth each lead on different metrics, and no baseline is strongest on every axis. OmniShowcase maintains competitive ViCLIP text alignment and video quality while improving identity and 3D consistency.

The full surface-normal geometry conditioning leads identity, 3D consistency, geometric accuracy, and viewpoint coverage on both evaluation sets. Removing geometry, replacing normals with textured renders, or thinning the turntable all reduce identity and geometric quality; on the held-out set, textured geometry gives the lowest geometric error among the ablations while sparse views give the best viewpoint coverage, and on the NAVI set removing geometry gives the highest geometric error. The full surface-normal conditioning outperforms the ablated variants on identity and 3D consistency metrics across both evaluation sets. Ablations show distinct trade-offs: removing geometry hurts identity and geometric accuracy most, textured geometry weakens text alignment most on the held-out set, and sparse views reduce viewpoint coverage relative to the full conditioning.

The first comparison establishes OmniShowcase as the only subject-to-video method that is generalizable without inference-time optimization while using explicit geometric conditioning and rendered 3D asset supervision, trained on less data and fewer reference views than prior approaches. The main benchmark evaluation validates that this design yields the strongest identity preservation and 3D consistency, with competitive text alignment and video quality, while baseline methods show weaker subject orbiting or trade off different strengths. Ablations confirm that removing or altering the full surface-normal geometry conditioning, including replacing normals with textured renders or thinning the turntable, generally degrades identity, geometric accuracy, and viewpoint coverage, though each ablation shows slightly different trade-offs.