One-sentence Summary

Researchers from CUHK, Shanghai AI Laboratory, and SenseTime introduce SenseNova-U1, a native unified multimodal paradigm built on NEO-unify that treats understanding and generation as synergistic views of one process, with $8\text{B}$8B dense and $30\text{B-A}3\text{B}$30B-A3B MoE variants rivaling top VLMs across understanding, X2I synthesis, VLA, and world-model tasks.

Key Contributions

• Introduces SenseNova-U1, a native unified multimodal paradigm built on NEO-unify, with two released variants (SenseNova-U1-8B-MoT and SenseNova-U1-A3B-MoT) that treat understanding and generation as synergistic views of a single underlying process.
• Demonstrates that the single native architecture rivals top-tier understanding-only VLMs across text understanding, vision–language perception, knowledge reasoning, agentic decision-making, and spatial intelligence, while achieving strong semantic consistency and visual fidelity in X2I synthesis, text-rich infographic generation, and interleaved vision–language generation.
• Shows the model extends into vision–language–action (VLA) and world model (WM) scenarios, and provides detailed model design, data preprocessing, pre-/post-training, and inference strategies to support community replication.

Introduction

Multimodal foundation models have historically kept perception and generation separate due to divergent system designs: understanding relies on pretrained vision encoders (VEs), while generation depends on latent variational autoencoders (VAEs). These choices create distinct learning objectives and feature representations, forcing early unified multimodal models (UMMs) to connect perception and generation through different tokenizers, latent spaces, or auxiliary modules rather than learning them jointly. Native vision-language models (VLMs) have attempted to bridge this gap, but discrete tokenization compresses non-linguistic signals into lossy representations, while continuous visual interfaces often trade off semantic abstraction against pixel-level fidelity, leaving a fundamental tension unresolved.

The authors address this by introducing SenseNova-U1, a native unified multimodal paradigm built on the NEO-unify model. They dispense with both pretrained vision encoders and deep decoder heads, learning directly from lossless inputs such as pixels and words. The model incorporates a near-lossless visual interface that preserves semantic structure and fine-grained detail without pretrained VEs or VAEs, a unified end-to-end objective coupling autoregressive cross-entropy for language with pixel-space flow matching for vision, and a native mixture-of-transformers (MoT) architecture that reduces objective interference while scaling efficiently. Two variants, SenseNova-U1-8B-MoT and SenseNova-U1-A3B-MoT, are launched on dense and mixture-of-experts backbones, respectively. Empirically, the model rivals top-tier understanding-only VLMs across text understanding, vision-language perception, knowledge reasoning, agentic decision-making, and spatial intelligence, while achieving strong any-to-image generation under a 32x compression ratio. It also supports visual-centric reasoning and interleaved generation for applications like illustrated guides, visual storytelling, and structured layouts. Preliminary results further suggest capabilities in vision-language-action and world modeling, indicating a shift toward natively unified multimodal intelligence without modular bridges.

Dataset

Dataset Composition and Sources

The authors organize the training data into two main stages: pre-training and mid-training, followed by a supervised fine-tuning (SFT) stage. The pre-training corpus combines large-scale web text, image-text pairs, and interleaved multimodal documents. The mid-training stage draws primarily from internal SenseNova V6.5 datasets. The generation corpus, covering text-to-image and image-editing tasks, is curated separately from web-scale sources.

Pre-training Stage

• Composition: Four categories: image-text pairs (32%), captions (17%), infographic understanding (14%), and pure text (37%).
• Curation pipeline: Four stages: cross-source deduplication, content and safety filtering, image quality filtering, and CLIP-ratio-balanced re-captioning to ensure balanced alignment across the corpus.

Mid-training Stage

• Composition: Four categories: General (39.2%), Agent and Spatial (22.3%), Knowledge Reasoning (19.3%), and Pure Text (19.2%).
• General category breakdown: General visual question answering (26.6%), multi-turn dialogue (26.4%), captioning (20.3%), OCR (18.6%), and multi-image understanding (8.2%).
• Knowledge Reasoning breakdown: Knowledge-oriented (12.0%) and reasoning-oriented (7.2%) data.
• Three-stage curation pipeline:
  1. Distribution-Balanced Sampling: Two-stage process. First, CLIP-based diversity sampling clusters visual embeddings via K-means and samples uniformly across clusters for long-tail coverage. Second, attribute profiling evaluates each sample on perceptual and semantic dimensions, followed by stratified sampling for balanced representation.
  2. Prompt Augmentation: Augments initial prompts along four dimensions: semantic expression, format and structural constraints, role and scenario, and task complexity. All answers are regenerated uniformly for consistent quality and style.
  3. Multi-Criteria Filtering: Automated model-based scoring evaluates each QA pair on correctness verification against ground-truth annotations, hallucination detection, and instruction-following assessment.

Supervised Fine-Tuning (SFT)

• Composition: Capability-atomic dimensions: spatial intelligence (15%), general multimodal understanding (13%), reasoning (12%), general NLP (11%), OCR and document analysis (11%), agentic function calling (10%), long-context conversation (8%), code (6%), multi-turn dialogue (4%), complex compositional understanding (4%), and supplementary data for the remaining proportion.
• Construction: Refines the mid-training candidate pool with a dual focus on quality and difficulty.
  • Quality-oriented selection: Reuses the multi-criteria filtering framework, scoring on visual fidelity, instruction clarity, response correctness, reasoning quality, and safety, with higher sampling of high-scoring examples.
  • Difficulty-oriented reconstruction: Rebalances supervision by concatenating short samples into long-context, multi-image, and multi-turn settings; applying rejection sampling for reasoning-intensive domains to retain intermediate difficulty examples; and rewriting under-specified queries with explicit constraints on format, style, and granularity.

Generation Data Organization

• Overall pipeline: Unified pipeline combining low-level filtering, deduplication, VLM-based captioning, and quality-aware filtering.
• Text-to-Image Data: Composition: Nature (40.5%), People (26.7%), and Design (20.7%), enriched with complex infographics, bilingual text rendering data, and a long tail of subclasses like posters, charts, and cityscapes.
• Image Editing Data: Sourced from web-scale data.
  • Content level: Natural scenes (52.3%) and human subjects (14.7%), with the remainder from infographic and synthetic edits.
  • Operation level: Subject addition/removal, background and color changes, identity transfer, motion manipulation, portrait editing, compositing, and reasoning-driven transformations.
  • Validation: Each editing pair is validated by decomposing its instruction into dynamic objectives (what changes, what stays) and verifying with a static physical-consistency constraint against the source image.
• Interleaved Data: Vision-text corpus with alternating text and image sequences. Four categories: Video, Lifestyle, Infographics, and Reasoning.
  • Distribution: Lifestyle (44%), including tutorials (26%), daily-life scenarios (14%), and picture books (4%); Infographics (29%); Video (19%); Reasoning (8%, each sample includes an explicit chain-of-thought trace).
  • Pipeline: Preprocessing, task-specific synthesis, and post-processing that verifies text quality, image quality, image-text consistency, and trajectory-level correctness.

Method

The authors introduce SenseNova-U1, a native, unified, and end-to-end framework that operates directly on pixels and words, eliminating reliance on pretrained vision encoders and variational autoencoders. As shown in the figure below, this parallel native architecture replaces heavy VAE-like encoders and decoders with lightweight encoding and decoding layers, enabling fully end-to-end learning of the representation space.

To map inputs into a unified token space, the model employs a patch encoding layer using two convolutional layers with GELU activation and 2D sinusoidal positional encoding, mapping each token to a 32x32 image patch. Visual and textual tokens are projected into a shared embedding space. For generation, a multi-layer perceptron head directly predicts pixel patches, bypassing deep diffusion heads. To handle varying resolutions, the authors introduce a resolution-adaptive noise scale $\sigma_R(H, W) = \sigma_0 \sqrt{N(H, W) / N_0}$σR​(H,W)=σ0​N(H,W)/N0​​, which preserves approximately constant per-token noise energy. This scale is explicitly fed to the denoiser via a sinusoidal MLP embedder, forming a joint time and noise-scale conditioning signal $\mathbf{s}_t = \tau_t + \text{NSEmb}(\bar{\sigma}(H, W))$st​=τt​+NSEmb(σˉ(H,W)).

The core backbone is a native Mixture-of-Transformers that unifies understanding and generation. The authors refine the native rotary position embedding to unify temporal and spatial encoding, reallocating pretrained LLM head dimensions across the T, H, and W axes. Within the MoT backbone, all modalities are processed under a shared self-attention mechanism. Text tokens attend causally, while image and noise tokens attend bidirectionally within their blocks while remaining causally conditioned on preceding context. Full parameter decoupling is adopted between the understanding and generation streams, with separate projections and feedforward blocks dynamically routed by token type. The framework is instantiated in two variants: an 8B dense symmetric parallel configuration and an A3B variant utilizing stream-wise mixture-of-experts.

The model is optimized end-to-end with a combined objective $\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\text{Und}} + \lambda_2 \mathcal{L}_{\text{Gen}}$Ltotal​=λ1​LUnd​+λ2​LGen​. For understanding, standard next-token prediction is used. For visual generation, the authors employ pixel-space flow matching with x-predict and v-loss. Given a clean image $\mathbf{x}$x and Gaussian noise $\epsilon$ϵ, the noisy sample is formed as $\mathbf{z}_t = t \mathbf{x} + (1 - t) \sigma_R \epsilon$zt​=tx+(1−t)σR​ϵ. The model regresses the clean signal to compute the velocity term $\mathbf{v}_\theta(\mathbf{z}_t, t) = (\hat{\mathbf{x}}_\theta(\mathbf{z}_t, t, \mathbf{s}_t) - \mathbf{z}_t) / (1 - t)$vθ​(zt​,t)=(x^θ​(zt​,t,st​)−zt​)/(1−t), optimized via mean squared error. For inference, a unified classifier-free guidance formulation independently modulates textual and visual conditions.

The training follows a progressive six-stage pipeline. Stage 1 initializes from a pretrained model, unifying QK projections and normalizations across temporal and spatial axes, followed by full-model continuation. Stage 2 freezes the understanding branch and pretrains the generation branch on text-to-image data across three phases of increasing resolution and task diversity. Stage 3 jointly trains both branches end-to-end on a curated mixture of understanding and generation data. Stage 4 fine-tunes the full model on high-quality instruction-following data. Stage 5 leverages reinforcement learning following Flow-GRPO. The authors design a dynamic resolution warmup strategy and employ three reward components: text rendering reward based on OCR Intersection-over-Union, style following reward via a VLM judge, and aesthetic reward using Human Preference Score. Stage 6 employs distribution matching distillation to reduce function evaluations from 100 to 8, optimizing only the generation branch.

To handle the distinct inference characteristics of understanding and generation pathways, the authors adopt a disaggregated deployment architecture using specialized engines for multimodal understanding and image generation. These engines exchange state through pinned shared memory. A hybrid attention kernel is introduced to efficiently support the mixed attention pattern where text rows remain causal while image rows attend to the full text prefix and image span.

To ensure high-quality training data, the authors implement systematic filtering workflows. For the understanding corpus, the pipeline involves distribution-balanced data curation via CLIP-based diversity sampling and attribute profiling, followed by prompt augmentation across semantic and structural dimensions, and finally multi-criteria quality filtering.

For the generation corpus, a unified four-stage flow is applied to text-to-image and editing data, encompassing low-level filtering based on resolution and sharpness, perceptual hash deduplication, VLM-based captioning, and quality filtering to remove watermarks and abnormal elements.

Experiment

SenseNova-U1 was evaluated across image and text understanding, generation, editing, and interleaved tasks, showing competitive or superior performance against strong baselines like Qwen3VL and Qwen3.5, particularly in mathematical reasoning, instruction following, and spatial intelligence. In generation, it matched or led open-source models on composition, text rendering, infographics, and reasoning-driven tasks, with chain-of-thought boosting results on WISE and RISEBench. Editing evaluations showed strong generalization despite gaps to specialized systems, while interleaved and unified reasoning benchmarks demonstrated genuine bidirectional synergy between understanding and generation. Ablations confirmed that the encoder-free design preserves semantic and pixel fidelity, understanding and generation co-evolve with minimal conflict in the MoT backbone, and the architecture scales efficiently with data.

SenseNova-U1 is introduced in two architectural variants: an 8B dense model and an A3B mixture-of-experts model. Both variants share a native multimodal design with unified temporal and spatial rotary position embeddings, but differ in depth, width, and expert configuration, with the A3B variant using 128 understanding and 32 generation experts. The 8B variant uses a dense configuration with 42 layers and a hidden size of 4,096, while the A3B variant uses 48 layers with a hidden size of 2,048. Both variants use the same patch size of 32x32 and identical head size allocation across temporal, height, and width axes. The A3B variant employs a mixture-of-experts design with 128 understanding experts and 32 generation experts, whereas the 8B variant uses a single expert per stream. The 8B variant includes pre-buffer layers for input mapping, while the A3B variant omits them.

SenseNova-U1 is trained in four progressive stages, starting with an understanding warmup that fuses attention projections and then continues with full-model training, followed by generation pre-training in three phases, unified mid-training, and a final supervised fine-tuning stage. The training recipe varies peak learning rates and schedulers across stages, with Stage 1 and Stage 2 phases using constant schedules and higher rates for generation, while Stage 4 uses cosine decay for two of its phases. All stages use AdamW with zero weight decay. Stage 1 includes an attention-fusion phase that halves QK parameter footprint by sharing projections across temporal and spatial axes, followed by a full-model continuation phase. Stage 2 is split into three phases for generation pre-training, with peak learning rates ranging from 1e-4 to 2e-4, higher than the understanding warmup stage. Stage 4 (unified SFT) uses cosine decay for two of its phases, while other stages and phases use constant learning rate schedules. All training stages use AdamW with beta1=0.9, beta2=0.95, epsilon=1e-8, and weight decay set to 0.0.

SenseNova-U1 demonstrates competitive or superior performance across multimodal understanding benchmarks, particularly excelling in mathematical reasoning and spatial intelligence tasks. Its encoder-free architecture yields notable gains in text-rich understanding and spatial reasoning compared to similarly sized baselines. SenseNova-U1 outperforms Qwen3VL-8B on multimodal reasoning benchmarks, with a clear advantage in mathematical reasoning. On spatial intelligence benchmarks, SenseNova-U1 shows strong performance across VSI-Bench, ViewSpatial, MindCube-Tiny, and 3DSR-Bench. The model matches or surpasses leading baselines on general vision-language benchmarks and hallucination evaluation.

SenseNova-U1 shows strong instruction-following performance, consistently outperforming the Qwen3.5 series on IFEval and IFBench. It also surpasses the Qwen3VL series on MMLU-Pro and SuperGPQA, narrowing the gap with larger Qwen3.5 models. The A3B variant demonstrates competitive agentic capabilities, approaching larger reasoning-focused baselines despite using fewer active parameters. SenseNova-U1 outperforms Qwen3.5 on both IFEval and IFBench, indicating superior instruction adherence. On MMLU-Pro and SuperGPQA, SenseNova-U1 exceeds Qwen3VL variants and approaches Qwen3.5 scores. The A3B variant shows strong multi-turn reasoning and agentic performance, nearly matching larger models on τ2-Bench and Claw-Eval.

SenseNova-U1 variants achieve the highest overall GenEval score among open-source models, outperforming several strong baselines. The model shows consistently high performance across most compositional subtasks, with a slight relative weakness in attribute binding. Its balanced profile leads to the best overall result. SenseNova-U1 with 8B total and 3B activated parameters scores 0.91 overall, matching the full 8B variant. The model leads open-source models, surpassing Qwen-Image, Lumina-DiMOO, and BAGEL on overall GenEval score. Performance is strong on single object, two object, counting, colors, and position, but attribute binding is slightly below the top specialized models.

SenseNova-U1 is evaluated across two architectural variants (8B dense and A3B mixture-of-experts) trained through a four-stage progressive recipe that includes understanding warmup, generation pre-training, unified mid-training, and supervised fine-tuning. The model demonstrates competitive or superior performance on multimodal understanding benchmarks, with particular strengths in mathematical reasoning, spatial intelligence, and text-rich understanding, outperforming similarly sized baselines like Qwen3VL-8B. It also shows strong instruction-following and knowledge reasoning, surpassing the Qwen3.5 series on IFEval and approaching larger models on MMLU-Pro and SuperGPQA, while the A3B variant exhibits competitive agentic capabilities despite fewer active parameters. In text-to-image generation, SenseNova-U1 achieves the highest overall GenEval score among open-source models, with balanced performance across compositional subtasks and only a slight weakness in attribute binding.