One-sentence Summary

Trained on the converted SenseNova-Vision Corpus with auxiliary data to preserve capabilities, SenseNova-Vision, a unified multimodal model from SenseTime Research, Nanyang Technological University, and collaborators, reformulates computer vision as text and image generation, eliminating task-specific architectures, and covering detection, segmentation, depth, and multi-view geometry via natural-language instructions and visual prompts.

Key Contributions

• This work introduces a formulation that treats computer vision as unified multimodal generation, where a single unified multimodal model (UMM) handles heterogeneous visual tasks through its native text and image generation spaces without task-specific architectures, and the resulting model matches leading task-specialized systems.
• The SenseNova-Vision Corpus is constructed by converting heterogeneous computer vision annotations into instruction-response examples that span text, image, and mixed text-and-image targets, enabling large-scale training of a single UMM across structured visual understanding, dense geometric prediction, segmentation, and multi-view geometry.
• The SenseNova-Vision model, trained on the SenseNova-Vision Corpus with auxiliary multimodal data and no task-specific prediction heads, matches leading task-specialized systems on detection, OCR, keypoints, segmentation, depth, surface normals, point maps, and camera pose estimation. It supports language-defined task variants that combine category, color, region, and other visual cues, and the model and corpus are publicly available.

Introduction

Unified multimodal models (UMMs) that can both understand language and generate text and images offer a natural foundation for consolidating the many fragmented tasks of computer vision. Prior efforts either serialize diverse outputs into text tokens, which becomes awkward for dense spatial maps, or rely on task-specific decoders for pixel-level predictions, or use image generation alone, which cannot express symbolic records like boxes and categories. The authors introduce SenseNova-Vision, a single UMM that treats heterogeneous vision tasks as unified multimodal generation: natural language instructions specify the task and output schema, while text generation handles symbolic answers and image generation naturally represents spatially aligned dense predictions such as masks, depth, and normals. This formulation mirrors the unification that GPT brought to NLP, allowing one model to absorb supervision from detection, segmentation, dense geometry, and multi-view 3D without any task-specific heads.

Dataset

The authors construct the SenseNova-Vision Corpus (SN-VC), a large-scale multimodal dataset built from publicly available images. It unifies heterogeneous computer vision annotations into a common instruction-response format, enabling training of a unified multimodal model. The released subset, SN-VC-50M, contains 50 million generated and curated examples, while the remaining corpus can be reproduced from the provided source lists, prompt templates, and conversion scripts.

Dataset composition and sources

• SN-VC is organized into four task families, each drawing from public datasets (see Fig. 5a for counts):
• Structured visual understanding: detection, referring, pointing, keypoint localization, OCR, layout understanding, and GUI grounding.
• Dense geometric prediction: depth maps and surface normals.
• Segmentation: single-target (referring, reasoning, interactive) and multi-region (generic, panoptic) segmentation, plus grounded conversation generation (GCG) segmentation.
• Multi-view visual geometry: 3D reconstruction (XYZ point maps) and camera pose estimation.
• When original annotations are directly convertible to a decodable target, they are used as-is. For incomplete or insufficient supervision, additional targets are generated or curated.

Key details for each subset and processing

• SN-VC-50M (curated/generated subset) includes:
• Structured visual understanding: extra detection and OCR data generated via the Rex-Omni pipeline.
• Dense geometric prediction: additional depth and normal targets generated by MoGe-2, filtered by validity and scene-content checks.
• Segmentation: curated mixed text-image targets (e.g., GCG segmentation) where region descriptions and color legends align with mask images.
• Multi-view visual geometry: sparse depth completed with LingBot-Depth; examples with invalid depth, missing camera info, or inconsistent metadata are removed.
• Reproducible portion of SN-VC: the remaining examples are converted directly from public annotations using deterministic templates. The released source lists, prompt templates, conversion rules, and examples allow full reproduction.
• All examples follow a common schema: one or more visual inputs (single image, image with visual prompts, or ordered image set), a natural-language instruction, and a decodable response.
• Structured understanding responses are text with normalized coordinates and lightweight markers (e.g., <bbox>, <point>).
• Dense predictions are rendered as images: inverse-depth grayscale for depth, RGB normal maps for surface normals.
• Segmentation uses binary masks for single-target tasks, and mixed text-image responses with color legends for multi-region tasks.
• Multi-view geometry outputs per-view XYZ point maps as images and camera poses as structured text with reserved tokens (<frame>, <quat>, etc.).

How the paper uses the data

• The entire SN-VC corpus is used for training a unified multimodal model. There is no separate mixture ratio mentioned; examples from all families are treated as independent samples, even when the same source image appears in multiple tasks.
• Training strictly respects official evaluation splits: for any dataset that overlaps with a benchmark, the corresponding evaluation images and annotations are excluded from training. All other converted examples constitute the training set.

Method

The authors establish a unified data protocol to standardize the training schema for the Unified Multimodal Model (UMM). This protocol defines a common sample structure comprising one or more visual inputs, a natural-language instruction, and a decodable target response. The instruction dictates the task intent, output schema, and decoding convention, while the response is formulated as text, an image, or a mixed text-image output. This design ensures that diverse computer vision tasks can be recovered into benchmark-compatible labels, coordinates, masks, dense maps, or camera parameters. The framework categorizes tasks into four distinct families: structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.

For structured visual understanding, the authors represent annotations as text generation targets, utilizing lightweight textual markers such as <p>, <bbox>, and <point> to delimit phrases and normalized coordinate fields. Dense geometric prediction tasks, including depth and surface normal estimation, leverage conditional image generation where the response image stores deterministic visual encodings of the dense signals. Segmentation tasks employ either binary mask images for single-target requests or mixed text-image responses with <color> markers for multi-region tasks. Multi-view visual geometry tasks represent dense scene geometry as per-view XYZ point maps in the image space, while camera pose outputs are encoded as structured text sequences using reserved special tokens for quaternion rotations, translations, and scales.

To construct the training corpus, the authors convert public computer vision datasets into instruction-response examples following the established protocol. Each source annotation is transformed via task templates into training samples, with visual inputs selected according to the task context. To address incomplete or insufficient supervision, the authors curate additional targets through specific filtering and generation workflows. For instance, they utilize MoGe-2 to densify incomplete supervision for geometric prediction and apply LingBot-Depth to complete sparse depth in multi-view geometry, followed by rigorous validity and scene-content filtering to ensure data quality.

The training process adapts an existing UMM to the unified vision-task corpus through supervised fine-tuning. The authors employ a mixed-task fine-tuning strategy that jointly optimizes the model on converted computer vision samples and general-purpose multimodal data. This approach enables the model to learn representations for benchmark-readable outputs while preserving broad capabilities like image understanding and generation. A joint sampling strategy draws mini-batches from a weighted mixture of task categories, producing interleaved text and visual supervision targets within the same optimization process. Text-form outputs are optimized using the standard cross-entropy loss under a next-token-prediction paradigm. Conversely, visual outputs are encoded into a VAE latent space and optimized via a rectified-flow training objective inherited from the base model. This heterogeneous learning mechanism allows the model to master diverse computer vision targets through native text and image decoders without introducing task-specific prediction heads.

For high-resolution and multi-view training, the authors maintain a SigLIP2 input resolution of up to 980 pixels for image-input tasks requiring fine spatial conditioning, such as segmentation. In multi-view visual geometry, training samples are formed by randomly selecting up to 10 views per scene to manage memory constraints. Point maps are aligned to the first view and center-normalized, while camera pose estimation repurposes dedicated vocabulary entries as special tokens to encode quantized pose parameters. The optimization utilizes the AdamW optimizer with a learning rate of $2.5 \times 10^{-5}$2.5×10−5 and no weight decay. The model is trained for 50,000 steps with a maximum context window of 32,000 tokens per sample, and the final evaluation relies on an Exponential Moving Average checkpoint with a ratio of 0.995.

Experiment

The evaluation covers four task families—structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry—under a unified multimodal generation setup where tasks are specified with natural language and outputs are parsed into structured text or image formats. Comparisons with recent generalist models show that unifying text and image generation enables broader coverage, and the model retains its pretrained multimodal capabilities while achieving strong results across diverse vision tasks. Qualitative analysis reveals that convergence speeds vary by task, with spatially aligned predictions converging fastest, and that capabilities learned from separate domains can be recombined to perform novel tasks, such as referring-style interactive segmentation and free-form language-to-mask generation, indicating flexible cross-modal correspondence.

The evaluation spans box-based detection, referring, OCR localization, point localization, GUI grounding, and keypoint detection. SenseNova-Vision is reported to achieve strong overall performance, especially on dense, long-tailed, small-object, referring, and OCR benchmarks. Among the compared baselines, LocateAnything leads on most spatial localization tasks, while Qwen3.5-9B attains the highest keypoint accuracy. LocateAnything achieves the best scores on COCO-Common, LVIS, Dense200, VisDrone, HierText, ICDAR15, and ScreenSpot-V2, with a 60.4 F1@mIoU on HierText and 58.7 on Dense200. Qwen3.5-9B delivers the highest keypoint localization performance (92.2 F1@mOKS) but its GUI grounding accuracy is only 11.4. Bagel and Qwen3-VL-8B-Instruct obtain competitive referring scores (above 70 F1@mIoU on HumanRef and RefCOCOg) but are outperformed by LocateAnything on HumanRef and RefCOCOg test.

The table compares dense depth and surface normal estimation across multiple benchmarks. Geometry-specialized models such as MoGe-2 achieve the strongest overall accuracy, while generation-based methods Marigold and DICEPTION show higher errors. The proposed SenseNova-Vision outperforms the other generation-based approaches and remains competitive with specialized models. MoGe-2, a geometry-specialized model, attains the lowest absolute relative depth error and highest δ1 on NYUv2, KITTI, ETH3D, and ScanNet. Generation-based methods Marigold and DICEPTION consistently show larger depth errors and lower normal accuracy than specialized models across all evaluated datasets. SenseNova-Vision surpasses Marigold and DICEPTION on both depth and normal estimation and reaches performance on par with specialized models like DepthAnything V2.

SenseNova-Vision provides competitive segmentation across multiple settings, with top results on reasoning segmentation and the best box-prompt interactive segmentation mIoU among the compared models. It trails specialized models on generic panoptic segmentation and referring benchmarks, where models with external mask priors hold an advantage. On grounded conversation generation segmentation, it ranks second behind X-SAM but outperforms LISA-7B. SenseNova-Vision leads all reported methods on reasoning segmentation, achieving the highest gIoU on both validation and test sets. For interactive segmentation with box prompts, it obtains the highest mIoU (73.9), while its point-prompt mIoU is lower than PSALM and X-SAM. In generic panoptic segmentation, its PQ is lower than PSALM and X-SAM, reflecting the strength of specialized models that leverage pretrained mask models.

Feed-forward geometric models consistently outperform generalist approaches across multi-view reconstruction and camera pose estimation. On ETH3D, VGGT and Depth Anything 3 achieve markedly higher accuracy and completeness than MapAnything and G2VLM, while for camera pose, feed-forward models lead in RTA and AUC, especially on RealEstate10K and CO3Dv2. The performance gap underscores the advantage of geometry-focused training and inductive biases. Depth Anything 3 achieves the highest reconstruction F1 on 7Scenes (90.5) and the best camera pose metrics on RealEstate10K (RTA 96.4, AUC 89.6) and CO3Dv2 (RTA 98.0, AUC 91.8). On ETH3D, VGGT reaches an accuracy of 0.177 and F1 of 80.9, substantially better than MapAnything (accuracy 0.400, F1 67.0) and G2VLM (accuracy 0.784, F1 36.7). Generalist models like G2VLM show notably lower AUC for camera pose (51.8 on Re10K, 55.2 on CO3Dv2), indicating weaker relative pose accuracy compared to feed-forward models.

SenseNova-Vision outperforms the vision-language understanding model Youtu-VL on detection, semantic segmentation, referring segmentation, and depth estimation. It remains competitive with the image-generation-centered model Vision Banana on segmentation and dense prediction benchmarks, despite their different output modalities. This broader coverage is enabled by unified multimodal generation over text, image, and mixed outputs, which naturally accommodates the heterogeneous output forms required by diverse computer vision tasks. SenseNova-Vision achieves higher detection mAP, semantic segmentation mIoU, and referring segmentation cIoU than Youtu-VL. Its depth estimation accuracy (δ1) on NYUv2 is substantially higher than Youtu-VL's. The model is competitive with Vision Banana on image-space segmentation and dense prediction even though Vision Banana focuses on image generation. Unified multimodal generation allows SenseNova-Vision to handle structured, semantic, and dense visual tasks without being restricted to a single output modality.

Experiments assess SenseNova-Vision on spatial localization, dense depth and normal estimation, segmentation, multi-view geometry, and comparisons with other generalist models. The model consistently performs strongly, outperforming generation-based methods on depth and normal tasks while leading in reasoning segmentation and box-prompt interactive segmentation. Its unified multimodal generation over text, image, and mixed outputs allows it to handle diverse visual tasks and often rival specialized models, though feed-forward geometry models still dominate on multi-view reconstruction and camera pose.