One-sentence Summary

Researchers from WeChat Vision, Tencent Inc. introduce WeVisOne-VL, a vision-language model that unifies object-level and pixel-level perception through structured next-token prediction and a lightweight pixel decoder activated by a special token, trained via three-stage curriculum SFT followed by mixed-task RL, and achieving leading performance across over 30 benchmarks for reliable real-world deployment.

Key Contributions

• WeVisOne-VL is introduced as a vision-language model that unifies object-level and pixel-level visual perception through structured next-token prediction, supporting object detection, referring localization, counting, OCR, segmentation, and matting with outputs from points and boxes to masks and alpha mattes.
• A lightweight pixel decoder is integrated into the VLM through a special <OBJ> token, and dedicated coordinate tokens encode instance predictions, enabling instance-level mask and alpha matte outputs within a single language-guided model.
• The model is trained with a three-stage curriculum SFT followed by mixed-task RL; across over 30 benchmarks, it improves mean detection F1 and counting accuracy over comparable-scale VLMs, advances text grounding and referring localization, and surpasses evaluated baselines for referring segmentation and image matting while maintaining robust rejection for object-absent queries.

Introduction

Computer vision systems must recognize image content, identify user-specified objects, and determine object locations, supporting applications such as content creation, industrial inspection, autonomous driving, and embodied intelligence. Recent progress from conventional detection and segmentation models to open-world perception models and large vision-language models has broadened the concepts a model can recognize, but current systems still often hallucinate nonexistent objects, misinterpret unfamiliar descriptions, and overlook non-salient instances, leading to incomplete recall. The authors introduce WeVisOne-VL, a vision-language model built on Qwen3-VL at 2B and 4B scales that unifies object-level and pixel-level visual perception through a single natural-language interface. It casts diverse visual tasks as structured next-token prediction and uses a lightweight pixel decoder for dense outputs, trained on approximately 100M images with fine-grained annotations through a multi-stage curriculum.

Dataset

The authors construct the WeVisOne-VL Data Engine (WDE), which draws on GRIT, CC3M, LAION, SA-1B, and task-specific datasets. The annotation inventory covers more than 100 million images and about 3.6 billion bounding-box annotations.

Sources and composition

• Category-level concepts are discovered with an optimized Tokenize Anything model.
• Referring expressions are generated by prompting Qwen3-VL, using attributes, states, actions, and relations.
• Both query types are organized in a knowledge graph initialized from Wikipedia and WordNet, including canonical names, aliases, taxonomic relations, and whole-part relations.
• New terms are added only after text retrieval, vision-language verification, and stable visual evidence; equivalent categories are normalized while keeping descriptive constraints.
• Targeted acquisition uses concept names/aliases, visual retrieval, and model error signals, with priority for concept coverage, image domain, target scale, occlusion, dense scenes, confusable categories, and recurring referring failures.
• The section does not report separate sizes for GRIT, CC3M, LAION, SA-1B, or task-specific subsets.

Annotation processing

• Detection: proposals come from SAM3 and an in-house open-vocabulary DINO-style detector, merged into one candidate pool; Qwen3-VL verifies category matches and resolves duplicates/conflicts.
• Referring: SAM3 Agent and Rex-Omni propose candidates; Qwen3-VL verifies the full expression meaning, including attributes, states, actions, relations, whole-object versus part, and context versus target.
• Binary masks: SAM2 predicts a mask for each localized instance, inheriting the same query and bounding box. Category queries supervise detection and category segmentation; referring expressions supervise referring localization and referring segmentation.
• Quality: a subset of masks and boxes is manually spot-checked; annotation source is recorded for traceability.
• Hard examples: queries are drawn from confusable concepts, visually similar categories, changed descriptive conditions, and model failures. Negatives are accepted only if no object satisfies all conditions. Hard negatives have an empty target set and no box or pixel targets; hard positives retain reliable annotations for missed, crowded, or occluded targets.
• Matting: each binary mask S receives alpha candidates from BiRefNet and MatAnyone2 on the same crop. BiRefNet is accepted only if its binarized foreground passes IoU, coverage, foreground precision, area, box IoU, and centroid checks; otherwise the pipeline falls back to MatAnyone2 when its binarized mask has IoU of at least 0.9 with S, or retains S.
• A dedicated matting annotator is trained by sampling initial labels and high-quality matting data in equal proportions. It uses a frozen SAM2.1-Base+ backbone with a trainable alpha refinement head and multi-scale image features, prompt-mask guidance, and RGB detail features.
• The trained matting annotator is applied to all annotated training instances, producing final alpha pseudo-labels that remain linked to the original instance description and bounding box.

Usage in the model and data governance

• The unified annotation connects language queries, boxes, binary masks, and alpha mattes for the same instance, enabling detection, referring localization, category segmentation, referring segmentation, and matting supervision.
• After each training round, model errors are reviewed by concept, domain, scale, occlusion, task, and failure type. Annotations are corrected or regenerated only after review; unresolved cases are excluded.
• Each data release records annotation provenance, the model version that exposed each error, and review/revision history.
• Original images, identified near duplicates, and derived task records stay in the same data partition across updates. A fixed monitoring set is reserved for internal regression and excluded from training and threshold calibration. Formal benchmarks are isolated from training and feedback-driven data construction.
• Exact overall training mixture ratios are not provided in this section, except for equal-proportion sampling during matting annotator training.

Method

The authors develop a comprehensive framework for unified visual perception, comprising the WeVisOne-VL Data Engine (WDE) for large scale data construction and a specialized model architecture with a multi-stage training pipeline.

As shown in the figure below:

The WDE orchestrates a systematic pipeline to generate high-quality annotations for both category level recognition and context dependent target selection. The process begins with semantic discovery, where an optimized Tokenize Anything model and a prompted Qwen3-VL model extract category concepts and referring expressions. These are organized into a shared knowledge graph to guide targeted image acquisition. The acquired image query pairs undergo task specific localization. For detection, proposals from SAM3 and an in-house open vocabulary detector are merged and verified by Qwen3-VL. For referring tasks, the SAM3 Agent and Rex-Omni provide candidate regions, which are similarly consolidated through semantic verification. Following localization, SAM2 generates binary masks for each instance. To support fine grained boundary supervision, the authors introduce a dedicated matting data engine. This engine combines expert assisted annotation using BiRefNet and MatAnyone2, trains a lightweight matting annotator based on a frozen SAM2.1-Base+ model, and applies it to the entire corpus to produce continuous alpha pseudo-labels. Finally, a closed-loop feedback mechanism analyzes model errors after each training round to guide subsequent data acquisition and annotation refinement.

Building upon this rich data foundation, the authors design the WeVisOne-VL model, which extends Qwen3-VL with dedicated coordinate tokens and a lightweight pixel decoder. All visual tasks are formulated as structured next token prediction, utilizing tokens from <0> to <1000> for normalized coordinates and an token to bridge autoregressive localization with pixel level decoding. The lightweight decoder, containing approximately 9.6 million parameters, transforms discrete token representations into dense predictions. It consists of three core modules. First, the visual feature adapter extracts multiscale features from intermediate ViT layers, mapping them to a shared decoder space with positional and level embeddings. The outputs are flattened and concatenated to form a visual memory. Second, the box guided query decoder initializes instance queries directly from the hidden states of the generated tokens and their predicted bounding boxes. It employs six deformable blocks with self-attention and multiscale deformable cross-attention to aggregate visual features from the memory. Third, the instance conditioned dense prediction stage fuses multiscale visual features and utilizes a dynamic mask head to generate instance aware kernels. A coarse to fine refinement module progressively upsamples and modulates the features to produce the final segmentation or matting predictions.

The training pipeline follows a three-stage supervised fine tuning curriculum followed by mixed-task reinforcement learning. In Stage 1, the authors adapt the model to the new tokens and structured outputs by training the language model on 90 million examples, establishing an initial correspondence between predicted instances and the token. Stage 2 unfreezes the vision encoder and scales the input resolution and maximum box count to handle dense and small objects. Stage 3 establishes the bridge between the VLM and the pixel decoder. The decoder is first warmed up on segmentation samples using a combined focal and Dice loss:

Subsequently, the VLM and decoder are jointly trained with alternating next token prediction and pixel level tasks:

The decoder is then specialized for continuous alpha prediction using matting samples and a composite loss combining $L_1$L1​, gradient, Laplacian, and hard pixel terms.

Following SFT, the authors perform mixed-task RL to directly optimize task rewards. They employ an uncertainty-aware sample selection strategy, computing an uncertainty score based on the mean and standard deviation of group rewards to identify moderately difficult samples. The selected data is used in Group Relative Policy Optimization (GRPO) with grounding rewards. For each prompt, multiple responses are sampled and parsed into predicted object sets. The grounding reward is calculated via one-to-one Hungarian matching between predicted and ground-truth boxes, utilizing a cubed intersection over union (IoU) metric to emphasize precise localization:

This reward structure penalizes missed and duplicate objects, ensuring the model learns to generate highly accurate structured outputs.

Experiment

The experiments evaluate WeVisOne-VL across five object-level tasks (object detection, referring object detection, object counting, object pointing, and OCR) and two pixel-level tasks (referring segmentation and promptable matting), using greedy decoding with 1,960 visual tokens and comparing against Qwen-series VLMs, Rex-Omni, LocateAnything, LISA/UniPixel, and SAM2-based matting baselines. Across crowded, long-tailed, referring, counting, pointing, OCR, segmentation, and matting benchmarks, WeVisOne-VL consistently outperforms comparable baselines, with notable strengths in precise localization, fine-grained counting, negative-target rejection, and boundary-sensitive matting. Qualitative results further show that it preserves instance-level distinctions, follows nuanced language specifications, and produces grounded captions that link descriptions to specific image regions.

WeVisOne-VL uses a unified coordinate-based output format that covers detection, referring, pointing, and counting tasks in a single interface. Qualitative comparisons show it often localizes text and referred objects more completely than baseline methods, segments fine details in matting, and can return no mask for absent targets. It also produces grounded captions that link descriptions to image regions. The unified format represents outputs as object descriptions followed by bounding boxes or point coordinates across detection, referring, pointing, and counting. In referring segmentation, WeVisOne-VL segments the intended object in attribute, relational, and multi-instance cases and returns no mask when the target is absent. In text-heavy images, WeVisOne-VL localizes and transcribes curved or small text more fully than the baselines compared. In box-prompted and text-prompted matting, it better preserves fine boundaries such as whiskers, hair strands, and logos, and selects multiple intended instances where baselines miss or include distractors.

WeVisOne-VL achieves higher mean F1 than comparable vision-language models across zero-shot COCO and LVIS as well as dense Dense200 and VisDrone benchmarks. The advantage is largest on dense and tiny-object datasets, where many small and crowded instances require accurate recognition. Strong results at strict IoU=0.95 indicate gains come from both object coverage and precise localization. Mean F1 improvements over VLMs are especially pronounced on Dense200 and VisDrone. Performance at the strict IoU=0.95 threshold shows improved localization rather than only broader detection coverage. WeVisOne-VL outperforms VLMs at comparable model scales on general, long-tailed, and dense detection benchmarks.

The table reports referring object detection F1 and absent-target rejection rates across several benchmarks. WeVisOne-VL is reported to consistently outperform existing VLMs in single-target, multi-target, and absent-target settings. Among the listed VLMs, Qwen3-VL-8B leads on most target-present F1 benchmarks, while Qwen3-VL-4B leads on WeRef-Bench and PR-Bench absent-target rejection. WeVisOne-VL consistently outperforms existing VLMs across single-target, multi-target, and absent-target referring object detection settings. Among the listed VLMs, Qwen3-VL-8B achieves the best RefCOCOg, HumanRef, Ref-Adv, DVG, and Ref-L4 F1, while Qwen3-VL-4B records the best WeRef-Bench mean and rejection rate. GroundingDINO-T fails to reject RefCOCOg-Rej absent-target queries and trails the VLM entries on referring object detection F1.

WeVisOne-VL achieves the highest exact-match accuracy on every reported object-counting benchmark and split, ahead of the VLM baselines and a specialized detector. Among the VLM baselines shown, Qwen3-VL-4B consistently performs best across CountQA, PixMo-Count, CLOC, KubriCount, and MixCount. Accuracy is much lower on cross-domain and fine-grained counting tasks such as CLOC, KubriCount, and MixCount than on PixMo-Count. WeVisOne-VL outperforms all reported models on every object-counting split, while the specialized Grounding DINO-T detector is excluded from the VLM ranking. Qwen3-VL-4B is the strongest VLM baseline in the comparison, and exact-match scores drop sharply on CLOC, KubriCount, and MixCount compared with PixMo-Count.

This object-pointing evaluation covers dedicated pointing benchmarks and several detection-derived benchmarks. WeVisOne-VL achieves the best reported results on PointBench and PixMo-Points and performs strongly across COCO, LVIS, Dense200, and VisDrone. This indicates accurate pointing in both natural images and fine-grained mixed-object scenes. WeVisOne-VL outperforms all reported baselines on PointBench and PixMo-Points. Strong results on detection-derived benchmarks suggest its object-pointing ability extends to diverse and fine-grained scenes. Among reported baselines, Qwen3-VL models tend to improve on detection-derived benchmarks as model size increases, while Rex-Omni and LocateAnything are especially strong on Dense200.

The experiments evaluate a unified coordinate-based interface across detection, referring, pointing, and counting tasks using general, dense, and fine-grained benchmarks. Qualitative results show more complete localization of text and referred objects, finer matting boundaries, correct rejection of absent targets, and grounded captions linking descriptions to image regions. Quantitatively, WeVisOne-VL consistently outperforms comparable vision-language models on detection, referring detection with absent-target rejection, object counting, and pointing benchmarks, with gains attributed to both precise localization and broader coverage.