Command Palette
Search for a command to run...
WeMM-Video: Die Grenzen des quelloffenen omni-modalen Videoverständnisses erweitern
WeMM-Video: Die Grenzen des quelloffenen omni-modalen Videoverständnisses erweitern
Zusammenfassung
Das Videoverständnis, insbesondere in Verbindung mit begleitendem Audio, bleibt eine zentrale Herausforderung der multimodalen Intelligenz. In diesem Bericht stellen wir WeMM-Video vor, eine Familie zweisprachiger (Chinesisch-Englisch) omni-modaler Basismodelle mit den Varianten 2B, 4B, 9B und 35B-A3B. Im Mittelpunkt steht die gemeinsame audio-visuelle Wahrnehmung und Schlussfolgerung; WeMM-Video vereint die Verarbeitung von Text, Bild, Audio und Video in einem hybriden Denkparadigma und unterstützt nahtlos sowohl einen Nicht-Denk-Instruktionsmodus für schnelle, direkte Antworten als auch einen Denkmodus für explizites schrittweises Schlussfolgern. Die Modelle werden auf einem großen multimodalen Korpus von etwa 1,2 Billionen Token durch progressives Vortraining und eine mehrstufige Post-Training-Pipeline trainiert, die Multi-Teacher-OPD, Anti-Repetitions-MrDPO und spezialisiertes RL umfasst. Evaluierungen über mehr als 120 Benchmarks zeigen, dass WeMM-Video eine überlegene Leistung im omni-modalen Videoverständnis und -schlussfolgern erzielt und gleichzeitig hochgradig wettbewerbsfähige Fähigkeiten in allgemeinen Audio-, Stummvideo-, Bildund Textaufgaben beibehält. Insbesondere in omni-modalen Videoevaluierungen setzt unser Flaggschiffmodell WeMM-Video-35B-A3B einen neuen quelloffenen Stand der Technik, übertrifft proprietäre Modelle wie Qwen3.5-Omni-Flash und Seed2.0-Mini-0428 deutlich und verringert zugleich den Abstand zum leistungsstarken Gemini-3.0-Flash; unser kompaktes WeMM-Video-9B übertrifft das größere quelloffene Qwen3-Omni-30B-A3B-Instruct. Wir veröffentlichen die WeMM-Video-Familie zusammen mit unserer Evaluierungsund Reproduktionspipeline, um weitere Forschung und Entwicklung in der Open-Source-Community zu fördern.
One-sentence Summary
Researchers from WeChat Vision, Tencent Inc., and Tsinghua University introduce WeMM-Video, a Chinese-English bilingual omni-modal foundation model family spanning 2B, 4B, 9B, and 35B-A3B variants that unifies text, image, audio, and video processing through a hybrid-thinking paradigm and, after training on approximately 1.2 trillion tokens with multi-teacher OPD, anti-repetition MrDPO, and specialist RL, achieves open-source state-of-the-art omni-modal video understanding and reasoning across more than 120 benchmarks while maintaining competitive performance on general audio, silent video, image, and text tasks.
Key Contributions
- WeMM-Video is a Chinese-English bilingual omni-modal model family spanning 2B, 4B, 9B, and 35B-A3B variants that unifies text, image, audio, and video processing and supports both a non-thinking instruct mode and a thinking mode for explicit step-by-step reasoning.
- The models are trained on a multimodal corpus of approximately 1.2 trillion tokens through progressive pre-training and a multi-stage post-training pipeline combining multi-teacher on-policy distillation, anti-repetition multi-round direct preference optimization, and specialist reinforcement learning.
- Evaluated across more than 120 benchmarks, the flagship WeMM-Video-35B-A3B establishes a new open-source state of the art in omni-modal video understanding, outperforming Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 while narrowing the gap with Gemini-3.0-Flash, and the compact 9B model surpasses the larger open-source Qwen3-Omni-30B-A3B-Instruct; model weights and the evaluation and reproduction pipeline are publicly released.
Introduction
Modern video understanding increasingly requires joint interpretation of continuous visual and acoustic signals, with applications in short-form video and embodied intelligence. Building on large language models, vision-language and audio models have advanced, but existing open-source systems remain brittle in complex real-world settings, often missing audio-visual details, misaligning sounds with visual events, and hallucinating cross-modal content. The authors introduce WeMM-Video, an open-source family of Chinese-English bilingual omni-modal foundation models spanning 2B to 35B-A3B, which unifies text, image, audio, and video processing. It combines a low-latency instruct mode with a reasoning mode that generates grounded visual-only and audio-visual traces to reduce hallucination, and it is trained with roughly 1.2 trillion tokens plus multi-stage post-training; the flagship model achieves open-source state-of-the-art results and competes with leading proprietary models.
Dataset
The authors use a comprehensive training corpus of about 1.2T tokens for WeMM-Video. The dataset composition and processing are organized as follows:
-
Joint audio-visual data This is the largest share, contributing 45.4% of the total corpus, and provides dense temporal supervision for multimodal dynamics.
-
Captioning data Captioning is the largest task category, accounting for 37.7% of tokens. It is divided into two complementary types:
- Fine-grained visual captions Sourced from action- and event-dense videos in competitive sports, esports, gaming, choreography, and outdoor activities. These captions describe rapid physical dynamics and subtle temporal state transitions.
- Joint audio-visual captions Sourced from films, short-form dramas, interviews, variety shows, and live-stream commerce. These captions jointly describe visual actions, spoken dialogue, and ambient acoustic signals along a continuous timeline.
- All captioning candidates go through automated quality scoring, semantic deduplication, and toxicity filtering.
-
Dedicated audio corpus This subset covers large-scale automatic speech recognition, music understanding, and general audio understanding. It is included because the base model natively lacks acoustic perception, so this data helps construct and align the acoustic representation space.
-
Cross-media understanding split Built for multi-source tasks such as pairwise video comparison, video-soundtrack compatibility analysis, and e-commerce product-to-ad consistency verification. It targets heterogeneous inputs, streaming feeds, and multi-video contexts.
-
Knowledge and reasoning QA collection Created through a dual strategy:
- Large-scale vision-language STEM and knowledge reasoning datasets augmented with high-quality thinking traces.
- Curated native audio-visual reasoning tasks. This collection transfers deep reasoning capabilities to omni-modal scenarios while preserving the backbone model’s problem-solving skills.
The authors use these subsets to train WeMM-Video, with the token distribution reflecting mixture ratios across tasks. Joint audio-visual data and captioning dominate the corpus, while the audio, cross-media, and reasoning subsets address acoustic alignment, multi-source generalization, and advanced reasoning. The provided text does not describe explicit cropping, metadata schema construction, or other detailed preprocessing beyond the caption filtering pipeline.
Method
The authors design the WeMM-Video architecture to process multimodal inputs comprising text, image, audio, and video, ultimately producing textual outputs autoregressively. The framework is built upon various Qwen backbone models, ranging from 2B to 35B parameters, with audio encoders initialized from Qwen2.5-Omni.
As shown in the figure below:
For visual processing, the authors retain the native visual encoders and projection mechanisms of the respective backbones to preserve pretrained image and video understanding capabilities. To handle continuous acoustic signals, the model incorporates a dedicated audio module consisting of a Whisper-based audio encoder and a window-level Q-Former. The Whisper encoder extracts acoustic features from audio inputs sampled at 16,000 Hz and segmented into 30-second chunks. The window-level Q-Former then compresses these features within local temporal windows of 0.1 seconds, mapping each window to a single audio token in the large language model input space. This yields a rate of 10 audio tokens per second. For video inputs accompanied by audio, visual and audio tokens are interleaved in chronological order and supplemented with explicit textual timestamps to facilitate fine-grained cross-modal temporal alignment.
The training methodology follows a unified pipeline comprising progressive multimodal pre-training and multi-stage post-training. The pre-training phase utilizes a three-stage curriculum designed to systematically expand the context window while enriching cross-modal representations.
In the first stage, Acoustic Modality Alignment, the authors focus on bridging the acoustic representation space with the pretrained backbone under a compact context window. Only the audio aligner is optimized while the backbone and modality encoders remain frozen. Training on foundational audio-text pairs equips the model with core auditory perception capabilities.
The second stage, Omni-Modal Knowledge Injection, expands the context window for large-scale knowledge injection. All model parameters are unfrozen except for the visual encoder. By training on a diverse mixture of textual, visual, auditory, and joint audio-visual data, this stage establishes a unified cross-modal representation space.
The final pre-training stage focuses on Long-Context Capability Scaling. Maintaining the same trainable parameter configuration, the authors further scale the context window to accommodate extended multimodal sequences, such as long-form videos and extended audio recordings, enhancing the capacity to capture long-range temporal dependencies.
Following pre-training, the authors implement a progressive three-stage post-training pipeline to address challenges in general capabilities, generation stability, and specialized tasks.
First, they employ Multi-Teacher OPD to enhance general language and vision-language capabilities. Stronger external teachers provide supervision for text and vision-language tasks, while teachers from the same model family retain existing audio capabilities. Cross-modal distillation is further leveraged to transfer knowledge from text to audio tasks.
Second, to combat the tendency toward repetitive decoding, the authors apply Anti-Repetition MrDPO. This method iteratively optimizes the model using offline-mined hard-negative samples exhibiting repetitive decoding. To maintain stability across multiple optimization rounds, only LoRA adapters are updated during each round and are subsequently merged and reinitialized.
Finally, the Specialist RL stage utilizes reinforcement learning to enhance challenging downstream capabilities. The authors curate diverse training instances and filter out reliably solved samples. Depending on task verifiability, they combine rule-based rewards for deterministic tasks with rubric-based and teacher-consistency rewards for open-ended tasks. A dynamic reward router assigns task-specific rewards, and both instruct and thinking modes are jointly optimized through stochastic sampling. This stage is further extended to long-context training to maintain generation stability over extended multimodal inputs.
Experiment
The evaluation covers the WeMM-Video family at 2B, 4B, 9B, and 35B-A3B scales across video, audio, image, and text benchmarks, comparing against leading proprietary and open-source baselines. Omni-modal video results show that WeMM-Video-35B-A3B sets a new open-source state of the art, with thinking mode providing clear reasoning gains, while smaller variants remain competitive. Silent video evaluation confirms strong visual understanding without degradation from added audio, and the audio, image, and text evaluations validate well-rounded audio capabilities along with preserved linguistic and visual reasoning.
WeMM-Video-35B-A3B is an open-source omni-modal video model that outperforms Seed2.0-Mini-0428 and Qwen3.5-Omni-Flash on most general audio-visual understanding benchmarks while approaching Gemini-3.0-Flash. Its thinking mode consistently improves over the instruct configuration across the listed benchmarks. Smaller WeMM-Video variants also remain competitive with larger open-source baselines. The thinking mode surpasses Seed2.0-Mini-0428 and Qwen3.5-Omni-Flash on 16 of 17 general audio-visual understanding benchmarks. On Video-MME with audio, WeMM-Video-35B-A3B (Thinking) scores 86.4, trailing Gemini-3.0-Flash by a small margin and leading other compared proprietary models. Across all listed general audio-visual benchmarks, the thinking configuration improves over the instruct configuration.
The WeMM-Video family shows clear gains over open-source omni-modal baselines on general audio-visual video benchmarks. The 9B and 4B thinking variants consistently lead across the listed tasks, while the compact 2B variant remains competitive and even surpasses the much larger Qwen baseline on selected benchmarks. WeMM-Video-9B achieves the best results on every listed general audio-visual understanding benchmark, with WeMM-Video-4B as the next strongest model. The 4B variant delivers especially wide gains over both open-source baselines on Video-MME-v2 and AV-SpeakerBench. The 2B variant outperforms the much larger Qwen3-Omni-30B-A3B-Instruct baseline on Video-MME-v2 and AVUT Human while staying competitive elsewhere.
On the six reported silent video benchmarks, WeMM-Video-35B-A3B consistently outperforms Qwen3.5-Omni-Flash in both Instruct and Thinking configurations. The Thinking configuration is competitive with the Qwen3.6-35B-A3B vision-language backbone and the proprietary Seed2.0-Mini-0428, particularly on Video-MME, Video-MME-v2, Video-MMMU, and TempCompass. Both WeMM-Video-35B-A3B configurations score higher than Qwen3.5-Omni-Flash on every reported benchmark, with especially large gaps on Video-MMMU and Video-MME-v2. The Thinking configuration nearly matches the Qwen3.6-35B-A3B backbone on most benchmarks and exceeds it on MME-VideoOCR and TempCompass.
On the reported silent video benchmarks, WeMM-Video-9B and WeMM-Video-4B consistently outperform the open-source omni-modal baselines Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o-4.5-9B. The 9B model also matches or surpasses its Qwen3.5-9B vision-language backbone on most tasks. The compact 2B model remains competitive, often outperforming larger baselines on several benchmarks. WeMM-Video-9B matches or exceeds Qwen3.5-9B on most reported tasks, including Video-MME, Video-MMMU, MME-VideoOCR, and TempCompass. WeMM-Video-4B surpasses both open-source omni-modal baselines across all reported benchmarks, with especially large gains on Video-MMMU and TempCompass.
The WeMM-Video family demonstrates strong, well-rounded audio performance even though it is primarily video-centric. In general audio understanding, the 35B-A3B variant surpasses open-source omni-modal baselines on reported benchmarks and performs on par with the proprietary Qwen3.5-Omni-Flash on several tasks, while the 9B model leads on MMAU-Pro and MMSU. The models also show competitive or leading performance in voice interaction, spoken instruction following, music understanding, speech recognition, and lyrics recognition. WeMM-Video-35B-A3B outperforms Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o4.5-9B across the reported general audio understanding benchmarks. WeMM-Video-9B records the highest scores on MMAU-Pro and MMSU, surpassing all listed baselines. WeMM-Video-35B-A3B ties the proprietary Qwen3.5-Omni-Flash on DCase 2025 AQA and is close to it on MMAR and MMAU. The WeMM-Video family is reported to lead on voice interaction and spoken instruction-following while remaining competitive in music, speech, and lyrics recognition.
The experiments evaluate WeMM-Video models across general audio-visual, silent video, and audio understanding benchmarks, comparing open-source omni-modal baselines and proprietary models in instruct and thinking configurations. The 35B-A3B variant approaches or surpasses proprietary baselines on many audio-visual and silent video tasks, and its thinking mode consistently improves over instruct. Smaller 9B and 4B variants lead open-source baselines across most benchmarks, while the compact 2B remains competitive and occasionally beats much larger models. Audio results show strong, well-rounded performance despite the video-centric design, with leading or competitive scores in voice interaction, spoken instruction following, music, speech, and lyrics recognition.