HyperAIHyperAI

Command Palette

Search for a command to run...

WeMM-Video : repousser les frontières de la compréhension vidéo omni-modale open source

Résumé

La compréhension vidéo, en particulier lorsqu'elle s'accompagne d'audio, demeure un défi central de l'intelligence multimodale. Dans ce rapport, nous présentons WeMM-Video, une famille de modèles de fondation omni-modaux bilingues chinois-anglais comprenant des variantes de 2B, 4B, 9B et 35B-A3B. Centré sur la perception et le raisonnement audio-visuels conjoints, WeMM-Video unifie le traitement du texte, de l'image, de l'audio et de la vidéo au sein d'un paradigme de pensée hybride, prenant en charge de manière transparente à la fois un mode instruction sans réflexion pour des réponses rapides et directes et un mode réflexion pour un raisonnement explicite étape par étape. Les modèles sont entraînés sur un corpus multimodal à grande échelle d'environ 1,2 billion de jetons grâce à un pré-entraînement progressif et à un pipeline de post-entraînement en plusieurs étapes intégrant l'OPD multi-enseignants, le MrDPO anti-répétition et l'apprentissage par renforcement spécialisé. Évalué sur plus de 120 bancs d'essai, WeMM-Video démontre des performances supérieures en compréhension et en raisonnement vidéo omni-modaux, tout en conservant des capacités hautement compétitives dans les tâches générales d'audio, de vidéo silencieuse, d'image et de texte. Plus précisément, dans les évaluations vidéo omni-modales, notre modèle phare WeMM-Video-35B-A3B établit un nouvel état de l'art open source, surpassant nettement des modèles propriétaires tels que Qwen3.5-Omni-Flash et Seed2.0-Mini-0428 tout en réduisant l'écart avec le puissant Gemini-3.0-Flash, et notre modèle compact WeMM-Video-9B dépasse le modèle open source plus grand Qwen3-Omni-30B-A3B-Instruct. Nous publions la famille WeMM-Video ainsi que notre pipeline d'évaluation et de reproduction afin de favoriser la poursuite de la recherche et du développement au sein de la communauté open source.

One-sentence Summary

Researchers from WeChat Vision, Tencent Inc., and Tsinghua University introduce WeMM-Video, a Chinese-English bilingual omni-modal foundation model family spanning 2B, 4B, 9B, and 35B-A3B variants that unifies text, image, audio, and video processing through a hybrid-thinking paradigm and, after training on approximately 1.2 trillion tokens with multi-teacher OPD, anti-repetition MrDPO, and specialist RL, achieves open-source state-of-the-art omni-modal video understanding and reasoning across more than 120 benchmarks while maintaining competitive performance on general audio, silent video, image, and text tasks.

Key Contributions

  • WeMM-Video is a Chinese-English bilingual omni-modal model family spanning 2B, 4B, 9B, and 35B-A3B variants that unifies text, image, audio, and video processing and supports both a non-thinking instruct mode and a thinking mode for explicit step-by-step reasoning.
  • The models are trained on a multimodal corpus of approximately 1.2 trillion tokens through progressive pre-training and a multi-stage post-training pipeline combining multi-teacher on-policy distillation, anti-repetition multi-round direct preference optimization, and specialist reinforcement learning.
  • Evaluated across more than 120 benchmarks, the flagship WeMM-Video-35B-A3B establishes a new open-source state of the art in omni-modal video understanding, outperforming Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 while narrowing the gap with Gemini-3.0-Flash, and the compact 9B model surpasses the larger open-source Qwen3-Omni-30B-A3B-Instruct; model weights and the evaluation and reproduction pipeline are publicly released.

Introduction

Modern video understanding increasingly requires joint interpretation of continuous visual and acoustic signals, with applications in short-form video and embodied intelligence. Building on large language models, vision-language and audio models have advanced, but existing open-source systems remain brittle in complex real-world settings, often missing audio-visual details, misaligning sounds with visual events, and hallucinating cross-modal content. The authors introduce WeMM-Video, an open-source family of Chinese-English bilingual omni-modal foundation models spanning 2B to 35B-A3B, which unifies text, image, audio, and video processing. It combines a low-latency instruct mode with a reasoning mode that generates grounded visual-only and audio-visual traces to reduce hallucination, and it is trained with roughly 1.2 trillion tokens plus multi-stage post-training; the flagship model achieves open-source state-of-the-art results and competes with leading proprietary models.

Dataset

The authors use a comprehensive training corpus of about 1.2T tokens for WeMM-Video. The dataset composition and processing are organized as follows:

  • Joint audio-visual data This is the largest share, contributing 45.4% of the total corpus, and provides dense temporal supervision for multimodal dynamics.

  • Captioning data Captioning is the largest task category, accounting for 37.7% of tokens. It is divided into two complementary types:

    • Fine-grained visual captions Sourced from action- and event-dense videos in competitive sports, esports, gaming, choreography, and outdoor activities. These captions describe rapid physical dynamics and subtle temporal state transitions.
    • Joint audio-visual captions Sourced from films, short-form dramas, interviews, variety shows, and live-stream commerce. These captions jointly describe visual actions, spoken dialogue, and ambient acoustic signals along a continuous timeline.
    • All captioning candidates go through automated quality scoring, semantic deduplication, and toxicity filtering.
  • Dedicated audio corpus This subset covers large-scale automatic speech recognition, music understanding, and general audio understanding. It is included because the base model natively lacks acoustic perception, so this data helps construct and align the acoustic representation space.

  • Cross-media understanding split Built for multi-source tasks such as pairwise video comparison, video-soundtrack compatibility analysis, and e-commerce product-to-ad consistency verification. It targets heterogeneous inputs, streaming feeds, and multi-video contexts.

  • Knowledge and reasoning QA collection Created through a dual strategy:

    • Large-scale vision-language STEM and knowledge reasoning datasets augmented with high-quality thinking traces.
    • Curated native audio-visual reasoning tasks. This collection transfers deep reasoning capabilities to omni-modal scenarios while preserving the backbone model’s problem-solving skills.

The authors use these subsets to train WeMM-Video, with the token distribution reflecting mixture ratios across tasks. Joint audio-visual data and captioning dominate the corpus, while the audio, cross-media, and reasoning subsets address acoustic alignment, multi-source generalization, and advanced reasoning. The provided text does not describe explicit cropping, metadata schema construction, or other detailed preprocessing beyond the caption filtering pipeline.

Method

The authors design the WeMM-Video architecture to process multimodal inputs comprising text, image, audio, and video, ultimately producing textual outputs autoregressively. The framework is built upon various Qwen backbone models, ranging from 2B to 35B parameters, with audio encoders initialized from Qwen2.5-Omni.

As shown in the figure below:

For visual processing, the authors retain the native visual encoders and projection mechanisms of the respective backbones to preserve pretrained image and video understanding capabilities. To handle continuous acoustic signals, the model incorporates a dedicated audio module consisting of a Whisper-based audio encoder and a window-level Q-Former. The Whisper encoder extracts acoustic features from audio inputs sampled at 16,000 Hz and segmented into 30-second chunks. The window-level Q-Former then compresses these features within local temporal windows of 0.1 seconds, mapping each window to a single audio token in the large language model input space. This yields a rate of 10 audio tokens per second. For video inputs accompanied by audio, visual and audio tokens are interleaved in chronological order and supplemented with explicit textual timestamps to facilitate fine-grained cross-modal temporal alignment.

The training methodology follows a unified pipeline comprising progressive multimodal pre-training and multi-stage post-training. The pre-training phase utilizes a three-stage curriculum designed to systematically expand the context window while enriching cross-modal representations.

In the first stage, Acoustic Modality Alignment, the authors focus on bridging the acoustic representation space with the pretrained backbone under a compact context window. Only the audio aligner is optimized while the backbone and modality encoders remain frozen. Training on foundational audio-text pairs equips the model with core auditory perception capabilities.

The second stage, Omni-Modal Knowledge Injection, expands the context window for large-scale knowledge injection. All model parameters are unfrozen except for the visual encoder. By training on a diverse mixture of textual, visual, auditory, and joint audio-visual data, this stage establishes a unified cross-modal representation space.

The final pre-training stage focuses on Long-Context Capability Scaling. Maintaining the same trainable parameter configuration, the authors further scale the context window to accommodate extended multimodal sequences, such as long-form videos and extended audio recordings, enhancing the capacity to capture long-range temporal dependencies.

Following pre-training, the authors implement a progressive three-stage post-training pipeline to address challenges in general capabilities, generation stability, and specialized tasks.

First, they employ Multi-Teacher OPD to enhance general language and vision-language capabilities. Stronger external teachers provide supervision for text and vision-language tasks, while teachers from the same model family retain existing audio capabilities. Cross-modal distillation is further leveraged to transfer knowledge from text to audio tasks.

Second, to combat the tendency toward repetitive decoding, the authors apply Anti-Repetition MrDPO. This method iteratively optimizes the model using offline-mined hard-negative samples exhibiting repetitive decoding. To maintain stability across multiple optimization rounds, only LoRA adapters are updated during each round and are subsequently merged and reinitialized.

Finally, the Specialist RL stage utilizes reinforcement learning to enhance challenging downstream capabilities. The authors curate diverse training instances and filter out reliably solved samples. Depending on task verifiability, they combine rule-based rewards for deterministic tasks with rubric-based and teacher-consistency rewards for open-ended tasks. A dynamic reward router assigns task-specific rewards, and both instruct and thinking modes are jointly optimized through stochastic sampling. This stage is further extended to long-context training to maintain generation stability over extended multimodal inputs.

Experiment

The evaluation covers the WeMM-Video family at 2B, 4B, 9B, and 35B-A3B scales across video, audio, image, and text benchmarks, comparing against leading proprietary and open-source baselines. Omni-modal video results show that WeMM-Video-35B-A3B sets a new open-source state of the art, with thinking mode providing clear reasoning gains, while smaller variants remain competitive. Silent video evaluation confirms strong visual understanding without degradation from added audio, and the audio, image, and text evaluations validate well-rounded audio capabilities along with preserved linguistic and visual reasoning.

WeMM-Video-35B-A3B is an open-source omni-modal video model that outperforms Seed2.0-Mini-0428 and Qwen3.5-Omni-Flash on most general audio-visual understanding benchmarks while approaching Gemini-3.0-Flash. Its thinking mode consistently improves over the instruct configuration across the listed benchmarks. Smaller WeMM-Video variants also remain competitive with larger open-source baselines. The thinking mode surpasses Seed2.0-Mini-0428 and Qwen3.5-Omni-Flash on 16 of 17 general audio-visual understanding benchmarks. On Video-MME with audio, WeMM-Video-35B-A3B (Thinking) scores 86.4, trailing Gemini-3.0-Flash by a small margin and leading other compared proprietary models. Across all listed general audio-visual benchmarks, the thinking configuration improves over the instruct configuration.

The WeMM-Video family shows clear gains over open-source omni-modal baselines on general audio-visual video benchmarks. The 9B and 4B thinking variants consistently lead across the listed tasks, while the compact 2B variant remains competitive and even surpasses the much larger Qwen baseline on selected benchmarks. WeMM-Video-9B achieves the best results on every listed general audio-visual understanding benchmark, with WeMM-Video-4B as the next strongest model. The 4B variant delivers especially wide gains over both open-source baselines on Video-MME-v2 and AV-SpeakerBench. The 2B variant outperforms the much larger Qwen3-Omni-30B-A3B-Instruct baseline on Video-MME-v2 and AVUT Human while staying competitive elsewhere.

On the six reported silent video benchmarks, WeMM-Video-35B-A3B consistently outperforms Qwen3.5-Omni-Flash in both Instruct and Thinking configurations. The Thinking configuration is competitive with the Qwen3.6-35B-A3B vision-language backbone and the proprietary Seed2.0-Mini-0428, particularly on Video-MME, Video-MME-v2, Video-MMMU, and TempCompass. Both WeMM-Video-35B-A3B configurations score higher than Qwen3.5-Omni-Flash on every reported benchmark, with especially large gaps on Video-MMMU and Video-MME-v2. The Thinking configuration nearly matches the Qwen3.6-35B-A3B backbone on most benchmarks and exceeds it on MME-VideoOCR and TempCompass.

On the reported silent video benchmarks, WeMM-Video-9B and WeMM-Video-4B consistently outperform the open-source omni-modal baselines Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o-4.5-9B. The 9B model also matches or surpasses its Qwen3.5-9B vision-language backbone on most tasks. The compact 2B model remains competitive, often outperforming larger baselines on several benchmarks. WeMM-Video-9B matches or exceeds Qwen3.5-9B on most reported tasks, including Video-MME, Video-MMMU, MME-VideoOCR, and TempCompass. WeMM-Video-4B surpasses both open-source omni-modal baselines across all reported benchmarks, with especially large gains on Video-MMMU and TempCompass.

The WeMM-Video family demonstrates strong, well-rounded audio performance even though it is primarily video-centric. In general audio understanding, the 35B-A3B variant surpasses open-source omni-modal baselines on reported benchmarks and performs on par with the proprietary Qwen3.5-Omni-Flash on several tasks, while the 9B model leads on MMAU-Pro and MMSU. The models also show competitive or leading performance in voice interaction, spoken instruction following, music understanding, speech recognition, and lyrics recognition. WeMM-Video-35B-A3B outperforms Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o4.5-9B across the reported general audio understanding benchmarks. WeMM-Video-9B records the highest scores on MMAU-Pro and MMSU, surpassing all listed baselines. WeMM-Video-35B-A3B ties the proprietary Qwen3.5-Omni-Flash on DCase 2025 AQA and is close to it on MMAR and MMAU. The WeMM-Video family is reported to lead on voice interaction and spoken instruction-following while remaining competitive in music, speech, and lyrics recognition.

The experiments evaluate WeMM-Video models across general audio-visual, silent video, and audio understanding benchmarks, comparing open-source omni-modal baselines and proprietary models in instruct and thinking configurations. The 35B-A3B variant approaches or surpasses proprietary baselines on many audio-visual and silent video tasks, and its thinking mode consistently improves over instruct. Smaller 9B and 4B variants lead open-source baselines across most benchmarks, while the compact 2B remains competitive and occasionally beats much larger models. Audio results show strong, well-rounded performance despite the video-centric design, with leading or competitive scores in voice interaction, spoken instruction following, music, speech, and lyrics recognition.


Créer de l'IA avec l'IA

De l'idée au lancement — accélérez votre développement IA avec le co-codage IA gratuit, un environnement prêt à l'emploi et le meilleur prix pour les GPU.

Codage assisté par IA
GPU prêts à l’emploi
Tarifs les plus avantageux

HyperAI Newsletters

Abonnez-vous à nos dernières mises à jour
Nous vous enverrons les dernières mises à jour de la semaine dans votre boîte de réception à neuf heures chaque lundi matin
Propulsé par MailChimp