HyperAIHyperAI

Command Palette

Search for a command to run...

WeMM-Video: 오픈소스 옴니모달 비디오 이해의 한계를 넓히다

초록

비디오 이해, 특히 동반 오디오를 포함한 비디오 이해는 멀티모달 지능의 핵심 과제로 남아 있다. 본 보고서에서는 2B, 4B, 9B, 35B-A3B 변형을 아우르는 중영 이중언어 옴니모달 기반 모델군인 WeMM-Video를 소개한다. WeMM-Video는 오디오-비주얼 공동 인식 및 추론을 중심으로 하이브리드 사고 패러다임 안에서 텍스트, 이미지, 오디오, 비디오 처리를 통합하여, 빠르고 직접적인 응답을 위한 비사고 지시 모드와 명시적 단계별 추론을 위한 사고 모드를 모두 원활하게 지원한다. 이 모델들은 점진적 사전학습과 다중 교사 OPD, 반복 방지 MrDPO, 전문가 강화학습을 포함하는 다단계 후처리 파이프라인을 통해 약 1.2조 토큰 규모의 대규모 멀티모달 말뭉치로 학습되었다. 120개 이상의 벤치마크에서 평가한 결과, WeMM-Video는 옴니모달 비디오 이해 및 추론에서 우수한 성능을 보이면서도 일반 오디오, 무음 비디오, 이미지, 텍스트 작업 전반에서 높은 경쟁력을 유지하는 것으로 나타났다. 구체적으로 옴니모달 비디오 평가에서 주력 모델인 WeMM-Video-35B-A3B는 새로운 오픈소스 최고 성능을 수립하여 Qwen3.5-Omni-Flash 및 Seed2.0-Mini-0428과 같은 독점 모델을 크게 능가하는 동시에 강력한 Gemini-3.0-Flash와의 격차를 좁혔으며, 소형 모델인 WeMM-Video-9B는 더 큰 오픈소스 모델인 Qwen3-Omni-30B-A3B-Instruct를 뛰어넘었다. 오픈소스 커뮤니티의 추가 연구와 개발을 촉진하기 위해 WeMM-Video 모델군과 평가 및 재현 파이프라인을 공개한다.

One-sentence Summary

Researchers from WeChat Vision, Tencent Inc., and Tsinghua University introduce WeMM-Video, a Chinese-English bilingual omni-modal foundation model family spanning 2B, 4B, 9B, and 35B-A3B variants that unifies text, image, audio, and video processing through a hybrid-thinking paradigm and, after training on approximately 1.2 trillion tokens with multi-teacher OPD, anti-repetition MrDPO, and specialist RL, achieves open-source state-of-the-art omni-modal video understanding and reasoning across more than 120 benchmarks while maintaining competitive performance on general audio, silent video, image, and text tasks.

Key Contributions

  • WeMM-Video is a Chinese-English bilingual omni-modal model family spanning 2B, 4B, 9B, and 35B-A3B variants that unifies text, image, audio, and video processing and supports both a non-thinking instruct mode and a thinking mode for explicit step-by-step reasoning.
  • The models are trained on a multimodal corpus of approximately 1.2 trillion tokens through progressive pre-training and a multi-stage post-training pipeline combining multi-teacher on-policy distillation, anti-repetition multi-round direct preference optimization, and specialist reinforcement learning.
  • Evaluated across more than 120 benchmarks, the flagship WeMM-Video-35B-A3B establishes a new open-source state of the art in omni-modal video understanding, outperforming Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 while narrowing the gap with Gemini-3.0-Flash, and the compact 9B model surpasses the larger open-source Qwen3-Omni-30B-A3B-Instruct; model weights and the evaluation and reproduction pipeline are publicly released.

Introduction

Modern video understanding increasingly requires joint interpretation of continuous visual and acoustic signals, with applications in short-form video and embodied intelligence. Building on large language models, vision-language and audio models have advanced, but existing open-source systems remain brittle in complex real-world settings, often missing audio-visual details, misaligning sounds with visual events, and hallucinating cross-modal content. The authors introduce WeMM-Video, an open-source family of Chinese-English bilingual omni-modal foundation models spanning 2B to 35B-A3B, which unifies text, image, audio, and video processing. It combines a low-latency instruct mode with a reasoning mode that generates grounded visual-only and audio-visual traces to reduce hallucination, and it is trained with roughly 1.2 trillion tokens plus multi-stage post-training; the flagship model achieves open-source state-of-the-art results and competes with leading proprietary models.

Dataset

The authors use a comprehensive training corpus of about 1.2T tokens for WeMM-Video. The dataset composition and processing are organized as follows:

  • Joint audio-visual data This is the largest share, contributing 45.4% of the total corpus, and provides dense temporal supervision for multimodal dynamics.

  • Captioning data Captioning is the largest task category, accounting for 37.7% of tokens. It is divided into two complementary types:

    • Fine-grained visual captions Sourced from action- and event-dense videos in competitive sports, esports, gaming, choreography, and outdoor activities. These captions describe rapid physical dynamics and subtle temporal state transitions.
    • Joint audio-visual captions Sourced from films, short-form dramas, interviews, variety shows, and live-stream commerce. These captions jointly describe visual actions, spoken dialogue, and ambient acoustic signals along a continuous timeline.
    • All captioning candidates go through automated quality scoring, semantic deduplication, and toxicity filtering.
  • Dedicated audio corpus This subset covers large-scale automatic speech recognition, music understanding, and general audio understanding. It is included because the base model natively lacks acoustic perception, so this data helps construct and align the acoustic representation space.

  • Cross-media understanding split Built for multi-source tasks such as pairwise video comparison, video-soundtrack compatibility analysis, and e-commerce product-to-ad consistency verification. It targets heterogeneous inputs, streaming feeds, and multi-video contexts.

  • Knowledge and reasoning QA collection Created through a dual strategy:

    • Large-scale vision-language STEM and knowledge reasoning datasets augmented with high-quality thinking traces.
    • Curated native audio-visual reasoning tasks. This collection transfers deep reasoning capabilities to omni-modal scenarios while preserving the backbone model’s problem-solving skills.

The authors use these subsets to train WeMM-Video, with the token distribution reflecting mixture ratios across tasks. Joint audio-visual data and captioning dominate the corpus, while the audio, cross-media, and reasoning subsets address acoustic alignment, multi-source generalization, and advanced reasoning. The provided text does not describe explicit cropping, metadata schema construction, or other detailed preprocessing beyond the caption filtering pipeline.

Method

The authors design the WeMM-Video architecture to process multimodal inputs comprising text, image, audio, and video, ultimately producing textual outputs autoregressively. The framework is built upon various Qwen backbone models, ranging from 2B to 35B parameters, with audio encoders initialized from Qwen2.5-Omni.

As shown in the figure below:

For visual processing, the authors retain the native visual encoders and projection mechanisms of the respective backbones to preserve pretrained image and video understanding capabilities. To handle continuous acoustic signals, the model incorporates a dedicated audio module consisting of a Whisper-based audio encoder and a window-level Q-Former. The Whisper encoder extracts acoustic features from audio inputs sampled at 16,000 Hz and segmented into 30-second chunks. The window-level Q-Former then compresses these features within local temporal windows of 0.1 seconds, mapping each window to a single audio token in the large language model input space. This yields a rate of 10 audio tokens per second. For video inputs accompanied by audio, visual and audio tokens are interleaved in chronological order and supplemented with explicit textual timestamps to facilitate fine-grained cross-modal temporal alignment.

The training methodology follows a unified pipeline comprising progressive multimodal pre-training and multi-stage post-training. The pre-training phase utilizes a three-stage curriculum designed to systematically expand the context window while enriching cross-modal representations.

In the first stage, Acoustic Modality Alignment, the authors focus on bridging the acoustic representation space with the pretrained backbone under a compact context window. Only the audio aligner is optimized while the backbone and modality encoders remain frozen. Training on foundational audio-text pairs equips the model with core auditory perception capabilities.

The second stage, Omni-Modal Knowledge Injection, expands the context window for large-scale knowledge injection. All model parameters are unfrozen except for the visual encoder. By training on a diverse mixture of textual, visual, auditory, and joint audio-visual data, this stage establishes a unified cross-modal representation space.

The final pre-training stage focuses on Long-Context Capability Scaling. Maintaining the same trainable parameter configuration, the authors further scale the context window to accommodate extended multimodal sequences, such as long-form videos and extended audio recordings, enhancing the capacity to capture long-range temporal dependencies.

Following pre-training, the authors implement a progressive three-stage post-training pipeline to address challenges in general capabilities, generation stability, and specialized tasks.

First, they employ Multi-Teacher OPD to enhance general language and vision-language capabilities. Stronger external teachers provide supervision for text and vision-language tasks, while teachers from the same model family retain existing audio capabilities. Cross-modal distillation is further leveraged to transfer knowledge from text to audio tasks.

Second, to combat the tendency toward repetitive decoding, the authors apply Anti-Repetition MrDPO. This method iteratively optimizes the model using offline-mined hard-negative samples exhibiting repetitive decoding. To maintain stability across multiple optimization rounds, only LoRA adapters are updated during each round and are subsequently merged and reinitialized.

Finally, the Specialist RL stage utilizes reinforcement learning to enhance challenging downstream capabilities. The authors curate diverse training instances and filter out reliably solved samples. Depending on task verifiability, they combine rule-based rewards for deterministic tasks with rubric-based and teacher-consistency rewards for open-ended tasks. A dynamic reward router assigns task-specific rewards, and both instruct and thinking modes are jointly optimized through stochastic sampling. This stage is further extended to long-context training to maintain generation stability over extended multimodal inputs.

Experiment

The evaluation covers the WeMM-Video family at 2B, 4B, 9B, and 35B-A3B scales across video, audio, image, and text benchmarks, comparing against leading proprietary and open-source baselines. Omni-modal video results show that WeMM-Video-35B-A3B sets a new open-source state of the art, with thinking mode providing clear reasoning gains, while smaller variants remain competitive. Silent video evaluation confirms strong visual understanding without degradation from added audio, and the audio, image, and text evaluations validate well-rounded audio capabilities along with preserved linguistic and visual reasoning.

WeMM-Video-35B-A3B is an open-source omni-modal video model that outperforms Seed2.0-Mini-0428 and Qwen3.5-Omni-Flash on most general audio-visual understanding benchmarks while approaching Gemini-3.0-Flash. Its thinking mode consistently improves over the instruct configuration across the listed benchmarks. Smaller WeMM-Video variants also remain competitive with larger open-source baselines. The thinking mode surpasses Seed2.0-Mini-0428 and Qwen3.5-Omni-Flash on 16 of 17 general audio-visual understanding benchmarks. On Video-MME with audio, WeMM-Video-35B-A3B (Thinking) scores 86.4, trailing Gemini-3.0-Flash by a small margin and leading other compared proprietary models. Across all listed general audio-visual benchmarks, the thinking configuration improves over the instruct configuration.

The WeMM-Video family shows clear gains over open-source omni-modal baselines on general audio-visual video benchmarks. The 9B and 4B thinking variants consistently lead across the listed tasks, while the compact 2B variant remains competitive and even surpasses the much larger Qwen baseline on selected benchmarks. WeMM-Video-9B achieves the best results on every listed general audio-visual understanding benchmark, with WeMM-Video-4B as the next strongest model. The 4B variant delivers especially wide gains over both open-source baselines on Video-MME-v2 and AV-SpeakerBench. The 2B variant outperforms the much larger Qwen3-Omni-30B-A3B-Instruct baseline on Video-MME-v2 and AVUT Human while staying competitive elsewhere.

On the six reported silent video benchmarks, WeMM-Video-35B-A3B consistently outperforms Qwen3.5-Omni-Flash in both Instruct and Thinking configurations. The Thinking configuration is competitive with the Qwen3.6-35B-A3B vision-language backbone and the proprietary Seed2.0-Mini-0428, particularly on Video-MME, Video-MME-v2, Video-MMMU, and TempCompass. Both WeMM-Video-35B-A3B configurations score higher than Qwen3.5-Omni-Flash on every reported benchmark, with especially large gaps on Video-MMMU and Video-MME-v2. The Thinking configuration nearly matches the Qwen3.6-35B-A3B backbone on most benchmarks and exceeds it on MME-VideoOCR and TempCompass.

On the reported silent video benchmarks, WeMM-Video-9B and WeMM-Video-4B consistently outperform the open-source omni-modal baselines Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o-4.5-9B. The 9B model also matches or surpasses its Qwen3.5-9B vision-language backbone on most tasks. The compact 2B model remains competitive, often outperforming larger baselines on several benchmarks. WeMM-Video-9B matches or exceeds Qwen3.5-9B on most reported tasks, including Video-MME, Video-MMMU, MME-VideoOCR, and TempCompass. WeMM-Video-4B surpasses both open-source omni-modal baselines across all reported benchmarks, with especially large gains on Video-MMMU and TempCompass.

The WeMM-Video family demonstrates strong, well-rounded audio performance even though it is primarily video-centric. In general audio understanding, the 35B-A3B variant surpasses open-source omni-modal baselines on reported benchmarks and performs on par with the proprietary Qwen3.5-Omni-Flash on several tasks, while the 9B model leads on MMAU-Pro and MMSU. The models also show competitive or leading performance in voice interaction, spoken instruction following, music understanding, speech recognition, and lyrics recognition. WeMM-Video-35B-A3B outperforms Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o4.5-9B across the reported general audio understanding benchmarks. WeMM-Video-9B records the highest scores on MMAU-Pro and MMSU, surpassing all listed baselines. WeMM-Video-35B-A3B ties the proprietary Qwen3.5-Omni-Flash on DCase 2025 AQA and is close to it on MMAR and MMAU. The WeMM-Video family is reported to lead on voice interaction and spoken instruction-following while remaining competitive in music, speech, and lyrics recognition.

The experiments evaluate WeMM-Video models across general audio-visual, silent video, and audio understanding benchmarks, comparing open-source omni-modal baselines and proprietary models in instruct and thinking configurations. The 35B-A3B variant approaches or surpasses proprietary baselines on many audio-visual and silent video tasks, and its thinking mode consistently improves over instruct. Smaller 9B and 4B variants lead open-source baselines across most benchmarks, while the compact 2B remains competitive and occasionally beats much larger models. Audio results show strong, well-rounded performance despite the video-centric design, with leading or competitive scores in voice interaction, spoken instruction following, music, speech, and lyrics recognition.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp