One-sentence Summary

The OpenMOSS Team at Fudan University, Shanghai Innovation Institute, and MOSI Intelligence presents MOSS-VL, an open vision-language model family that treats real-time perception while speaking as a first-class capability via gated cross-attention, a synthesized interaction corpus, and a staged curriculum, achieving the best average on three of four streaming benchmarks among open-source streaming models, including a 66.0 versus 37.5 score over the best baseline on OmniMMI Proactive Alerting, and widening its time-to-first-token advantage over same-backbone Qwen3-VL-8B from $2.8\times$2.8× to $5.1\times$5.1× as visual context grows.

Key Contributions

• MOSS-VL is an open vision-language model family whose gated cross-attention decoder with XRoPE and absolute timestamps receives incoming frames during generation; a synthesized interaction corpus and a lightweight Realtime-SFT stage teach when to speak, stay silent, or revise. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three among open-source streaming models and scores 66.0 versus 37.5 for the best baseline on OmniMMI Proactive Alerting.
• MOSS-VL-Instruct, trained without the real-time corpus, remains competitive at comparable scale and leads temporal-reasoning video benchmarks including Minerva, TOMATO, and VideoMME-Logical.
• Keeping visual tokens outside the decoded sequence gives the 11.3B-parameter model a growing serving advantage over same-backbone interleaved Qwen3-VL-8B in SGLang, with time-to-first-token lead widening from 2.8x to 5.1x as visual context grows; the release includes all five checkpoints, the staged training curriculum, and real-time inference code.

Introduction

Most open vision-language models process video offline: they read a finished clip end to end and then answer questions. In live settings, however, an assistant must watch an unfolding scene, decide when to speak, and keep observing even while generating a response. Recent streaming models allow continuous input and persistent queries, but they remain visually blind for the duration of each reply, so they cannot revise an answer when the evidence changes. The authors introduce MOSS-VL, an open vision-language model family that treats real-time interaction as a first-class capability through co-design. Its gated cross-attention decoder keeps visual tokens out of the decoded sequence and lets new frames append to the cross-attention cache while text is generated. XRoPE and absolute timestamp tokens align vision and language on a shared timeline. Synthesized real-time interaction data and a light Realtime-SFT stage teach the model when to speak, stay silent, or revise, while the same weights serve offline, streaming, and real-time modes through a shared system prompt.

Dataset

The authors introduce the Realtime-SFT corpus, a streaming video interaction dataset of 0.56M samples and about 34.8B tokens.

• Composition and sources:

  • Combines open-source streaming video understanding datasets with strict filtering and re-annotation.
  • Adds internally synthesized data for behaviors that existing datasets rarely cover: staying silent until evidence appears, revising an answer as the scene evolves, and recovering when a new event interrupts a reply.
  • Samples cast the model in an explicit interaction role: standing instructions that fire once when a condition is met, resident questions whose answers update as evidence accumulates, continuous real-time commentary, counting across a stream, probes for the right moment to speak, and video-independent dialogue for identity consistency.
  • A share of offline QA and general multimodal data is retained to preserve offline ability.

• Synthetic data construction:

  • Follows the caption-driven pipeline introduced in MOSS-Video-Preview.
  • Hierarchical, densely time-anchored captions are mined for state transitions of a focal object.
  • Each transition yields a question, an immediately answerable reply, and a trajectory of updates.
  • Replies are anchored to visual moments, laid out over frames with silence in between, and quality filtered.
  • Temporal anchoring is verified frame by frame against the actual footage: each reply is assigned the moment its evidence becomes visible and the moment it stops being valid.
  • Interrupted replies are rewritten as plain-language self-corrections rather than marked with a dedicated interrupt token.
  • A quality gate checks every sample against its frames and keeps only replies grounded in what is visible at emission time.

• How the data is used:

  • Used for Realtime-SFT, which teaches the model to treat incoming video as a stream of decisions.
  • Training samples interleave text with frames in arrival order.
  • Every frame is followed by a decision slot with one of three forms:
    • <|silence|>: keep watching.
    • <|response|> followed by text: speak now.
    • A reply ending with <|silence|>: finish speaking.
  • A reply is spread over consecutive slots frame by frame, emulating rate-limited real-time output.
  • A single user turn may contain several separate emissions.
  • The method adds exactly two new vocabulary entries for the state tokens, initialized from semantically related existing token embeddings.
  • The speak-or-wait decision is ordinary next-token prediction: if the most probable next token is <|silence|>, the model waits for the next frame; otherwise it decodes a reply. No dedicated decision head is attached.

• Corpus statistics and processing:

  • The model is supervised on 2.2M emission decisions.
  • 58.7% of emission decisions are self-timed rather than prompted by a fresh user question.
  • In 5.1% of samples, the target event never occurs, and the correct behavior is to stay silent throughout.
  • Streams run at 1 fps for up to 768 frames, about 12.8 minutes per window.
  • The corpus is decontaminated against evaluation suites by excluding benchmark videos via held-out lists.

Method

MOSS-VL pairs a native-resolution vision encoder with a language decoder initialized from Qwen3-8B. The two interact exclusively through gated cross-attention. Visual tokens do not enter the decoded sequence; instead, each frame contributes timestamp and placeholder tokens to the text stream, while patch tokens serve as cross-attention keys and values.

The vision encoder is a 27-layer transformer processing images at native resolution. The projection module merges 2x2 patch groups into the decoder hidden space. The decoder consists of 48 layers: 36 self-attention layers and 12 gated cross-attention layers placed at every fourth position. The cross-attention layers follow a gated design where queries come from text hidden states and keys and values come from visual tokens, implemented with grouped-query attention and QK-RMSNorm. Tanh gates with zero-initialized scalars wrap the attention and feed-forward paths, ensuring training starts from an intact language backbone.

To provide position information to the visual stream, the authors introduce XRoPE (cross-attention rotary position embedding). XRoPE places text tokens and visual patches in a shared three-axis coordinate space ordered by their logical position in the stream.

A text token advances all three axes together, while frame patches tile across the height and width dimensions riding on a shared temporal anchor. Rotations are applied to text-side queries and vision-side keys before they meet in cross-attention. For real-time operation, when a new frame arrives, only that frame is encoded, and its keys and values are appended to the cross-attention cache. Earlier frames are neither re-encoded nor recomputed. The decoded sequence grows only by timestamp and placeholder tokens, keeping patch tokens on the vision side. This append-only design ensures that an arriving stream leaves the decoding state intact, allowing the next generated token to attend to the updated cache efficiently.

The training process follows a four-stage pre-training curriculum: vision-language alignment, large-scale multimodal pre-training, high-quality multimodal pre-training, and annealing with long-context training. The curriculum progressively increases the maximum sequence length from 8K to 256K tokens. Following pre-training, the model undergoes post-training in two supervised stages. Standard supervised fine-tuning produces an offline instruction follower. A subsequent Realtime-SFT stage installs the real-time interaction paradigm, teaching the model to decide at every frame whether to speak or stay silent.

Mode control relies on a single system prompt for both streaming and real-time modes. To address the imbalance between silence slots and emission decisions during Realtime-SFT, the authors reweight the two state tokens using a focal factor and inverse-frequency class coefficients. This ensures the decision to speak and the decision to stay silent carry equal aggregate weight in the state-token loss.

Experiment

Each model is evaluated in its intended regime: MOSS-VL-Instruct on an offline suite of 39 benchmarks across five capability domains, and MOSS-VL-Realtime on four streaming benchmarks spanning levels L2 to L4, with serving efficiency and qualitative live sessions completing the assessment. Offline results show MOSS-VL-Instruct is strongest in perception, temporal video reasoning, adversarial referring grounding, document parsing, and selected reasoning tasks, while the main gaps appear in MMMU, document understanding, and standard referring and temporal grounding. In streaming evaluations, MOSS-VL-Realtime leads the average on three of four benchmarks and performs particularly well on proactive behavior subsets that require unprompted, correctly timed responses, though AURA retains an edge on perception QA over the current scene. Efficiency comparisons show lower time-to-first-token and end-to-end latency than Qwen3-VL-8B, with the advantage widening as visual context grows, and qualitative demos confirm timely conditional responses and continuous real-time commentary.

MOSS-VL-Realtime reaches the highest level in the capability hierarchy by continuing to perceive while it generates, with live demos and released real-time inference code demonstrating this behavior. On four streaming benchmarks covering the intermediate levels, it posts the best average on three and leads on proactive-response subsets, while a competitor retains the edge on current-scene perception QA. Its serving latency advantage over a shared-backbone model widens as visual context grows, despite carrying more vision tokens. MOSS-VL-Realtime achieves the best average scores on three of the four streaming benchmarks evaluated. The strongest gains appear on proactive-response subsets that test speaking unprompted at the right moment. It ranks second on the StreamingBench visual average, where perception QA over the current scene is the main focus. Compared with a model sharing the same language backbone, its latency advantage grows with visual context even though it carries about twice the vision tokens.

MOSS-VL uses a language decoder initialized from Qwen3-8B with gated cross-attention placed every fourth layer, while visual patches are consumed as cross-attention keys and values instead of entering the decoded sequence. Each video frame contributes timestamp tokens and a placeholder token to the text stream, and sampling is motion-adaptive from 1 to 16 fps so timestamps keep wall-clock time explicit. The released processor defaults to 1 fps and 256 frames; evaluations keep 1 fps and raise the cap to 768 frames. The decoder has 48 layers, with 36 self-attention layers and 12 gated cross-attention layers placed every fourth layer. Visual patches never enter the decoded sequence; each frame contributes only timestamp tokens and one placeholder token to the text stream. Video sampling is motion-adaptive from 1 to 16 fps, with absolute timestamp tokens marking wall-clock time for each frame. The released processor defaults to 1 fps and 256 frames, while evaluation settings keep 1 fps and raise the cap to 768 frames.

Five checkpoints are released for one architecture, comprising an 0708 lineage and an earlier independent 0408 lineage. The 0708 run moves from a pre-trained base to an offline instruction-tuned model through supervised fine-tuning, then to a real-time model through Realtime-SFT. The realtime checkpoint is listed for streaming evaluation, while the instruct checkpoint is listed for offline evaluation. The 0708 lineage follows a Base to Instruct to Realtime progression, with Realtime initialized from the instruct checkpoint. Realtime-SFT is the final supervised stage that installs real-time interaction behavior and uses a small fraction of total training tokens. The 0408 checkpoints are an earlier independently trained run of the same architecture, released for research continuity.

The MOSS-VL curriculum progresses from short alignment examples to much longer, higher-quality multimodal sequences, with maximum sequence length rising from 8K to 256K tokens. Compute shifts toward later stages while sample counts fall sharply, concentrating training on fewer but denser examples. The final Realtime-SFT stage is a small fraction of total training tokens but introduces the model's real-time speak-or-wait behavior. Maximum sequence length expands from 8K to 256K tokens over the course of training. Stage 3 consumes the largest token budget of the curriculum and focuses on high-quality multimodal data. Realtime-SFT accounts for under 3% of total training tokens yet installs the real-time interaction paradigm.

MOSS-VL-Instruct leads the compared open models most consistently in multimodal perception and temporal video understanding. It also records top scores on selected grounding, document parsing, and reasoning benchmarks. Its clearest gaps are on MMMU, document understanding, and standard referring and temporal grounding. MOSS-VL-Instruct posts the best result on five of twelve multimodal perception benchmarks, with especially large margins on MME-RealWorld and BLINK. On video understanding, it leads temporal-reasoning and long-form benchmarks such as Minerva, TOMATO, VideoMME-Logical, and EgoSchema. It also wins on adversarial Ref-Adv grounding and OmniDocBench document parsing, and shares the top score on ERQA reasoning. Relative weaknesses appear in MMMU, document understanding, and standard referring and temporal grounding, where other open models remain stronger.

The experiments evaluate MOSS-VL across streaming behavior, architecture, released checkpoints, training curriculum, and offline benchmarks. MOSS-VL-Realtime uses gated cross-attention over timestamped motion-adaptive frames and achieves the best average on three of four streaming benchmarks, with its latency advantage over a shared-backbone model growing as visual context increases, though it ranks second on current-scene perception QA. The training curriculum expands sequence length from 8K to 256K tokens and includes a final small Realtime-SFT stage that installs real-time speak-or-wait behavior. Offline, MOSS-VL-Instruct leads most consistently in multimodal perception and temporal video understanding, while showing gaps on MMMU, document understanding, and standard referring and temporal grounding.