One-sentence Summary

The TLive-Omni Team at Taobao & Tmall Group of Alibaba presents TLive-Omni, an omni-modal understanding model for e-commerce live streaming that maps image, video, audio, and text inputs into a unified representation space and introduces Per-vGrid temporal alignment, a three-stage supervised training recipe, and Faithful-RFT reinforcement fine-tuning to improve answer faithfulness, achieving strong performance on e-commerce live streaming benchmarks and general benchmarks.

Key Contributions

• TLive-Omni is an omni-modal understanding model for e-commerce live streaming that maps image, video, audio, and text inputs into a unified representation space. A scenario-oriented atomic capability taxonomy and compact data production engine convert live-commerce audio, image, and video streams into supervision for speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA.
• Per-vGrid organizes timestamped audio-video evidence by grouping each video grid with temporally corresponding audio inside explicit boundary tokens, and the training recipe combines a three-stage supervised fine-tuning stage with Faithful-RFT reinforcement fine-tuning. Faithful-RFT scores final answers directly with task-verifiable rewards to improve answer faithfulness and expression quality while meeting real-time demands.
• Synchronized length-grouped sampling reduces padding while preserving comparable workloads across workers, and a lightweight dynamic sampling strategy regenerates rollout groups with near-zero reward variance to maintain meaningful relative advantages for GRPO. Experiments on in-house live-commerce and general multimodal benchmarks demonstrate strong live-commerce task performance and competitive or improved results over corresponding Qwen3.5 4B and 9B backbones.

Introduction

E-commerce live streaming demands omni-modal understanding that jointly interprets speech, video frames, product images, overlaid text, and user queries, aligning evidence to product-centric moments across long streams. General-purpose omni models lack fine-grained, product-focused capabilities, and existing e-commerce extensions do not systematically address atomic live-streaming tasks such as speech recognition, visual grounding, and temporal grounding. The authors present TLive-Omni, a model built on a Qwen3.5 backbone that integrates a pretrained audio encoder, supports up to 256K multimodal tokens, and introduces Per-vGrid for audio–video alignment by grouping matched visual and audio evidence with explicit temporal markers. Training proceeds through three-stage supervised fine-tuning followed by Faithful-RFT, a reinforcement stage that suppresses redundant reasoning traces and directly optimizes answer faithfulness with task-verifiable rewards, while dynamic rollout resampling and synchronized length-grouped batching handle heterogeneous modality batches. A scenario-oriented capability taxonomy and a data production engine map live-commerce streams into atomic supervision signals, and an in-house evaluation suite verifies strong performance on business-oriented tasks alongside competitive results on general-purpose benchmarks.

Dataset

The authors construct omni-modal live-commerce supervision from raw e-commerce data and curated general-domain data. Because audio, image, and video sources have different noise patterns, they use separate modality-specific pipelines that convert noisy inputs into task-grounded annotations. Exact dataset sizes are not reported in this section.

Audio pathway

• Applies voice-activity detection and audio normalization.
• Uses cross-model ASR ensemble agreement to obtain ASR pseudo-labels.
• Uses an LLM to mine domain-specific low-frequency terms and build a live-specific keyword lexicon for in-context ASR.
• Cross-validates speaker labels from an audio-only diarization model and an MLLM using transcript plus visual cues, with temporal IoU consistency checks; mismatches are verified with video frames and lip motion.
• Builds audio captions and audio QA pairs by generating sound, music, speech content, and speaker rhythm dimensions with audio-LLM ensembles, then merging them with an LLM.

Image pathway

• Builds product visual grounding data through a VLM Detector-Judger loop: the detector proposes candidate boxes and the judger filters inaccurate boxes into rejected-sample data.
• For open-source generic detection data, clusters raw text labels into standardized categories and applies category-balanced sampling.
• For Markdown and HTML parsing, re-renders parsed content and compares it with the source image instead of trusting VLM output directly.
• Filters caption and selling-point data by source consistency, using a VLM judger to remove unsupported descriptions.
• Constructs reasoning-enriched Image QA by generating an answer with a reasoning trace using a VLM, then scoring with rule-based or LLM judging and keeping samples whose answer remains consistent with the trace.

Video pathway

• Uses TransNet V2 to split videos at physical shot boundaries for dense captioning; ASR extracts speech, a VLM describes visual content, and an LLM merges them into dense captions.
• For Video QA and temporal grounding, segments videos at semantic event boundaries and uses a VLM to generate QA pairs and temporal-grounding annotations.
• Applies a shared refinement pipeline with scene-oriented resampling for long-tail coverage and temporal calibration for different durations, then a VLM judger checks factual and logical consistency as well as timestamp alignment.
• Builds reasoning-enriched Video QA from no-audio videos, generating answer and reasoning trace with a VLM and validating multiple-choice and numerical answers through rule-based matching and free-form answers with an LLM judge.

The resulting annotations are used as task-grounded supervised fine-tuning data for omni-modal live-commerce understanding, covering audio captioning, audio QA, image grounding, image QA, dense video captioning, video QA, temporal grounding, and reasoning-enriched QA.

Method

TLive-Omni is a text-only output omni-modal understanding model designed for image, video, audio, and text inputs. The authors leverage a Qwen3.5 backbone as the language and vision substrate. For visual processing, spatial merging yields $(h/32) \times (w/32)$(h/32)×(w/32) visual tokens per image, and a multi-layer perceptron maps these features to the backbone embedding dimension. To incorporate audio, the model grafts the audio transformer encoder from Qwen3-Omni into the same embedding space through a lightweight audio aligner. This encoder consumes 128-dimensional mel-spectrogram features and compresses speech into approximately 13 tokens per second.

To enable long live streaming understanding, the authors introduce Per-vGrid, which organizes video and corresponding audio into a sequence of timestamped video grids. The visual content of a temporal grid and the audio segment covering the same time interval are placed in the same local span. Per-vGrid prepends an explicit textual timestamp to each grid, makes grid boundaries explicit, keeps video and audio tokens contiguous, and separates neighboring grids at the sequence level. The timestamps and audio spans are derived from the realized video sampling process, preserving precise temporal alignment even when integer frame selection causes the realized sampling rate to differ from the requested rate.

Refer to the framework diagram:

Constructing supervision for omni-modal live-commerce understanding requires modality-specific construction due to distinct noise patterns in audio, image, and video sources. The authors process raw e-commerce data through separate pathways, each with its own filtering and quality control. The audio pathway uses cross-model agreement voting across an ASR ensemble and cross-validates speaker labels using both acoustic features and visual cues. The image pathway employs a vision-language model Detector-Judger loop for product visual grounding and applies category-balanced sampling to reduce label noise. The video pathway utilizes TransNet V2 for physical shot boundaries and semantic event boundaries to generate dense captions, QA pairs, and temporal-grounding annotations.

As shown in the figure below:

Following data construction, TLive-Omni follows a three-stage supervised fine-tuning recipe. Stage 1 freezes the language model and audio encoder, training only the audio aligner on 5M ASR samples to establish an initial mapping. Stage 2 introduces a broader audio mixture over 26M samples, training the audio encoder and aligner while keeping the language model frozen. Stage 3 performs joint multimodal supervised fine-tuning over 14M samples, freezing the encoders and optimizing the aligners and language model for diverse tasks.

To address the batching challenges posed by heterogeneous multimodal data with varying token lengths, the authors propose a synchronized length-grouped sampler. This sampler partitions samples by modality, sorts them by token length, and splits them into global batches. This organization reduces padding while preserving a fixed sample count, ensuring comparable workloads across workers during training.

To improve answer faithfulness and expression quality for real-time live streaming, the authors introduce Faithful-RFT after the three-stage SFT. This approach uses Group Relative Policy Optimization with task-verifiable rewards. The data is organized into four streams: image, video-with-audio, video-without-audio, and audio. For each multimodal prompt $x_i$xi​, the policy samples a group of $G$G candidate responses. The visual and audio encoders and aligners remain frozen. The group-relative advantage is computed as:

$$\widehat{A}_{i,g} = \frac{R_{i,g} - \mathrm{mean}_{g'}(R_{i,g'})}{\mathrm{std}_{g'}(R_{i,g'}) + \epsilon_s}$$Ai,g​=stdg′​(Ri,g′​)+ϵs​Ri,g​−meang′​(Ri,g′​)​

The implementation minimizes the group-level loss:

$$\mathcal{L}_i(\theta) = \frac{1}{G} \sum_{g=1}^G \frac{1}{|y_g|} \sum_{t=1}^{|y_g|} \left[ -\min\left\{\frac{\pi_\theta(y_{g,t} \mid x, y_{g,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{g,t} \mid x, y_{g,<t})} \widehat{A}_g, \operatorname{clip}\left(\frac{\pi_\theta(y_{g,t} \mid x, y_{g,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{g,t} \mid x, y_{g,<t})}, 1-\epsilon_l, 1+\epsilon_h\right) \widehat{A}_g\right\} + \beta d_{g,t}^{\mathrm{KL}} \right]$$Li​(θ)=G1​g=1∑G​∣yg​∣1​t=1∑∣yg​∣​[−min{πθold​​(yg,t​∣x,yg,<t​)πθ​(yg,t​∣x,yg,<t​)​Ag​,clip(πθold​​(yg,t​∣x,yg,<t​)πθ​(yg,t​∣x,yg,<t​)​,1−ϵl​,1+ϵh​)Ag​}+βdg,tKL​]

Faithful-RFT suppresses unnecessary explicit think traces rather than encouraging visible reasoning traces.

The rollout strategy extends the synchronized length-grouped sampler with task-aware bucketing. Examples are assigned to buckets defined by modality and task identifier, ordered by sequence length to reduce input-side padding. A lightweight dynamic resampling strategy retains groups with nonzero reward variance and regenerates candidate groups in cases of near-zero variance to ensure meaningful relative advantages.

The scalar reward required for optimization is computed through task-conditioned reward routing. Each reward function declares an applicable task set and is evaluated only when the task matches. The configured weights are normalized over the valid rewards for each candidate:

$$\widetilde{w}_{i,g,j} = \begin{cases} \frac{v_{i,g,j} w_j}{\sum_k v_{i,g,k} w_k}, & \sum_k v_{i,g,k} w_k > 0, \\ 0, & \text{otherwise.} \end{cases}$$wi,g,j​={∑k​vi,g,k​wk​vi,g,j​wj​​,0,​∑k​vi,g,k​wk​>0,otherwise.​

The final scalar reward is $R_{i,g} = \sum_{j: v_{i,g,j}=1} \widetilde{w}_{i,g,j} r_{i,g,j}$Ri,g​=∑j:vi,g,j​=1​wi,g,j​ri,g,j​. This ensures each example is scored only by reward functions applicable to its task, utilizing rule-based rewards for deterministic targets and LLM-judge rewards for open-ended responses.

Experiment

TLive-Omni is evaluated on a comprehensive suite of live-commerce tasks spanning speech transcription, speaker-attributed ASR, product visual grounding, temporal grounding, dense video captioning, and shot understanding, alongside an extensive set of general multimodal benchmarks that assess image reasoning, hallucination, OCR, video understanding, and omni-modal perception. The model achieves strong performance on both the domain-specific and general evaluations, indicating that its specialization for live commerce does not narrow its broad multimodal understanding. Qualitative examples further demonstrate the model's ability to handle nuanced audio-visual-temporal reasoning in live-commerce and open-domain scenarios.

The live-commerce audio results show that Gemini Pro models generally outperform Flash models among the listed closed-source systems, with Gemini 3 Pro leading on speaker-attributed ASR, audio description, hallucination, and audio QA, while Gemini 2.5 Pro has the lowest ASR error. The accompanying text reports TLive-Omni-9B achieves the lowest reported CER on ASR, with TLive-Omni-4B close behind and both showing competitive cpWER. Overall, stronger performance is concentrated in Pro-scale models and the TLive-Omni family for live-commerce speech and audio understanding. Gemini 2.5 Pro records the lowest live-commerce ASR error among the closed-source Gemini models, ahead of Gemini 3 Pro and Gemini 3.5 Flash. TLive-Omni-9B achieves the lowest ASR CER in the reported comparison, with TLive-Omni-4B close behind, and both models show low speaker-attributed ASR cpWER. Gemini 3 Pro leads the listed closed-source models on speaker-attributed ASR, audio description accuracy, hallucination rate, and audio QA. Gemini 2.5 Flash and Gemini 3 Flash trail the Pro models on audio description accuracy and hallucination rate, with notably higher unsupported content rates. Audio QA accuracy among closed-source Gemini models is highest for Gemini 3 Pro, followed by Gemini 3.5 Flash, while Gemini 3 Flash ranks lowest.

TLive-Omni variants lead on product-image grounding and product text understanding, including text localization, recognition, and classification, while remaining competitive with the strongest closed-source model on live-stream product grounding. Among reported closed-source models, Gemini 3.5 Flash shows the highest live-stream grounding, but TLive-Omni is stronger on several product-image and text-oriented metrics. The results indicate broad multimodal capability beyond the live-commerce domain. TLive-Omni variants achieve the best product-image grounding, text localization, and text classification among evaluated models, along with the lowest recognition edit distances. Gemini 3.5 Flash leads live-stream product grounding among closed-source models, while TLive-Omni remains competitive on that metric.

Among the reported Gemini models, capabilities are uneven across live-commerce video tasks. Gemini 2.5 Flash shows the strongest dense caption accuracy and lowest hallucination, while Gemini 2.5 Pro leads video question answering and most shot understanding scores. Gemini 3 and 3.5 models generally improve temporal grounding but do not consistently improve caption quality or hallucination control. Gemini 2.5 Flash reports the highest dense caption accuracy and the lowest hallucination rate among listed models. Gemini 2.5 Pro leads video question answering accuracy as well as layout and shot size metrics. Gemini 3 Pro and Gemini 3.5 Flash tie for the highest temporal grounding mIoU, modestly above Gemini 2.5 models. Shot understanding is mixed: camera understanding is led by Gemini 2.5 Flash, while content understanding is led by Gemini 3 Pro.

General image benchmark results show that closed-source models, especially Gemini 3 Pro, lead most reported image-centric metrics, while Qwen3.5-Omni Flash performs competitively across several math and MMBench tasks. TLive-Omni is described as retaining strong image reasoning and improving over its Qwen3.5 4B and 9B backbones on the majority of these benchmarks. The 9B variant is reported to achieve best open-source results on many metrics, with the 4B variant often ranking first or second among open-source models. Gemini 3 Pro achieves the highest reported scores among listed models on most image-centric benchmarks, including MMMU, MathVista, DynaMath, MMBench, RealWorldQA, MMStar, and SimpleVQA. Qwen3.5-Omni Flash outperforms Gemini 2.5 Flash on MMMU, MathVista, DynaMath, MMBench, and RealWorldQA, but falls behind on SimpleVQA and slightly on MMStar. TLive-Omni is reported to improve over Qwen3.5 4B and 9B backbones on a majority of general image benchmarks, with its 9B variant achieving best open-source results on many metrics.

The general image benchmark results span hallucination, diagram and chart understanding, OCR perception, visual grounding, embodied reasoning, and spatial reasoning. TLive-Omni maintains competitive general image reasoning, with particular strength in hallucination, OCR-centric perception, and spatial reasoning. Across model sizes, the 9B variant achieves the best open-source results on many reported metrics, while the 4B variant also reaches best or second-best open-source results on multiple benchmarks. TLive-Omni shows particular strength on hallucination, OCR-centric perception, and spatial reasoning tasks relative to its broader image benchmark profile. The 9B variant leads open-source models on many reported image benchmarks, while the 4B variant is often best or second-best among open-source models.

Across live-commerce audio, product-image and text understanding, live-commerce video, and general image benchmarks, the evaluations compare TLive-Omni variants with Gemini and Qwen models. TLive-Omni and Gemini Pro-scale models lead in live-commerce audio and product-oriented tasks, with TLive-Omni showing particular strength in ASR, product-image grounding, text localization, recognition, and classification, while Gemini 3 Pro leads several closed-source audio metrics. Live-commerce video results are mixed, with Gemini 2.5 Flash excelling at dense captioning and hallucination control, Gemini 2.5 Pro leading video QA and shot understanding, and newer Gemini models only improving temporal grounding. On general image benchmarks, Gemini 3 Pro leads most closed-source results, while TLive-Omni improves over its Qwen backbones and achieves best open-source performance on many metrics, especially hallucination, OCR-centric perception, and spatial reasoning.