One-sentence Summary

The Kling Team presents Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio-video, and interleaved inputs, and analyzes the data-conditioning-alignment-evaluation chain by introducing reference-grounded captioning, prompt enhancement with generator-in-the-loop reinforcement learning, perception-trained text encoders, and paired reward models to improve controllable video generation.

Key Contributions

• Kling-MLLM is a shared multimodal foundation model that unifies image, video, audio, synchronized audio-video, and interleaved inputs and is specialized for each downstream module in the data-conditioning-alignment-evaluation chain.
• Reference-grounded captioning extends captioning into a dense semantic contract with explicit reference bindings, using pointers, supplementary descriptions, and scoped preservation or change records. Prompt enhancement translates user intent into this caption distribution, with basic prompt enhancement refined by text-level and generator-in-the-loop reinforcement learning and agentic prompt enhancement refining the story and retrieving compelling examples.
• Reliable text encoders come from perceptual training rather than stronger language ability and can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable localized critiques with a decision reward model for fast multidimensional scoring, and task-specific automatic evaluation protocols expose scoring shortcuts and trace failures upstream.

Introduction

The authors examine video generation as it shifts from clip-level fidelity to sequence-level control, where camera language, identity persistence, reference binding, and temporal continuity must hold across multiple shots. Prior work often treats multimodal understanding as broad benchmark performance, but VQA-style scores do not reliably indicate whether the right semantics reach the generator, and failures can arise from caption omissions, conditioning bottlenecks, reward shortcuts, or weak evaluators. Drawing on Kling 1.0 through 4.0 and related systems, the authors argue that video generation quality is the projection of multimodal understanding through the entire data-conditioning-alignment-evaluation chain. They specialize Kling-MLLM into captioners, prompt enhancement models, text encoders, reward models, and evaluators to make that chain explicit and traceable.

Dataset

Dataset sources and composition

• Public data: The authors continuously collect public corpora by monitoring Hugging Face Daily Papers and trending Hugging Face datasets. Candidate datasets are extracted from work on text, image, video, audio, audio-video, and multimodal foundation models. Sources are merged and deduplicated against each other and against a historical registry.
• Kling-specific task data: Internal data target image and video generation and editing requirements. They include interleaved image-text world knowledge, bilingual OCR, counting, grounding, metadata-aware description, pose understanding, dense image and video captioning, and editing/reference examples with paired source-target images or videos.
• Multimodal agent data: The mixture covers general tool use and multimodal perception-to-action tasks. Examples include API and transactional workflows, web research, software engineering, terminal interaction, data transformation, safety assistance, image-grounded retrieval, GUI interaction, agentic video understanding, and Blender-based 3D tasks. Supervision ranges from complete tool-use loops to partial trajectories and direct responses, and some examples are model generated with consistency checks.

Processing and schema

• All heterogeneous sources are represented through a common conversation-oriented schema. Media metadata and modality boundaries are retained so a single loader can construct image, video, audio, synchronized audio-video, and interleaved examples.
• Public data follow an AutoResearch, Auto Data Process, Auto Revise workflow. AutoResearch discovers and filters candidate datasets. Auto Data Process downloads datasets and applies modality-specific conversion. Auto Revise performs coarse formatting and diversity checks, fine perplexity checks with Qwen3-Omni, multimodal review with Gemini, and human review. Accepted data are assigned to a training stage; low-quality data are removed or repaired and revalidated.
• Captioning follows a semantic contract that emphasizes explicit entity naming, shot-boundary timestamps, faithful OCR transcription, and temporal integration of audio. The caption supervision pipeline includes inspection and routing, perceptual evidence preservation, routed expert claims, integration and conflict resolution, and quality gating.
• For reference-grounded captions, the authors pair videos with sufficient visual or acoustic similarity. One video acts as the reference and another as the target. The caption states what is shared and what must change, which turns arbitrary similar video pairs into training examples even without original edit instructions.
• Prompt-enhancement data are built backward from real captions. A teacher model writes a user prompt that could have produced a caption, and the system learns the mapping from prompt to caption. Prompts are sampled for length, language mixture, register, typo rate, omitted slots, references, multi-shot requests, and exclusions. Prompts that leak the caption structure are rejected. Source captions are selected from compelling videos using aesthetic, motion, and vividness scores with domain and length quotas.

How the data are used

• Public data, Kling-specific task data, and multimodal agent data are combined into the training corpus and assigned to appropriate training stages.
• Standard captions and reference-conditioned examples are used together. A single target video can contribute multiple reference-conditioned examples by pairing it with different references.
• Real captions serve as training targets for prompt enhancement, helping align user prompts with caption format, lexicon, and sentence patterns.
• The provided excerpt does not report exact dataset counts or mixture ratios; the construction is described as production-scale and continuous.

Method

The authors leverage a multimodal understanding foundation model, Kling-MLLM, which serves as a shared perception and reasoning backbone. The model family comprises dense and mixture-of-experts variants. At the input interface, every supported stream is mapped to a common sequence consumed by a single large language model.

Refer to the framework diagram:

The visual path consists of a Vision Transformer, spatial pooling, and a vision aligner. Images are processed at dynamic resolution while preserving aspect ratios. Videos use dynamically selected frame rates and counts, with each visual-token block associated with its frame physical timestamp. The audio path uses scale-dependent towers. Larger variants employ a dual-branch tower that fuses Mel spectrogram and waveform-VAE features. A learned scalar gate adds aligned VAE features residually to Mel features, and the output is projected to the LLM hidden width. Modality-specific boundary tokens make each span self-delimiting, and absolute timestamps provide temporal anchors. Synchronized audio and video are serialized jointly within one element, preserving the original order within each modality.

To bridge the gap between user requests and the dense caption distribution the generator is trained on, the authors design a Prompt Enhancement system split into two modules: Basic Prompt Enhancement and Agentic Prompt Enhancement.

As shown in the figure below:

Basic Prompt Enhancement is a trained rewriter that produces the final condition in the caption dialect. It is trained through three stages. The first stage uses caption-aligned supervised fine-tuning with backward-constructed prompts from compelling videos. The second stage applies text-level reinforcement learning to optimize faithfulness, reasonableness, and vividness. The third stage performs reinforcement learning with the diffusion transformer in the loop using video-level rewards. To ensure the output matches the generator training distribution, alignment is required at the format, lexicon, and sentence pattern levels.

Refer to the framework diagram:

Agentic Prompt Enhancement runs before the basic module at inference. It performs story refinement to resolve plot contradictions in user prompts and retrieves top captions of compelling videos as creative evidence. The basic module then writes the final condition, treating retrieved captions as optional evidence.

To provide an optimization signal for post-training and inference-time selection, the authors build Reward Models split into a General Reward Model for explainable evaluation and a Decision Reward Model for fast multi-dimensional scoring. The training pipeline progresses through three stages.

As shown in the figure below:

Stage 1 focuses on visual adaptation for low-level artifact perception by freezing the language model and optimizing the vision encoder and aligner. Stage 2 trains the General Reward Model using cold-start supervised fine-tuning and reinforcement learning with verifiable rewards to produce structured critiques and localized defect evidence. Stage 3 initializes the Decision Reward Model from the General Reward Model. It trains the model to answer multiple fine-grained questions in a single prefill using a decision-head warm-up, unified pointwise and pairwise supervised fine-tuning, and symmetry calibration for fast inference.

Experiment

The experiments evaluate multimodal understanding, captioning, reference-grounded generation, prompt enhancement, text encoders, reward models, and automatic evaluation protocols. Kling-MLLM-397B-A17B shows the clearest gains in video perception, temporal understanding, audio temporal grounding, and audio-video captioning, while image, speech recognition, and interleaving results remain mixed. Kling-Captioner and Kling-RGC outperform baselines on downstream captioning and reference-binding tasks, and cumulative text-encoder refinements improve preference, instruction following, and aesthetics. The work further argues that evaluation should rely on atomic source-grounded criteria, downstream generative validity, and diagnostic per-dimension reliability rather than surface lexical scores.

The data pipelines define modality-specific normalization and validation checks alongside representative supervision objectives for text, image, video, audio, audio-video, and interleaved media. Validation ranges from language and deduplication checks for text to synchronization and media-reference checks for audio-video and interleaved content. Supervision similarly spans next-token prediction, captioning and VQA, temporal and acoustic-event reasoning, and contextual QA over multiple media items. Text and image pipelines focus on encoding, deduplication, and media-level checks, while video and audio pipelines add temporal segmentation, signal-quality checks, and timestamp or transcript alignment. Supervision objectives shift from single-modality tasks such as text instruction following and image VQA to cross-modal tasks such as joint transcription, audio-video grounding, and contextual reasoning over interleaved media.

Audio evaluation shows a mixed leaderboard across speech recognition, audio understanding, and temporal grounding. Kling-MLLM variants lead on clean LibriSpeech, Chinese LATBench, TREA, English temporal grounding, and English FENSE, while Doubao2.0-Lite leads on LibriSpeech other and WorldSense. The LATBench Chinese and English results often diverge within individual models. Kling-MLLM-397B-A17B achieves the lowest LibriSpeech clean word error rate and the strongest Chinese LATBench results among evaluated models. Kling-MLLM-27B records the strongest TREA and English LATBench temporal-grounding results, and Kling-MLLM-35B-A3B leads English LATBench FENSE. Doubao2.0-Lite variants achieve the best LibriSpeech other and WorldSense scores, indicating stronger performance in those areas.

Performance on synchronized audio-video benchmarks is distributed across models: Qwen3.5-Omni-Plus has the lowest video-SALMONN miss rate, MiniCPM-o-4.5 has the lowest hallucination rate, and Kling-MLLM-397B-A17B has the lowest total error while leading OmniCap-IF. Doubao2.0-Lite (thinking, high) leads UGC-VideoCap, and Kling-MLLM-397B-A17B is the strongest Kling variant on multimodal interleaving, with IV-Bench and MVU-Eval results within about half a point of the best reported scores. Qwen3.5-Omni-Plus achieves the lowest video-SALMONN miss rate, while MiniCPM-o-4.5 achieves the lowest hallucination rate. Kling-MLLM-397B-A17B records the lowest video-SALMONN total error and leads both OmniCap-IF metrics. Doubao2.0-Lite (thinking, high) leads UGC-VideoCap. Kling-MLLM-397B-A17B is the strongest Kling model on IV-Bench and MVU-Eval, scoring within 0.36 and 0.55 points of the best reported results.

Qwen3.5-Omni-Plus leads internally evaluated MMBench, Qwen3.5-397B-A17B leads OCRBench, and Qwen3-Omni-30B-A3B-Instruct leads internal TextPecker among the listed image models. PerceptionBench submetrics remain mixed across English and Chinese, and MiniCPM-o-4.5 trails on MMBench, OCRBench, and its internal TextPecker result. Qwen3.5-Omni-Plus has the highest internally evaluated MMBench score, while Qwen3.5-397B-A17B has the highest OCRBench score. Qwen3-Omni-30B-A3B-Instruct leads internally evaluated TextPecker, and MiniCPM-o-4.5 records the lowest MMBench and OCRBench scores among the listed models.

Across the video benchmarks, Qwen3.5-Omni-Plus is the most consistently strong model, leading on LongVideo Bench, MVBench, MME-VideoOCR, ShortVid-Bench, DSI-Bench, Event-Bench, and Motion-Bench. Qwen3-VL-32B is strongest on VSI-Bench, TimeLens, and CameraMotion VQA, while the remaining models are less competitive overall. Qwen3.5-Omni-Plus leads the majority of video understanding metrics, including MVBench, MME-VideoOCR, and Motion-Bench. Qwen3-VL-32B leads VSI-Bench, TimeLens, and CameraMotion VQA, indicating relative strength in temporal and camera motion tasks.

The experiments evaluate multi-modal data pipelines across text, image, video, audio, audio-video, and interleaved media, using modality-specific validation checks and supervision objectives that range from language deduplication to temporal alignment and cross-modal reasoning. Audio and synchronized audio-video leaderboards remain mixed: Kling-MLLM variants are strong in speech recognition, temporal grounding, and several audio-video metrics, while Doubao2.0-Lite leads in robustness-related audio tasks and Qwen3.5-Omni-Plus and MiniCPM-o-4.5 show strengths on specific error and hallucination measures. Image evaluation shows Qwen3.5-Omni-Plus and Qwen3.5-397B-A17B leading on MMBench and OCRBench respectively with MiniCPM-o-4.5 trailing, whereas video evaluation identifies Qwen3.5-Omni-Plus as the most consistently strong model and Qwen3-VL-32B as strongest on temporal and camera-motion tasks.