One-sentence Summary

Amazon and UCF propose ViLL-E, a unified Video-LLM architecture with an adaptive embedding generation mechanism that "thinks longer" for complex videos and stops early for easy ones, trained via a three-stage joint generative-contrastive pipeline that significantly improves temporal localization and video retrieval while unlocking zero-shot composed retrieval and long-text retrieval capabilities.

Key Contributions

• ViLL-E extends a VideoLLM with an adaptive embedding-generation mechanism that supports text-to-video retrieval and temporal localization while preserving text-generation capabilities. This mechanism allows the model to process more reasoning steps for complex videos and terminate earlier for simpler videos.

• A three-stage generative-contrastive training protocol is developed: large-scale video-caption pre-training, continual training on detailed-caption data, and task-specific fine-tuning on a multi-task dataset for VideoQA, temporal localization, video retrieval, and video-text matching. This protocol enables one model to maintain strong text generation and produce strong multimodal embeddings.

• ViLL-E improves temporal localization by about 7% on average over other VideoLLMs and improves video retrieval by up to 4% over dual-encoder models, while remaining competitive on VideoQA. Its joint training also enables zero-shot composed video retrieval and retrieval from long text, exceeding prior state-of-the-art methods by about 5% and about 2%, respectively.

Introduction

Video Large Language Models extend Large Language Models to video by aligning visual features with language models, enabling tasks such as captioning and question answering. This approach matters because video understanding is central to search, moderation, assistant systems, and temporal reasoning.

Existing VideoLLMs are mainly optimized for text generation, but important tasks like text-to-video retrieval and temporal localization require discriminative video-text embeddings. Prior retrieval and localization models can be strong, yet they are usually task-specific or dataset-specific and do not share a single generative backbone with VideoLLMs.

The authors introduce ViLL-E, a unified VideoLLM that supports both text generation and embedding generation. It uses a learnable attentive-pooling embedding head, joint generative and contrastive training, and a three-stage training pipeline. ViLL-E remains competitive on video question answering, improves temporal localization over VideoLLMs, and achieves strong video retrieval performance.

Dataset

• The authors build a three-stage training data pipeline using licensed Shutterstock stock videos as the primary source, supplemented by established video-text datasets for fine-tuning.

• For large-scale pre-training, they use video-caption pairs from the licensed Shutterstock dataset, which contains 10M unique captions. By keeping one random clip per caption, they obtain 10M video-caption pairs, a scale similar to WebVid-10M.

• For high-quality intermediate training, they use a re-captioned subset of Shutterstock. They filter the original keyword labels by removing keywords with fewer than 30 occurrences, then add up to 500 videos per keyword to the candidate pool, prioritizing frequent keywords.

• To create a more diverse and higher-quality caption set within budget constraints, they generate captions for 200,000 videos. They use Claude-3-Sonnet with 20 sampled frames per video, each resized to 480 pixels in height, and a prompt requesting detailed narrative captions covering theme, visuals, characters, actions, scenes, and key moments.

• This filtering and re-captioning process is intended to balance the dataset and reduce over-representation of common concepts, following the idea that under-sampling frequent categories can improve generalization on rare visual concepts.

• For fine-tuning, they assemble a multi-task dataset of 100,000 samples drawn from a small number of video-text datasets. These samples support video captioning, retrieval, matching, and other instruction-style supervision.

• MSR-VTT contributes short video captions and question-answering supervision. ActivityNet and DiDeMo contribute longer videos with temporally localizable captions for video chunks, enabling localization supervision. ActivityNet additionally provides question-answering annotations.

• To preserve long-tail diversity, the authors mix in a random subset of original Shutterstock video-caption pairs alongside the curated fine-tuning data.

Method

The authors leverage a unified multimodal video language model that supports both generative outputs and dense embeddings for retrieval-oriented tasks. The system is initialized from a pretrained multimodal language model, specifically PaliGemma 3B, and is extended with a vision processing branch, a low rank adaptation path for efficient training, and a separate embedding head. The overall pipeline processes video frames through a frozen vision encoder, projects the extracted visual features into the language model space, and concatenates them with text prompt embeddings. The language model then performs autoregressive generation and, after an end of sequence token is produced, passes the relevant generated representations to the embedding head to form a compact video embedding. Refer to the architecture diagram.

The backbone follows a prefix language model style attention pattern. Visual tokens and input text prompts are processed with bidirectional attention, while the model generated suffix is processed with causal attention. This design preserves strong language modeling behavior for tasks such as video question answering and captioning, while also allowing the generated tokens to serve as a variable length representation for embedding extraction. A language modeling head produces the text output, and the special end of sequence token acts as a signal that the model has completed its reasoning trace and that the subsequent embedding pooling operation can begin.

The embedding head is one of the key components of the system. It is designed to be parameter efficient, to provide a bottleneck into a dense embedding space, and to adapt to videos of different complexity. Instead of using a fixed pooling scheme or a fixed set of learned queries, the authors introduce a KV-Former style attentive pooling module. The module takes final-layer hidden states associated with the generated tokens and uses them as queries. A set of $P$P learnable key and value pooling tokens are then used to aggregate information from the query tokens through an attention operation. The attended representations are projected through a multilayer perceptron and mean pooled to produce the final video embedding. This mechanism gives the model flexibility because the number of generated tokens can vary with the difficulty of the video. Complex videos can elicit longer reasoning traces, while simpler videos can produce shorter outputs before the end of sequence token.

Text embeddings are generated through a simpler path. Because text inputs do not require variable length autoregressive reasoning in the same way, the model uses standard inference to produce a single text embedding. This asymmetry keeps video embeddings adaptive while maintaining efficiency for text-side retrieval inputs.

Training is organized as a multi-stage, multi-task process. In the first stage, the authors perform generative-contrastive pretraining on large video-caption data. The model is trained with two complementary supervision signals. For each video-caption pair, the first forward pass processes the video frames together with a prompt asking the model to describe the video. This pass simultaneously produces the generated caption and the video embedding. A second, lighter forward pass processes the text input alone to produce the text embedding. The generation part is supervised with a next-token prediction loss, while the embedding part is supervised with a CLIP-style contrastive objective that aligns matched video and text embeddings and separates them from in-batch negatives. The combined objective can be written as

$$\mathcal{L} = \mathcal{L}_{gen} + \lambda \mathcal{L}_{ctr},$$L=Lgen​+λLctr​,

where $\mathcal{L}_{gen}$Lgen​ encourages fluent and contextually grounded video descriptions, and $\mathcal{L}_{ctr}$Lctr​ strengthens cross-modal alignment.

The second stage is continued pretraining on a smaller set of higher quality captions. The authors balance keyword coverage and under-sample overly common concepts so that the model preserves visual detail and generalizes to rarer content. This stage is important because the first-stage captions can be short or noisy, while dense descriptive captions help preserve fine-grained evidence in the learned representation space. The benefit of replacing terse original annotations with more detailed generated captions is illustrated in the caption comparison image below.

The final stage is multi-task fine-tuning. The training data combine supervision from video captioning, question answering, text-video retrieval, text-video matching, and temporal localization. During this stage, the language model backbone is adapted with low rank adapters, while the vision projection module and the embedding head are fully trained. This setup allows the model to retain pretrained generative competence while learning task-specific retrieval behavior with strong parameter efficiency.

The retrieval objective is contrastive and encourages each video-text pair to rank above negative pairs. Captioning and question answering are trained with next-token prediction, where the prompt guides the model toward the requested answer or description. The matching task adds a binary alignment signal, teaching the model to decide whether a given video and text form a true pair. Temporal localization is trained with a contrastive formulation over segment windows, where a positive window is contrasted with hard negatives mined by sliding windows that have limited temporal overlap, for example intersection over union below $0.2$0.2. This forces the model to distinguish visually similar but semantically incorrect time intervals. Across all stages, the pipeline connects video understanding, language generation, adaptive embedding generation, and retrieval-oriented contrastive learning in a single trainable architecture.

Experiment

The experiments evaluate a unified video language model across retrieval, localization, and question answering benchmarks, then probe novel tasks enabled by joint generative and embedding capabilities. The results show that the model delivers strong video understanding, competitive or leading performance on many retrieval and QA settings, and supports composed retrieval and re-ranking workflows. Ablations indicate that combining generative and contrastive supervision, using an effective embedding head, and pretraining at scale are all important for maintaining accuracy across both embedding and generation tasks.

The comparison shows a clear gap between embedding based retrieval models and LLM based VideoLLMs. ViLL-E unifies both by producing embeddings while retaining generation, allowing it to match retrieval focused expert models. It remains competitive on generative video question answering tasks, though stronger general VideoLLMs can still lead on some generation benchmarks. Expert retrieval models provide embeddings but lack LLM generation General VideoLLMs provide generation but lack embeddings needed for retrieval Text-to-video retrieval exposes a limitation of standard VideoLLMs Two-stage retrieval with VideoLLM re-ranking adds a small improvement over embedding retrieval Zero-shot composed video retrieval outperforms recent baselines Contrastive supervision is important for retrieval and localization quality

The dataset collection covers video-language tasks spanning captioning, question answering, and localization across four sources. ActivityNet provides the broadest task coverage, while MSR-VTT focuses on captioning and QA, DiDeMo on captioning and localization, and Shutterstock mainly supplies captions. The listed sources differ in scale and annotation emphasis, supporting a mixed training setup. ActivityNet is the most task-diverse subset, including caption, QA, and localization labels. MSR-VTT supports caption and QA tasks but not localization in this dataset collection. DiDeMo contributes caption and localization supervision but lacks QA labels. Shutterstock is included with a relatively large sample but is listed as caption-only for this collection.

The model shows strong unified retrieval performance across both moment retrieval and text-video retrieval settings. It outperforms localization-specialized Video LLMs on moment retrieval and remains competitive with dataset-specific expert models. For video retrieval, it approaches or exceeds fine-tuned retrieval baselines across standard benchmarks. Outperforms localization-specialized Video LLMs on moment retrieval tasks. Approaches or surpasses dataset-specific expert models on moment retrieval. Matches or exceeds fine-tuned image and video retrieval models on text-video retrieval benchmarks.

ViLL-E is presented as competitive with state-of-the-art VideoLLMs on video question answering across both simple QA benchmarks and comprehensive video understanding evaluations. The comparison shows that general VideoLLMs can lead on comprehensive benchmarks, while temporally specialized models remain strong on focused QA datasets. Overall, ViLL-E maintains competitive performance across QA-style tasks. ViLL-E is competitive with strong VideoLLMs across simple QA and comprehensive QA benchmarks. General VideoLLMs show advantages on comprehensive benchmarks, but ViLL-E remains competitive overall. Temporally specialized models perform strongly on specific QA tasks, indicating benchmark-dependent strengths.

The two-stage retrieval and matching procedure improves over single-stage embedding retrieval on MSR-VTT by using VideoLLM matching loss to re-rank top candidates. The gains are most visible in recall metrics, indicating that generative reasoning can complement embedding-based video caption retrieval. Two-stage re-ranking gives a small top-accuracy gain over single-stage retrieval for text-to-video matching. It improves video-to-text recall, especially at the broader recall threshold. The method surpasses the VidLA baseline on selected two-stage MSR-VTT recall scores.

The experiments evaluate a unified video-language model over retrieval, moment localization, and generation-oriented question answering, using mixed video-language data with varied annotation emphasis. They show that the model can provide retrieval-quality embeddings while retaining generation, performing well on both moment and text-video retrieval compared with specialized experts and general VideoLLMs. Additional analyses on video QA and two-stage reranking indicate competitive generation capability and that VideoLLM-based re-ranking can modestly complement embedding retrieval.