SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries

Retrieving unlabeled videos by textual queries, known as Ad-hoc Video Search(AVS), is a core theme in multimedia data management and retrieval. The successof AVS counts on cross-modal representation learning that encodes both querysentences and videos into common spaces for semantic similarity computation.Inspired by the initial success of previously few works in combining multiplesentence encoders, this paper takes a step forward by developing a new andgeneral method for effectively exploiting diverse sentence encoders. Thenovelty of the proposed method, which we term Sentence Encoder Assembly (SEA),is two-fold. First, different from prior art that use only a single commonspace, SEA supports text-video matching in multiple encoder-specific commonspaces. Such a property prevents the matching from being dominated by aspecific encoder that produces an encoding vector much longer than otherencoders. Second, in order to explore complementarities among the individualcommon spaces, we propose multi-space multi-loss learning. As extensiveexperiments on four benchmarks (MSR-VTT, TRECVID AVS 2016-2019, TGIF and MSVD)show, SEA surpasses the state-of-the-art. In addition, SEA is extremely ease toimplement. All this makes SEA an appealing solution for AVS and promising forcontinuously advancing the task by harvesting new sentence encoders.