Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning

We present a simple approach which can turn a ViT encoder into an efficientvideo model, which can seamlessly work with both image and video inputs. Bysparsely sampling the inputs, the model is able to do training and inferencefrom both inputs. The model is easily scalable and can be adapted tolarge-scale pre-trained ViTs without requiring full finetuning. The modelachieves SOTA results and the code will be open-sourced.