Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding

Understanding animals' behaviors is significant for a wide range ofapplications. However, existing animal behavior datasets have limitations inmultiple aspects, including limited numbers of animal classes, data samples andprovided tasks, and also limited variations in environmental conditions andviewpoints. To address these limitations, we create a large and diversedataset, Animal Kingdom, that provides multiple annotated tasks to enable amore thorough understanding of natural animal behaviors. The wild animalfootages used in our dataset record different times of the day in extensiverange of environments containing variations in backgrounds, viewpoints,illumination and weather conditions. More specifically, our dataset contains 50hours of annotated videos to localize relevant animal behavior segments in longvideos for the video grounding task, 30K video sequences for the fine-grainedmulti-label action recognition task, and 33K frames for the pose estimationtask, which correspond to a diverse range of animals with 850 species across 6major animal classes. Such a challenging and comprehensive dataset shall beable to facilitate the community to develop, adapt, and evaluate various typesof advanced methods for animal behavior analysis. Moreover, we propose aCollaborative Action Recognition (CARe) model that learns general and specificfeatures for action recognition with unseen new animals. This method achievespromising performance in our experiments. Our dataset can be found athttps://sutdcv.github.io/Animal-Kingdom.