Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape

Accurately estimating the 3D pose and shape is an essential step towardsunderstanding animal behavior, and can potentially benefit many downstreamapplications, such as wildlife conservation. However, research in this area isheld back by the lack of a comprehensive and diverse dataset with high-quality3D pose and shape annotations. In this paper, we propose Animal3D, the firstcomprehensive dataset for mammal animal 3D pose and shape estimation. Animal3Dconsists of 3379 images collected from 40 mammal species, high-qualityannotations of 26 keypoints, and importantly the pose and shape parameters ofthe SMAL model. All annotations were labeled and checked manually in amulti-stage process to ensure highest quality results. Based on the Animal3Ddataset, we benchmark representative shape and pose estimation models at: (1)supervised learning from only the Animal3D data, (2) synthetic to real transferfrom synthetically generated images, and (3) fine-tuning human pose and shapeestimation models. Our experimental results demonstrate that predicting the 3Dshape and pose of animals across species remains a very challenging task,despite significant advances in human pose estimation. Our results furtherdemonstrate that synthetic pre-training is a viable strategy to boost the modelperformance. Overall, Animal3D opens new directions for facilitating futureresearch in animal 3D pose and shape estimation, and is publicly available.