H2O: Two Hands Manipulating Objects for First Person Interaction Recognition

We present a comprehensive framework for egocentric interaction recognitionusing markerless 3D annotations of two hands manipulating objects. To this end,we propose a method to create a unified dataset for egocentric 3D interactionrecognition. Our method produces annotations of the 3D pose of two hands andthe 6D pose of the manipulated objects, along with their interaction labels foreach frame. Our dataset, called H2O (2 Hands and Objects), providessynchronized multi-view RGB-D images, interaction labels, object classes,ground-truth 3D poses for left & right hands, 6D object poses, ground-truthcamera poses, object meshes and scene point clouds. To the best of ourknowledge, this is the first benchmark that enables the study of first-personactions with the use of the pose of both left and right hands manipulatingobjects and presents an unprecedented level of detail for egocentric 3Dinteraction recognition. We further propose the method to predict interactionclasses by estimating the 3D pose of two hands and the 6D pose of themanipulated objects, jointly from RGB images. Our method models both inter- andintra-dependencies between both hands and objects by learning the topology of agraph convolutional network that predicts interactions. We show that our methodfacilitated by this dataset establishes a strong baseline for joint hand-objectpose estimation and achieves state-of-the-art accuracy for first personinteraction recognition.