OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Many image-based perception tasks can be formulated as detecting, associatingand tracking semantic keypoints, e.g., human body pose estimation and tracking.In this work, we present a general framework that jointly detects and formsspatio-temporal keypoint associations in a single stage, making this the firstreal-time pose detection and tracking algorithm. We present a generic neuralnetwork architecture that uses Composite Fields to detect and construct aspatio-temporal pose which is a single, connected graph whose nodes are thesemantic keypoints (e.g., a person's body joints) in multiple frames. For thetemporal associations, we introduce the Temporal Composite Association Field(TCAF) which requires an extended network architecture and training methodbeyond previous Composite Fields. Our experiments show competitive accuracywhile being an order of magnitude faster on multiple publicly availabledatasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. Wealso show that our method generalizes to any class of semantic keypoints suchas car and animal parts to provide a holistic perception framework that is wellsuited for urban mobility such as self-driving cars and delivery robots.