PSG is a large-scale dataset with fine annotations. The dataset contains 48,749 COCO images, and based on the existing panoramic segmentation annotations of COCO, the relationship between the segmented blocks is annotated. The dataset contains 133 object categories, of which 80 are objects and 53 are stuff.

The authors carefully define 56 types of relationships, including positional relationships (over, in front of, etc.), common object relationships (hanging from, etc.), common biological actions (walking on, standing on, etc.), human behaviors (cooking, etc.), relationships in traffic scenes (driving, riding, etc.), relationships in sports scenes (kicking, etc.), and background relationships (enclosing, etc.).