The ability to understand the ways to interact with objects from visual cues,a.k.a. visual affordance, is essential to vision-guided robotic research. Thisinvolves categorizing, segmenting and reasoning of visual affordance. Relevantstudies in 2D and 2.5D image domains have been made previously, however, atruly functional understanding of object affordance requires learning andprediction in the 3D physical domain, which is still absent in the community.In this work, we present a 3D AffordanceNet dataset, a benchmark of 23k shapesfrom 23 semantic object categories, annotated with 18 visual affordancecategories. Based on this dataset, we provide three benchmarking tasks forevaluating visual affordance understanding, including full-shape, partial-viewand rotation-invariant affordance estimations. Three state-of-the-art pointcloud deep learning networks are evaluated on all tasks. In addition we alsoinvestigate a semi-supervised learning setup to explore the possibility tobenefit from unlabeled data. Comprehensive results on our contributed datasetshow the promise of visual affordance understanding as a valuable yetchallenging benchmark.