MarrNet: 3D Shape Reconstruction via 2.5D Sketches

3D object reconstruction from a single image is a highly under-determinedproblem, requiring strong prior knowledge of plausible 3D shapes. Thisintroduces challenges for learning-based approaches, as 3D object annotationsare scarce in real images. Previous work chose to train on synthetic data withground truth 3D information, but suffered from domain adaptation when tested onreal data. In this work, we propose MarrNet, an end-to-end trainable model thatsequentially estimates 2.5D sketches and 3D object shape. Our disentangled,two-step formulation has three advantages. First, compared to full 3D shape,2.5D sketches are much easier to be recovered from a 2D image; models thatrecover 2.5D sketches are also more likely to transfer from synthetic to realdata. Second, for 3D reconstruction from 2.5D sketches, systems can learnpurely from synthetic data. This is because we can easily render realistic 2.5Dsketches without modeling object appearance variations in real images,including lighting, texture, etc. This further relieves the domain adaptationproblem. Third, we derive differentiable projective functions from 3D shape to2.5D sketches; the framework is therefore end-to-end trainable on real images,requiring no human annotations. Our model achieves state-of-the-art performanceon 3D shape reconstruction.