Recent work achieved impressive progress towards joint reconstruction ofhands and manipulated objects from monocular color images. Existing methodsfocus on two alternative representations in terms of either parametric meshesor signed distance fields (SDFs). On one side, parametric models can benefitfrom prior knowledge at the cost of limited shape deformations and meshresolutions. Mesh models, hence, may fail to precisely reconstruct details suchas contact surfaces of hands and objects. SDF-based methods, on the other side,can represent arbitrary details but are lacking explicit priors. In this workwe aim to improve SDF models using priors provided by parametricrepresentations. In particular, we propose a joint learning framework thatdisentangles the pose and the shape. We obtain hand and object poses fromparametric models and use them to align SDFs in 3D space. We show that suchaligned SDFs better focus on reconstructing shape details and improvereconstruction accuracy both for hands and objects. We evaluate our method anddemonstrate significant improvements over the state of the art on thechallenging ObMan and DexYCB benchmarks.