Existing approaches of hand reconstruction predominantly adhere to amulti-stage framework, encompassing detection, left-right classification, andpose estimation. This paradigm induces redundant computation and cumulativeerrors. In this work, we propose HandOS, an end-to-end framework for 3D handreconstruction. Our central motivation lies in leveraging a frozen detector asthe foundation while incorporating auxiliary modules for 2D and 3D keypointestimation. In this manner, we integrate the pose estimation capacity into thedetection framework, while at the same time obviating the necessity of usingthe left-right category as a prerequisite. Specifically, we propose aninteractive 2D-3D decoder, where 2D joint semantics is derived from detectioncues while 3D representation is lifted from those of 2D joints. Furthermore,hierarchical attention is designed to enable the concurrent modeling of 2Djoints, 3D vertices, and camera translation. Consequently, we achieve anend-to-end integration of hand detection, 2D pose estimation, and 3D meshreconstruction within a one-stage framework, so that the above multi-stagedrawbacks are overcome. Meanwhile, the HandOS reaches state-of-the-artperformances on public benchmarks, e.g., 5.0 PA-MPJPE on FreiHand and 64.6%[email protected] on HInt-Ego4D. Project page: idea-research.github.io/HandOSweb.