Kinematic structures are very common in the real world. They range fromsimple articulated objects to complex mechanical systems. However, despitetheir relevance, most model-based 3D tracking methods only consider rigidobjects. To overcome this limitation, we propose a flexible framework thatallows the extension of existing 6DoF algorithms to kinematic structures. Ourapproach focuses on methods that employ Newton-like optimization techniques,which are widely used in object tracking. The framework considers bothtree-like and closed kinematic structures and allows a flexible configurationof joints and constraints. To project equations from individual rigid bodies toa multi-body system, Jacobians are used. For closed kinematic chains, a novelformulation that features Lagrange multipliers is developed. In a detailedmathematical proof, we show that our constraint formulation leads to an exactkinematic solution and converges in a single iteration. Based on the proposedframework, we extend ICG, which is a state-of-the-art rigid object trackingalgorithm, to multi-body tracking. For the evaluation, we create ahighly-realistic synthetic dataset that features a large number of sequencesand various robots. Based on this dataset, we conduct a wide variety ofexperiments that demonstrate the excellent performance of the developedframework and our multi-body tracker.