One-sentence Summary

Researchers at Ropedia present DONIC, the first generalist bimanual tracker jointly trained across operation categories and human-motion sources, along with BiHOI, a multi-view dataset of 1,575 sequences spanning 229 objects; DONIC couples proximal policy optimization with behavioral cloning, object-centered per-hand rewards, and clip-centered advantages, achieving 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, surpassing state-of-the-art bimanual tracking baselines by 77.36 and 45.73 percentage points.

Key Contributions

• The paper introduces BiHOI, a multi-view bimanual hand-object interaction dataset with 1,575 sequences spanning 229 objects, where reconstructed interactions are retargeted into robot motion references.
• The paper presents DONIC, a generalist bimanual tracker jointly trained across operation categories and human-motion sources; it combines proximal policy optimization with behavioral cloning in a feedback loop, an object-centered per-hand reward, and Clip-Centered Advantages to coordinate per-hand contact roles.
• Results show that a single DONIC checkpoint reaches 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, outperforming bimanual tracking baselines by 77.36 and 45.73 percentage points respectively.

Introduction

Dexterous bimanual manipulation is important for household assistance, industrial assembly, and assistive care because many everyday tasks require one hand to stabilize an object while the other opens, rotates, or repositions it. Prior reference trackers have mostly focused on single-hand control, and public bimanual datasets with fully tracked, sustained two-hand contact remain scarce. Shared bimanual training is also difficult because the policy must handle changing hand roles, asynchronous contact timing, and unequal clip difficulty without masking one hand's progress. The authors address these gaps with DONIC, a generalist bimanual reference tracker trained across operation categories and human motion datasets, and with BiHOI, a multi-view hand-object interaction dataset. Their object-centered per-hand reward and Clip-Centered Advantages enable one checkpoint to generalize to unseen trajectories and objects without fine-tuning, substantially outperforming adapted bimanual baselines.

Dataset

The authors introduce BiHOI, a multi-view bimanual hand-object interaction dataset designed to expand limited bimanual training data.

• Capture setup: Recorded with 32 calibrated RGB cameras around a tabletop workspace at 1920×1232 resolution and 30 fps. Calibration provides a common metric coordinate system.
• Dataset composition: Contains 1,575 interaction sequences, 229 objects, and 14.77 hours of unique interaction. Objects include household items, containers, utensils, tools, and electronics. Interactions are natural rather than fixed templates and cover symmetric bimanual manipulation, asymmetric bimanual coordination, and coordinated multi-object manipulation.
• Bimanual emphasis: Among frames with at least one hand in contact, 65.91% have both hands in contact, which the authors report is higher than the analyzed public subset H2O at 38.57%.
• Object reconstruction: SAM3 provides object masks; multi-view Depth Anything 3 provides metric-aligned depth; MV-SAM3D reconstructs a canonical object mesh; FoundationPose recovers 6-DoF object motion using multi-view pose consensus.
• Hand reconstruction: Per-participant MANO shapes are calibrated from triangulated keypoints. WiLoR provides per-view hand observations, which are triangulated, temporally smoothed, and fitted to the MANO model with fixed shape. Joint refinement against multi-view depth produces bimanual MANO parameters, a metric object mesh, and 6-DoF object trajectories.
• Retargeting: Human motion is mapped to Wuji robot hands through frame-wise kinematic fitting. Wrist frames are aligned, hand keypoints are fitted to robot links under joint limits, and each frame is initialized and regularized against the previous solution. The resulting wrist and finger references are paired with the object trajectory.
• Usage: The dataset provides paired robot hand and object references for bimanual training. The provided section does not specify a train/validation split or mixture ratios.

Method

The authors introduce DONIC, a framework designed to learn a generalist bimanual tracker across diverse operation categories and human data sources. To build a comprehensive training pool, they leverage the BiHOI dataset alongside public datasets. Multi-view recordings are first reconstructed into metric hand-object trajectories using foundation models for depth and pose estimation, and then retargeted to robot kinematics through framewise fitting.

As shown in the figure below:

The core of the control system is a single stochastic policy that governs both robot hands. The policy predicts bounded corrections to the upcoming reference targets based on an observation space that concatenates per-hand proprioception, object pose and velocity, reference targets, state errors, and geometric shape encodings. These predicted corrections are added to the next reference and clamped to joint limits to produce proportional-derivative control targets, ensuring the corrections are anchored to the reference at every step without accumulating errors. The control target is formulated as:

where $\boldsymbol{s}$s sets the correction range, $\odot$⊙ denotes element-wise multiplication, and the clamp function enforces coordinate limits.

To ensure physical interaction and successful object manipulation, the authors design an object-centered per-hand reward. This reward structure prioritizes object motion tracking while supervising hand contacts only when required by the reference. The object tracking term penalizes errors in position, orientation, and linear velocity, effectively preventing stationary solutions for moving references. For contact supervision, the system evaluates the distance and simulated contact force of required fingertips. These contact and hold rewards are aggregated over the engaged hands using fixed weights. This aggregation ensures that asynchronous contact roles are properly supported and avoids rewarding attachment during reference release. The final reward combines these signals with a standard hand-tracking term using non-negative coefficients that sum to one.

The policy is trained using an iterative loop of Proximal Policy Optimization and behavioral cloning. During PPO updates, shared batches mix diverse contact patterns and coordination demands, which can lead to unequal critic accuracy across different reference clips. To address this, the authors introduce Clip-Centered Advantages. Standard Generalized Advantage Estimation can leave clip-dependent offsets that unfairly penalize difficult clips. By subtracting the mean advantage of each reference clip, the method preserves within-clip advantage differences while removing the common empirical offset. The centered advantage is calculated as:

where $\mathcal{G}_{m(j)}$Gm(j)​ groups transitions by their reference identity. This adjustment acts solely on the policy learning signal without modifying the reward or the critic regression targets.

The final actor objective combines the PPO loss using these centered advantages with a behavioral cloning regression loss over a library of verified successful demonstrations. The joint objective is defined as:

where $\lambda$λ weights the behavioral cloning component and the expectation averages over sampled demonstration pairs. This combined approach allows the policy to learn from both simulated interaction and verified expert actions, enabling it to track unseen trajectories and objects without fine-tuning.

Experiment

The experiments evaluate bimanual manipulation tracking on held-out splits for unseen trajectories and unseen objects, using public bimanual datasets and comparing DONIC against DexTrack-bi and Dexplore-bi under the same protocol. Results show that DONIC generalizes without fine-tuning and consistently improves bimanual success, completion, and hand-object tracking, especially by coordinating both hands rather than relying on single-hand accuracy. Ablations confirm that contact-consistency alignment and the contact-ownership gate both contribute to joint bimanual execution, and adding BiHOI data further improves performance across all tested conditions.

BiHOI introduces a multi-view RGB dataset of 1,575 interaction sequences spanning 229 objects. Its defining characteristic is sustained bimanual contact, with 65.91% of contact frames involving both hands, compared with 38.57% for the next highest analyzed public dataset. This emphasis on simultaneous two-hand interaction distinguishes BiHOI from datasets with larger frame counts or object counts. BiHOI records the highest bimanual contact rate among the analyzed datasets, exceeding every public comparison by at least 27 percentage points. Despite having fewer objects than HOI4D and fewer frames than GigaHands, BiHOI prioritizes two-hand interaction coverage across 229 objects.

A single DONIC checkpoint improves both held-out splits without fine-tuning, achieving much higher success and completion rates than the baselines. It also lowers most tracking errors, with the exception of object rotation on unseen trajectories. Ablations show that both learning signals contribute to joint bimanual success and completion, and adding all available bimanual sequences further improves every reported metric. DONIC raises success and completion rates substantially over DexTrack-bi and Dexplore-bi on both unseen trajectory and unseen object splits. Removing CCA or the contact-ownership gate reduces success and completion on both splits, while adding all BiHOI sequences improves all seven metrics across the two splits.

The experiments first evaluate BiHOI against existing datasets and then assess DONIC on held-out unseen trajectory and object splits. BiHOI stands out for its high bimanual contact rate, with 65.91% of contact frames involving both hands, far exceeding other datasets despite having fewer objects or frames. DONIC substantially improves success and completion rates over DexTrack-bi and Dexplore-bi without fine-tuning, and ablations show that both the CCA and contact-ownership gate contribute, while adding all BiHOI sequences improves every reported metric.