MANIPTRANS Framework and DexManipNet Dataset: A Breakthrough in Human-to-Robot Skill Transfer Efficiently transferring human dexterity skills to robots has long been a challenging task in the field of robotics. Recently, a joint team from the Institute for Advanced Robotics and Intelligent Systems (IARIS) has open-sourced the MANIPTRANS framework along with its accompanying dataset, aimed at addressing this issue. The framework is designed to overcome key challenges such as body differences and cumulative motion errors through a two-phase learning approach. Phase One: Motion Simulator The first phase involves the development of a "motion simulator" that focuses on replicating basic human hand movements while temporarily ignoring interactions with objects. This simulator uses large-scale motion capture (MoCap) data and the Proximal Policy Optimization (PPO) reinforcement learning algorithm for pre-training. By mastering human hand movement patterns, the motion simulator can effectively map human motor intentions to various types of robotic hands, significantly mitigating issues related to physical differences and providing a natural, fluid foundation for hand movements. Phase Two: Residual Learning Module Despite the progress made in the first phase, the motion simulator alone cannot ensure adherence to physical rules or effective object manipulation. To address this, MANIPTRANS introduces a residual learning module in the second phase. This module learns the necessary fine-tuning adjustments to correct the outputs of the initial model. It receives detailed state information, including real-time object status (position, velocity, shape) and contact forces. Based on these interactions, the residual module computes the required adjustments to the simulated actions, ensuring they are both smooth and precise. Dr. Jiacheng Li, a leading researcher on the project, highlighted the importance of the residual strategy: "By continuously improving the residual strategy, we achieved successful bimanual operations where the left hand removed a pen cap while the right hand held the pen body and naturally inserted it into the cap. This task not only requires precision but also high coordination between both hands. Our success with MANIPTRANS confirms its effectiveness in skill transfer and marks a significant milestone in our research." Design and Efficiency The two-phase design cleverly decomposes the complex learning task into simpler subtasks, reducing the dimensionality of the action space and enhancing training efficiency. As a result, MANIPTRANS can proficiently transfer intricate bimanual mechanical skills from humans to robotic hands, particularly those involving detailed cooperation between both hands. To support this framework, the research team also created the DexManipNet dataset, which consolidates multiple representative hand-object interaction datasets, such as FAVOR and OakInk-V2, onto dexterous robotic hands. The dataset currently includes 33,000 robotic hand operation segments, covering 12,000 objects, with a total of 1.34 million frames. Notably, it contains 600 sequences that involve complex bimanual tasks, encompassing 61 different types, such as pen cap insertion and bottle opening. Dr. Li believes that DexManipNet, being one of the largest datasets supporting complex tasks, will help train diverse robotic operation models capable of performing more generalized, flexible, and cooperative dexterous hand operations in various environments. Industry experts have praised MANIPTRANS and DexManipNet, noting that these developments not only advance the research on skill transfer but also lay a solid foundation for achieving higher precision and stability in robotic operations. Novel Soft Robotic Hand for Mouse Operation Scientists led by Professor Guang Li have recently developed a new type of soft robotic hand capable of operating a computer mouse with high dexterity. This research explores the integration of physical intelligence and mechanical intelligence, aiming to create a cost-effective and versatile robotic hand suitable for applications requiring precise and gentle manipulation, such as medical rehabilitation and assistive gaming. Standardized Design Process The team introduced a standardized rapid design process for developing customized dexterous robotic hands tailored to specific tasks and scenarios. This approach reduces development costs and shortens the research cycle. Using this process, they created a soft robotic hand that can perform a range of mouse operations, including clicking, double-clicking, scrolling, and dragging, by mimicking human finger movements. Performance and Energy Efficiency Experiments showed that the soft robotic hand has very high operational precision and stability, successfully completing multiple complex tasks. The advanced physical driving and transmission system ensures low power consumption, allowing the hand to operate for extended periods on a single charge. Future Applications Professor Li envisions broader applications for this soft robotic hand, including smart home environments and industrial production lines. It can assist individuals with mobility impairments in their daily activities and also enhance productivity and technological capabilities in specialized fields. “We are exploring ways to further optimize its performance to simulate a wider range of hand movements and expand its utility,” he said. The development of this soft robotic hand was published in the journal Advanced Intelligent Systems. The paper emphasizes that this innovative robotic hand opens new possibilities for people with limited mobility and introduces new directions in robot design. Industry observers see this as a groundbreaking technology, underscoring the vast potential of soft robotics, especially in scenarios requiring high flexibility and precise control. Evaluation and Company Profiles Both MANIPTRANS and the novel soft robotic hand represent significant advancements in robotics research. MANIPTRANS's two-phase design and DexManipNet dataset tackle the problem of skill transfer by simplifying the learning process and enhancing the adaptability of robotic hands to various tasks. This approach not only improves the efficiency of training but also the performance and versatility of the robotic hands. The soft robotic hand, developed by Professor Guang Li's team, demonstrates a practical application of advanced robotics in assistive technology. By using a simplified and efficient driving system, it reduces the complexity and cost typically associated with dexterous hands. Its potential applications in daily life and specialized industries highlight the growing importance of soft robotics in improving accessibility and productivity. IARIS is a leading institution in advanced robotics and intelligent systems, dedicated to enhancing the application of robots across various industries. Their ongoing research and innovations, such as MANIPTRANS and the soft robotic hand, are making significant strides in the field of robotics, setting new standards and benchmarks for future developments.