Panasonic HD Co-Develops a Robot Manipulation AI That Performs Dexterous Bimanual Tasks by Reacting Instantly to Fingertip Touch

T-Rex integrates low-frequency "action generation based on vision and language" with "high-speed action correction" driven by tactile feedback from the fingertips into a single model. While the motion generation module, which processes vision and language, generates the overall motion at approximately 5 Hz ⁽*³⁾, the action correction module, which processes tactile input, captures contact conditions in real time at approximately 20 Hz and finely corrects the action. This allows the robot to respond quickly to slippage or excessive pressing during contact without having to rerun high-latency image recognition each time.
For training data, the research team combined more than 200 types of everyday objects with 22 types of basic actions to collect over 7,700 episodes-a total of 100 hours-of bimanual robot manipulation data. Each episode synchronously records video from head and wrist cameras, the robot's states and actions, force and deformation information obtained through tactile sensors on 10 fingertips, and natural-language instructions. T-Rex first learns broad visual and motion knowledge from 22,889 hours of human egocentric video, then learns robot-specific contact-rich behaviors from this tactile-annotated data (mid-training), and is finally fine-tuned with approximately 100 demonstrations per task. In evaluations on 12 contact-rich real-world tasks-including turning pages, transferring eggs, applying toothpaste, unlocking with a key, pulling out a card, and screwing in a light bulb-T-Rex achieved an average success rate of 65%, far exceeding the 35% achieved by the previously best-performing baseline. Without tactile information, the average success rate dropped to just 42%, confirming the effectiveness of tactile sensing and the high-speed reactive mechanism.