Researchers at the Bosch Center for AI and Carnegie Mellon University have developed a new AI system to improve the dexterity of humanoid robots.
The model, called Humanoid Transformer with Touch Dreaming (HTD), helps robots predict the outcomes of touch and force, enabling them to plan and execute tasks with greater spatial awareness.
The researchers said the system addresses a longstanding limitation in humanoid robotics: coordinating whole-body movement while simultaneously performing advanced object manipulation tasks.
“Real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes,” the researchers wrote in the paper’s abstract.
To try to bridge this gap, the team used reinforcement learning and VR-based data collection to teach robots dexterity and manipulation.
The system also combines tactile sensing, multi-view vision and proprioception (a neurological understanding of the body in space) to train robots in manipulation tasks.
Using this, HTD was trained not only to predict future actions, but also to anticipate how touch and force would evolve during manipulation — a process the researchers dubbed “touch dreaming” and from which the system gets its name.
In tests, HTD improved average task success rates by 90.9% across five real-world manipulation tasks: Insert-T (a specific robot assembly task), book organization, towel folding, cat litter scooping and tea serving.
Looking ahead, the team said they plan to scale HTD’s learning framework, integrate human demonstrations and expand applications to a broader range of tasks such as household chores, assisting customers in retail settings, and industrial and manufacturing environments.