One Action Interface, Seven Dexterous Hands
Can a policy trained on several robot hands control a hand it has never seen?
Question
Robot hands differ in size, number of joints, and finger layout, so a policy trained on one hand usually does not work on another. I'm testing whether training on more hand types helps a policy transfer zero-shot to a hand it never saw during training.
What I built
- A shared action space for all seven hands: the wrist pose plus the positions of the five fingertips in the palm frame. Per-hand inverse kinematics converts it into joint commands. If a hand has fewer fingers, the extra slots stay empty. The 3D viewer at the top of this page uses the same mapping.
- A teleoperation setup from Apple Vision Pro to Isaac Sim, built on NVIDIA Isaac Lab's CloudXR streaming. I used it to collect the demonstrations for all seven hands.
- A leave-one-hand-out benchmark with Shadow, Inspire RH56, Allegro, LEAP, Sharpa Wave, Wuji, and Wuji 2, plus the tooling that runs and tracks over a hundred training and evaluation jobs on our Slurm cluster.
Result
With the same amount of demonstration data, a policy trained on six hands reached 1.9 times the zero-shot success of one trained on a single hand (61% vs 32% on average) and did better on all seven target hands. It beat the best single source hand on four of seven targets, but not on average, and the best source is hard to know before testing. A few five-hand runs scored about as well as six hands, which suggests most of the gain comes before the last hand is added, but there are too few of them to draw a curve. I closed the study after 90 single- and six-hand evaluations.
Credits
- Advisor: Prof. Danfei Xu.
- Policy: HAT (Human Action Transformer), using its official code.
- Hand models: open-source URDFs from Shadow Robot (BSD-3-Clause), Wonik Robotics/SimLab (BSD-2-Clause), LEAP Hand/CMU (MIT), Sharpa (Apache-2.0), Wuji Technology (MIT) and Inspire Robots (RH56 model via Renesas RDK / dex_urdf). Sources and licenses: /hands/CREDITS.json.
- Simulation: Isaac Sim with DexVerse environments. IK: dex-retargeting.