One Action Interface, Seven Dexterous Hands

The question

Dexterous hands differ in size, joint count, and finger layout, so a policy trained on one hand usually cannot drive another. I am studying whether co-training on more hand types improves zero-shot transfer to a hand that was held out of training entirely.

What I built

• A shared action interface — wrist pose plus five palm-frame fingertip positions — and per-hand inverse kinematics that turns it into each hand's joint commands. Fingers a hand does not have are left empty rather than invented. The viewer above runs the same idea live in your browser.
• An Apple Vision Pro → Isaac Sim teleoperation workflow (on NVIDIA Isaac Lab's CloudXR streaming), which I used to collect demonstrations on all seven simulated hands.
• A leave-one-hand-out benchmark over seven hands — Shadow, Inspire RH56, Allegro, LEAP, Sharpa Wave, Wuji, and Wuji 2 — and the experiment platform that schedules and tracks well over a hundred training and evaluation jobs on a Slurm cluster.

Early result

With the same demonstration budget, a policy co-trained on six hands reaches about twice the zero-shot success of single-hand training on the held-out hand, and is higher on every target hand evaluated so far. It does not yet beat the single best source hand; the full curve from one to six training hands is in progress.