Vision Pro Teleoperation for Dexterous Hands

I built an Apple Vision Pro teleoperation setup for seven simulated robot hands and recorded 357 grasping demonstrations with it. On hardware, I teleoperate a Wuji Hand 2 with the Vision Pro to collect tactile data.

What it does

The operator wears an Apple Vision Pro and simply moves their hand. Hand and head poses stream to a GPU machine running Isaac Sim, get retargeted to whichever robot hand is loaded, and the rendered scene streams back into the headset, so the operator sees the robot hand in 3D and steers it like their own. Each successful grasp is saved as a training demonstration.

Pipeline: Vision Pro hand tracking, GPU session with Isaac Sim, retargeting, 60 Hz simulation of 7 hands, the view streamed back, then episode loop, dataset, review and conversion.
Teal boxes are the parts I built or extended; grey boxes are the existing tools underneath.

What I built

  • A one-click launcher that starts a full teleoperation session (Isaac Sim, the CloudXR runtime and the task) on a lab workstation or a Slurm GPU node, without Docker.
  • An in-headset control panel, added to NVIDIA's open-source visionOS client. It follows the operator's view, shows the task, and has Start, Restart episode and End collection buttons, plus clear error messages.
  • Six more robot hands next to the Shadow hand that DexVerse ships with: Inspire RH56, Allegro, LEAP, Sharpa Wave, Wuji and Wuji 2, each with a URDF bundle, DexPilot finger mapping and mimic-joint fixes.
  • The recording loop: an episode is saved once the cube is lifted 20 cm and held for 10 steps. Tracking dropouts under 0.5 s resume the episode; longer ones discard it. Recordings are then checked in a 3D replay and converted into the shared wrist + fingertip action labels used for training.
  • A small native visionOS app that records ARKit hand and head poses on the headset and exports them as JSON lines, for capturing human motion without the simulator.

Demonstrations

With this setup I recorded 357 Pick Cube demonstrations myself: 51 for each of the seven hands, each 1.4 to 2.7 seconds of simulated time at 60 Hz. They are the training data for the cross-embodiment project (One Action Interface, Seven Dexterous Hands).

The first recorded demonstration for each of the seven hands, replayed from the saved camera images at half speed.
Five saved Shadow-hand demonstrations, re-rendered for review.

On a real robot

The video at the top is the same idea on hardware: I teleoperate a Wuji Hand 2 on a Franka arm with the Vision Pro while the hand's tactile readings are recorded, here while a colleague touches the robot fingers.

Credits

  • Advisor: Prof. Danfei Xu.
  • Headset app: NVIDIA's open-source Isaac XR teleop sample client for Apple Vision Pro, which I patched. Streaming and simulation: NVIDIA CloudXR, Isaac Sim and Isaac Lab.
  • Pick Cube task, wrist retargeter and the original VR recorder: DexVerse by Yunchao Yao.
  • Robot hand models: open-source URDFs from the hand makers (sources on the cross-embodiment project page).