Humanoid Manipulation from Human Demonstrations Only
Policies trained only on egocentric human demonstrations, with no robot data, run zero-shot on a real RB-Y1 humanoid. Along the way I studied what makes retargeted demonstrations learnable.
On the real robot
I train RB-Y1 policies only on egocentric human demonstrations recorded with Aria glasses, with no robot data at all. The first task was to walk to a table, pick up a toy and put it in a basket, learned from 145 human demonstrations. Policies on RGB images learned to navigate but not to manipulate. A single point cloud helped one stage or the other: in the robot's camera frame it navigated well but never finished the task, and cropped around the hand it manipulated better but lost its way. What worked was a dual-stream point cloud: a global cloud in the camera frame for navigation, plus a local cloud cropped around the right hand for manipulation, with a DP3 PointNet encoder and an HPT backbone.
The same recipe then transferred to a harder bimanual task, lifting a soft basket by both handles while moving the base, from only 50 human demonstrations. Colors near both hands are neutralized so human and robot hands look alike to the policy. The video at the top is that policy running zero-shot on the real robot (2x speed). It works for some basket placements but not all yet; the main failures are grasping only one handle, navigation errors that lead to unseen states, and jerky motion at action-chunk boundaries.
Jerky motion at chunk boundaries made the robot drop things, so I changed how actions are executed. The policy predicts 32 steps and executes 16. Running it synchronously at 10 Hz is smooth but slow. With inference-time real-time chunking (RTC) at 20 Hz, each new chunk is made to continue the motion that is still being executed, so the robot keeps moving smoothly while the next chunk is computed. In the two runs below, the same lift and place took about 32 s instead of about 50 s.
Which observations carry over from humans to the robot?
To see which inputs survive the jump from human to robot, I compared eight observation variants for human-only policies in a digital twin of the basket task, 30 rollouts each. With RGB images, no variant finished the task. Re-rendering the human view from the robot's camera height (EgoHumanoid-style reprojection with inpainting) never even reached the basket, and erasing the human arm or rendering the robot arm in its place did not help. RGB also carries a bigger sim-to-real gap: simulated images are much sharper than real ones (Laplacian variance 1,245 in simulation against 748 for robot and 643 for human camera images). Point clouds reached the basket every time, and replacing the human hand's points with robot-hand surface points doubled the grasps (20 vs 9 of 30) and gave the most full successes.

Earlier study: what makes retargeted demonstrations learnable?
Retargeting maps human motion onto a robot. In a follow-up to WARP, two retargeting methods applied to the same source demonstrations produced policies with very different success (14% vs 37% on a coffee task in simulation, each evaluated in its own scene layout). I studied which properties of the action data, such as consistency, smoothness, diversity, and how well actions can be predicted from observations, make demonstrations easier to imitate.
What I did
- Recorded 45 of the 98 egocentric human demonstrations (Aria glasses) for a two-handed basket carry. The WARP pipeline retargets them to the Rainbow RBY1 humanoid, and I ran the policy rollouts on the real robot.
- Compared how the two demonstrators move in the retargeted data. Their speed and timing are clearly different.
- Set up a controlled simulation study on the RBY1 (DexMimicGen coffee task) with 7 training-data conditions, 14 policies, and 1,400 closed-loop rollouts. It tests whether data-quality metrics from the literature (action variance, state-action mutual information, spectral arc length, signature-kernel diversity) explain the gap.
Findings
In the WARP scene, success goes up as the share of WARP-retargeted data increases. The two methods also place the robot base about 17 cm apart, though, so this result does not separate the effect of the scene layout from the effect of the data. None of the dataset-level metrics follows success consistently across tasks. Looking at individual frames shows a clearer pattern: the policy makes larger errors where its nearest training demonstrations disagree about the next action.
What came next
The dataset-level metrics did not predict closed-loop success well enough to guide data collection, so I closed this study. I moved to training policies directly on the human demonstrations, with point clouds as input; the real-robot results are at the top of this page.
Current work: why only some placements work
To find out why the basket policy only works for some placements, I evaluate human-only policies in a simulated copy of the task. They do reach and pick up the basket, but they don't steer toward it. In the human demonstrations, wherever the basket starts sideways, it ends up in the same place in front of the body by first contact (slope 0.00 across 53 demos). The baseline policy carries the starting offset straight into contact (slope 1.13 across 29 simulated trials), and the basket ends up 31 cm to its right on average.
I'm testing a change to the model structure: plan the base motion relative to the basket's position. In simulation it brings the robot to the basket in 30 of 30 trials, against 13 for the baseline. It doesn't lift the basket more often, though, so full-task success hasn't improved yet. On a second set of 45 recorded layouts it approached the basket in 41 cases (baseline 31), while the baseline finished more full tasks (6 vs 0).
Credits
- Advisor: Prof. Danfei Xu.
- Part of the WARP project in RL2, led by Chuye Zhang (WARP: arXiv 2606.29940).
- DexMimicGen retargeted datasets, the baseline checkpoints used in the learnability figure, and the training and rollout code: Zhenyang Chen. I trained the 14 policies for the controlled study with that setup and did the analysis.
- Basket simulation for the RB-Y1, its renderer, and the baseline human-only policy: Chuye Zhang. Evaluation, analysis, and the basket-anchored variant: mine.