What Makes Retargeted Demonstrations Learnable?

The question

Retargeting maps human motion onto a robot. In a follow-up to WARP, two retargeters applied to the same source demonstrations produced policies with very different success (14% vs 37% on a coffee-making task in simulation, each in its own scene layout). I study which properties of the resulting action data — consistency, smoothness, diversity, and how predictable actions are from observations — make demonstrations easier to learn by imitation.

What I did

• Recorded egocentric human demonstrations with Aria glasses (45 of the 98 demos of a two-handed basket carry), which the WARP pipeline retargets to the Rainbow RBY1 humanoid, and ran the real-robot policy rollouts.
• Characterized how a demonstrator's style survives retargeting: the two demonstrators' motions differ clearly in speed and timing.
• Designed a controlled simulation study on the RBY1 (DexMimicGen coffee task): 7 training-data conditions, 14 policies, 1,400 closed-loop rollouts, testing whether data-quality metrics from the literature — action variance, state–action mutual information, spectral arc length, and signature-kernel diversity — explain the gap.

Findings so far

In the WARP scene, success rises with the share of WARP-retargeted data — though the two retargetings also place the robot base about 17 cm apart, so scene layout and data are not yet separated. None of the dataset-level metrics tracks success consistently across tasks. Looking frame by frame is more telling: a policy errs more exactly where its nearest training demonstrations disagree about what to do next.