Adapting Robot Foundation Models to a 16-DoF Hand

At RLWRLD I worked on getting large pretrained policies (GR00T N1.6, π0.5, and a 3D geometry-aware policy) to work on DexJoCo, a public benchmark with 11 dexterous manipulation tasks.

GAM on a dexterous hand

GAM is a recent 3D geometry-aware policy designed for parallel-jaw grippers. I adapted it to a Franka arm with a 16-DoF Allegro hand, which meant a new data loader and policy server, delta actions, depth inputs, and a fixed training recipe. One issue I found was that the training step limit counts micro-batches, not optimizer updates, so with our gradient accumulation the early runs got about 156 updates instead of 10k. On the water-plant task, success went from 11% to about 55% (one training run, 6 evaluation seeds × 50 episodes, flat between 20k and 70k updates).

Diagram of GAM: a frozen geometry foundation model split at layer 12, a causal future predictor, and action, depth and feature losses.
GAM as published, in a diagram I drew while porting it. Blue: pretrained and frozen. Orange: trained with GAM. Purple: training losses.
The policy's predicted depth for the next step (middle) next to the depth that actually follows (right). The prediction is smoother and misses thin parts like the peg.
Where the action tokens look: attention from GAM's action tokens to image patches in the front, left-wrist and right-wrist cameras (columns) at four layers (rows), over the same Tower of Hanoi task.

Fine-tuning and benchmarking VLAs

I fine-tuned GR00T N1.6 on all 11 DexJoCo tasks and got 46.2% mean success, compared with 40.3% reported for GR00T N1.5. I also re-ran the released π0.5 checkpoints in the same evaluation setup. The mean matched the reported 52.5%, though individual tasks varied.

Per-task success on the 11 DexJoCo tasks for GR00T N1.6 (fine-tuned) and π0.5.
Hammer Nail from the same starting scene. Fine-tuned GR00T N1.6 drives the nail, while π0.5 hovers until time runs out. Across all 50 episodes π0.5 still does better on this task (44 vs 36).

Reproducible evaluation

Running the same seed twice changed the outcome in 22.5% of episodes. The causes were unseeded flow-matching noise in the policy and nondeterministic GPU rendering. After fixing both, reruns gave identical results, so we could tell real differences between models apart from noise. I also built the training and evaluation platform for these runs: a web console that launches, monitors and collects results for training and evaluation jobs on two Slurm clusters and a Kubernetes cluster.

The training and evaluation console: a running DexJoCo evaluation job with per-GPU memory on 8 H200 GPUs.
The console's job page: an 11-task DexJoCo evaluation running on 8 H200 GPUs, with progress and scheduler state.
Before the fix: the same checkpoint and seed, run twice. Rendering differences alone make one run fail and the other succeed.

Credits

  • Work done as a Research Engineer at RLWRLD.
  • DexJoCo benchmark: Wang et al., arXiv 2605.16257. GAM: Han et al., arXiv 2606.17046. GR00T N1.6: NVIDIA. π0.5: Physical Intelligence.