Tianyue Wu^2,3,1,*,‡, Boyuan An^2,1,*, Shuqi Zhao¹, Heyu Guo², Wanli Xing²,
Yi Ma^3,1, Kaifeng Zhang², Ruihai Wu^1,†, Masayoshi Tomizuka^1,†

¹UC Berkeley    ²Sharpa    ³The University of Hong Kong
^*Equal contribution    ^†Co-advisors    ^‡Corresponding author

GALATEA does two things:

1. Hand–object reconstruction from generated videos: recovers a metric MANO hand trajectory and a 6-DoF object trajectory from each clip. Code: recon/.
2. A general hand–object tracking policy trained with reinforcement learning (RL): one policy for many objects and trajectories, on a Franka arm with a Sharpa Wave hand. Code: rl/.

Generated hand–object interaction videos provide a controllable way to propose manipulation motions. GALATEA combines generated videos with simulation-based HOI grounding: generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, while at deployment time video models produce motion plans that are executed by the learned tracker. With minimal manual intervention, the reconstruction pipeline yields approximately 2,000 usable references from 2,500 generated clips, of which more than 1,500 trajectories are grounded in simulation. In closed-loop real-world experiments, the distilled controller performs functional grasps, non-prehensile manipulation, and post-grasp object-pose tracking.

We will publicly release the remaining item below before November 2026.

• Release the HOI reconstruction code and example generated videos.
• Release the reinforcement-learning tracking code and pretrained checkpoints.
• Release the example datasets for reinforcement-learning tracking.

Each component has its own environments and documentation. To run the bundled reconstruction example:

git clone https://github.com/boyuan-an/GALATEA.git
cd GALATEA/recon
bash scripts/setup/install.sh

Then obtain the model files described in recon/docs/model-assets.md and run:

bash scripts/setup/check_install.sh --gpu
bash scripts/run_example.sh --gpu 0

See the reconstruction README for inputs, outputs, and options, and the tracking README for data preprocessing and for training and evaluating the tracking policy.

If you use this code in your research, please cite GALATEA:

@article{wu2026galatea,
  title   = {Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers},
  author  = {Wu, Tianyue and An, Boyuan and Zhao, Shuqi and Guo, Heyu and Xing, Wanli and Ma, Yi and Zhang, Kaifeng and Wu, Ruihai and Tomizuka, Masayoshi},
  journal = {arXiv preprint arXiv:2609.10050},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.10050}
}

Parts of the reconstruction code build on the code framework of GRAIL. The reconstruction pipeline integrates MoGe-2, GroundingDINO, SAM 2, FoundationPose, WiLoR, and HOI-DETR. We thank the authors of these projects for making their work available. Please also cite the works relevant to your use; component links are collected in recon/THIRD_PARTY.md. The tracking code builds on ManipTrans, SimToolReal, SAPG, rl_games, and Isaac Gym Envs; see rl/THIRD_PARTY.md.

This repository is released under the NVIDIA License, which limits use to non-commercial research and evaluation, except for rl/. The tracking component in rl/ is modified from GPL-3.0 code and is released under the GNU General Public License v3.0 (rl/LICENSE). Third-party code, checkpoints, and MANO data retain their own terms; see recon/THIRD_PARTY.md and rl/THIRD_PARTY.md.