Baoyu Li*, Xinchen Yin*, Mengying Lin, Yixin Zhang, Danfei Xu
Georgia Institute of Technology β’ CoRL 2026
Project Page | arXiv | Data | Checkpoints
@article{li2026egowam, title = {EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data}, author = {Li, Baoyu and Yin, Xinchen and Lin, Mengying and Zhang, Yixin and Xu, Danfei}, journal = {arXiv preprint arXiv:2607.08436}, year = {2026} }
- π οΈ Installation
- π₯οΈ Data
- π§ Training
- π€ Simulation (RoboTwin)
- π¦Ύ Real-Robot Deployment
- β Troubleshooting
EgoWAM/
βββ egowam/
β βββ trainHydra.py # training entrypoint (Hydra + Lightning)
β βββ algo/ # HPT world-action model
β βββ models/ # stems, trunk, flow-matching action head, WM heads
β βββ rldb/ # zarr data layer, embodiment keymaps + transforms
β βββ hydra_configs/ # model / data / trainer / launcher configs
β βββ pl_utils/ # Lightning module + datamodule wrappers
β βββ robot/ # real-robot stack (ARX5 + Aria + RealSense)
β βββ resources/ # robot kinematics model (model_x5.xml)
β βββ compat.py # alias for pre-rename checkpoints
β βββ scripts/
β βββ data_download/ # S3 / SQL episode sync
β βββ eva_process/ # robot demos -> zarr
β βββ aria_process/ # human video -> zarr
β βββ 3dflow_process/ # Track4World 3D-flow targets
β βββ rae_process/ # RAE stage-1 decoder training
β βββ robotwin_process/ # simulation bridge, deploy + eval
β βββ evaluation/ # checkpoint validation
βββ external/ # vendored: RoboTwin, RAE, Track4World, cosmos-policy
βββ sbatch/ # cluster launchers (6 training + robotwin)
βββ docs/ # documentation
βββ assets/ # teaser + figures
βββ pyproject.toml
All variants share the trunk and the flow-matching action head, and differ only in the world target.
| Paper name | World target | Robot-only config | Co-train config | Launcher |
|---|---|---|---|---|
| BC | none (action only) | model/hpt_bc_eva |
model/hpt_bc_cotrain |
hpt_eva.sh, hpt_cotrain.sh |
| Pixel | Wan VAE latent of the future frame | model/hpt_wm_eva |
model/hpt_wm_cotrain |
hpt_wm_vae.sh |
| Pixel-PT | same target, head pretrained on Wan2.1 video | model/hpt_wm_eva_pixelpt |
model/hpt_wm_cotrain_pixelpt |
hpt_wm_pixelpt.sh |
| DINO | DINOv2 patch features, RAE wide-DDT head | model/hpt_wm_eva_rae |
model/hpt_wm_cotrain_rae |
hpt_wm_rae.sh |
| 3D Flow | camera-stabilized dense 3D motion field | model/hpt_wm_eva_3dflow |
model/hpt_wm_cotrain_3dflow |
hpt_wm_3dflow.sh |
git clone https://github.com/GaTech-RL2/EgoWAM.git && cd EgoWAM uv venv emimic --python 3.11 && source emimic/bin/activate uv pip install -r requirements.txt uv pip install -e . export EGOWAM_DATA_ROOT=/path/to/zarr_datasets # BC co-training on a 1% slice -- fastest check, no third-party assets at all python egowam/trainHydra.py --config-name=train_zarr.yaml \ data=debug model=hpt_bc_cotrain trainer=debug \ name=debug description=smoke # DINO co-training on a 1% slice -- ~10 min, needs the stage-1 decoder (docs/install.md) python egowam/trainHydra.py --config-name=train_zarr_wm.yaml \ data=debug_wm model=hpt_wm_cotrain_rae trainer=debug \ model.robomimic_model.wm_prediction_mode=joint \ model.robomimic_model.training_stage=3 \ 'model.robomimic_model.auxiliary_ac_keys.eva_bimanual=[future_image_latent]' \ 'model.robomimic_model.auxiliary_ac_keys.aria_bimanual=[future_image_latent]' \ model.robomimic_model.auxiliary_loss_weights.future_ee_pose=0.0 \ model.robomimic_model.auxiliary_loss_weights.future_flow=0.0 \ name=debug description=smoke # 3D-flow co-training on a 1% slice -- needs no gated assets # First run also computes point-flow statistics (~40 min, cached afterwards); see docs/troubleshooting.md python egowam/trainHydra.py --config-name=train_zarr_wm.yaml \ data=debug_wm3dflow model=hpt_wm_cotrain_3dflow flow3d=point3d trainer=debug \ name=debug description=smoke
Episodes are synced on demand into EGOWAM_DATA_ROOT the first time a config referencing them is
trained β configure credentials once first, per docs/data.md.
Full setup in docs/install.md; every variant and regime in docs/training.md.
On the full splits, expect a slow start. The smoke run above finishes in minutes, but normalization statistics are computed over the whole dataset before epoch 0 and are I/O-bound β tens of minutes on the flagship splits. No epoch logged for the first hour is normal, not a hang.
This repository is built on EgoVerse, an ecosystem for egocentric human data processing and BC algorithm training.