From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning

🎮 NavBench-GS: 3D-GS Benchmark for Visual Navigation

Hardware Recommendations

Installation

1. Install IsaacSim 5

# Clone the repository (includes third_party submodules)
git clone -b main --recurse-submodules https://github.com/VAIL-UCLA/S2E.git ~/S2E
cd ~/S2E
# If you already cloned without --recurse-submodules:
# git submodule update --init --recursive
# Install Isaacsim 5.x from github repo: https://github.com/isaac-sim/IsaacSim
suppose that you install it in:
# ${HOME}/IsaacSim/
and the builded software in:
# _build/linux-x86_64/release
Then try:
cd ~/S2E
ln -s ${HOME}/IsaacSim/_build/linux-x86_64/release ./_isaac_sim
# Isaac Sim root directory
export ISAACSIM_PATH="${HOME}/IsaacSim/_build/linux-x86_64/release"
# Isaac Sim python executable
export ISAACSIM_PYTHON_EXE="${ISAACSIM_PATH}/python.sh"
# note: you can pass the argument "--help" to see all arguments possible.
${ISAACSIM_PATH}/isaac-sim.sh

2. Install IsaacLab with extended features to support 3D-GS camera

# Bundled at third_party/isaaclab-extension
cd third_party/isaaclab-extension
# install the isaaclab following the doc

3. Install gsplat to support 3D-GS rendering

# Bundled at third_party/gsplat
pip install ninja numpy jaxtyping rich
cd third_party/gsplat
pip install -e . --no-build-isolation -v
# For the Vid2Sim rasterizer, use third_party/vid2sim-rasterizer-gsplat instead

4. Download 3D-GS scenarios from UCLA-VAIL/NavBenchGS

pip install -e NavbenchGS
python NavbenchGS/scripts/download_scenarios.py --output-dir data/NavBenchGS

data/NavBenchGS/
├── vid2sim_raw/ # Raw per-scene captures
│ ├── 0000/, 0001/, 0003/, ..., 0044/ # 4-digit scene IDs (non-contiguous)
│ ├── old-ped/ # Legacy pedestrian captures
│ ├── invalid_list.txt # Scene IDs to skip
│ └── upload.sh
├── vid2sim_torch/ # PyTorch-formatted per-scene data
│ └── 0000/, 0001/, 0003/, ... # same scene IDs as vid2sim_raw
├── vid2sim_starting_ending_position.json # Start/goal positions per scene
├── gs.gif # Preview animation
└── README.md

5. Debug the robots (empty scene)

# COCO — Ackermann car on a circle
${ISAACSIM_PYTHON_EXE} NavbenchGS/scripts/run_waypoint_to_action.py --enable_cameras \
 --robot coco --trajectory circle --num_steps 3000
# Go2 — JIT policy; defaults to robots/go2/unitree_go2_actor.pt
${ISAACSIM_PYTHON_EXE} NavbenchGS/scripts/run_waypoint_to_action.py --enable_cameras \
 --robot unitree_go2 --trajectory circle --forward_speed 0.6

6. Simulation with Isaac-Sim and Isaac-Lab (Please stay tuned!)

🧠 Model Zoo: Run the Pretrained Model

pip install -U "huggingface_hub[cli]"
hf download UCLA-VAIL/Navigation-Model-Zoo-Public --include "S2E/*" --local-dir ./nav_model_zoo

import numpy as np
from S2E.inference import S2ENavigator # run from ./nav_model_zoo
nav = S2ENavigator(device="cuda") # device="cpu" if no GPU
# obs: the robot's last 11 RGB frames, (1, 11, 3, H, W) float32 in [0, 1]
obs = np.random.rand(1, 11, 3, 256, 256).astype(np.float32)
# point-goal in the standard frame (x = forward, y = left, meters)
traj, scores = nav.inference_trajectory(obs, goal_xy=np.array([5.0, 0.0])) # (1, 1, 10, 2) m
vw, best = nav.inference_vw(obs, goal_xy=np.array([5.0, 0.0])) # (1, 2) = [v, ω]
nav.reset() # clear PD smoothing between episodes

📚 Pretraining with IL on Web Videos

🚀 Finetuning with RL in Simulator

📌 TODOs

💘 Acknowledgement

📎 Citation

@inproceedings{he2025seeing,
 title={From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning},
 author={Honglin He and Yukai Ma and Brad Squicciarini and Wayne Wu and Bolei Zhou},
 booktitle={International Conference on Learning Representations},
 year={2026}
}