MEMOBench

MEMOBench benchmark suite overview

MEMOBench is a process-level memory benchmark for robotic manipulation. It annotates 30 history-dependent tasks with 4,200 executable checkpoints labeling three memory operations: Storage, Update, and Compression. Evaluations show leading VLA policies store information well but fail to update and compress it, reaching only 31.9% average success.

Both are available at HuggingFace.

MEMOBench builds on LIBERO, vendored under src/third_party/LIBERO:

git clone https://github.com/SunSeaLucky/MEMOBench.git
cd MEMOBench
pip install -e src/third_party/LIBERO
pip install imageio tyro tqdm numpy

Evaluation talks to the policy through the openpi websocket client, so also install openpi_client (from the openpi repository).

From HuggingFace:

hf download SunSeaLucky/MEMOBench MEMOBench.zip --repo-type dataset
hf download SunSeaLucky/MEMOBench assets.zip --repo-type dataset
unzip MEMOBench.zip # -> MEMOBench/{object,procedural,spatial,temporal}/*.hdf5

Extract assets.zip and copy the extracted object/scene assets into the LIBERO assets directory src/third_party/LIBERO/libero/libero/assets/.

Start a policy server exposing the openpi websocket API (default 0.0.0.0:8000), e.g. with openpi's scripts/serve_policy.py. The client sends observation/image, observation/wrist_image, observation/state, and prompt, and expects an action chunk in return.

The executable task configs (task_config.bddl + task_config.meta.json with checkpoint annotations) live under src/bddl/:

python src/eval.py \
 --task-dir src/bddl/temporal \
 --num-trials-per-task 50 \
 --memory-level 1 \
 --video-out-path data/libero_temporal/videos \
 --failure-out-path data/libero_temporal/failures.json

Key arguments: --host / --port (policy server), --memory-level (0–2, selects the memory operation labels), --max-steps, --replan-steps, --seed. The script reports overall success rate, per-operation rates (storage / update / compression), a failure-mode taxonomy (storage failure, stale memory, compression collapse, reference drift, execution failure), and saves rollout videos plus an optional failure summary JSON.

python src/get_object_pool.py # writes .scrapy/objects.json

MEMOBench is licensed under CC BY-NC 4.0 (see LICENSE): free for non-commercial scientific research with attribution; commercial use is not permitted. Third-party code under src/third_party/ (e.g., LIBERO) remains under its own license (MIT).