RoboDemo-full50
v0.1-evaluation · November 20, 2026
Release notes | Versioned files
Reconciling Demonstration Intent and Execution Progress for Long-Horizon Manipulation
Code | Project page | All checkpoints | Complete numerical results
This is the 240,000-update checkpoint evaluated on Imitator Game, conditioned on a human demonstration. The release includes the complete policy state dictionary, inference configuration and normalization. Model architecture and evaluation are available; training code is planned for the next release. Anonymous manuscript under review.
Reported Results
| Split | Successful episodes | Episodes | SR (%) |
|---|---|---|---|
| Seen (50 tasks) | 1692 | 2000 | 84.6 |
Each task is evaluated at four scene levels, with ten trials per level and a 500-control-step horizon.
Every task and level | Task-level CSV | Episode CSV | Episode JSONL
Choose Your First Run
- Explore without a GPU: clone the code and run
python examples/predict_actions.py --inspect-onlyto inspect the included video, state and action arrays (NumPy required). - Evaluate this checkpoint: follow the one-episode command below, then increase
--max-episodes. - Learn the interfaces: the two minimal examples cover human-video-to-action inference (Full50 weights) and raw-media reference retrieval (CPU supported).
Reproduction navigator and validation status
Quickstart
1. Install The Evaluation Environment
Use Linux, Python 3.10 and an NVIDIA GPU. The recorded environment uses PyTorch 2.6.0 / CUDA 12.4 and Transformers 4.57.6.
git clone --branch v0.1-evaluation --single-branch https://github.com/robodemo-2027/RoboDemo.git
cd RoboDemo
python3.10 -m venv .venv
source .venv/bin/activate
pip install torch==2.6.0 torchvision==0.21.0 \
--index-url https://download.pytorch.org/whl/cu124
pip install -r environments/requirements-evaluation.txt
pip install flash-attn==2.7.4.post1 --no-build-isolation
See the environment guide for the simulator revision, Vulkan and OptiX setup.
2. Prepare The Checkpoint And Smoke Assets
The asset preparation helper downloads this checkpoint, the required encoder files, simulator source/assets and the reference video for one towel-folding episode:
python scripts/prepare_assets.py --root /data/robodemo-assets \
--profile smoke --model full50 --extract-assets
source /data/robodemo-assets/robodemo-env.sh
The policy file is approximately 16.5 GB and contains the complete policy
state dictionary. The temporal encoder checkpoint is downloaded separately;
the needed temporal source is bundled with the code. Benchmark media and
simulation assets come from Imitator Game.
Use --dry-run first to inspect the download plan.
3. Run One Complete Episode
From the repository root, select a GPU allocated to your job:
export CUDA_VISIBLE_DEVICES=0
python environments/check_environment.py --assets \
--output outputs/full50-environment.json &&
bash scripts/evaluate/full50-single.sh \
--checkpoint "$ROBODEMO_HOST_ASSETS/checkpoints/RoboDemo-full50/pytorch_model.pt" \
--data-root "$ROBODEMO_DATA_ROOT" \
--task-level L0_TwoRobotFoldTowel-v1 \
--workers-per-gpu 1 --max-episodes 1 \
--output outputs/full50-smoke
This runs one 500-control-step episode. Use a new output directory for each run. Recorded integration tests cover all five checkpoints.
4. Run The Full Evaluation
Prepare all references required by this model's complete trial plan:
python scripts/prepare_assets.py --root "$ROBODEMO_HOST_ASSETS" \
--profile evaluation --model full50 --extract-assets
CUDA_VISIBLE_DEVICES=0 python evaluation/evaluate.py \
--model full50 \
--checkpoint "$ROBODEMO_HOST_ASSETS/checkpoints/RoboDemo-full50/pytorch_model.pt" \
--data-root "$ROBODEMO_DATA_ROOT" \
--output outputs/full50-evaluation
This evaluates all 2,000 episodes across 50 tasks.
The output contains summary.json, model_load.json, and per-trial
result.json, trajectory.npz and rollout.mp4 files.
The full usage guide
describes all model choices and output fields.
Container And Parallel Evaluation
The shared Docker environment provides pinned dependencies and driver/asset checks for all five models. A built Linux amd64 image is available from RoboDemo-Docker, with streaming download and mounted-data evaluation instructions. The release's expanded suite passed 113 CPU tests per policy variant inside the image. The image also passed a complete Full50 GPU simulator episode and both minimal examples. November 20 release checks. The GPU test used Apptainer on H200 and peaked at 21.38 GiB, including OptiX rendering. Image runtime and results and validation status document the tested configurations.
For managed notebook containers without Docker Engine, follow the Apptainer guide. New image builds are versioned separately and do not inherit earlier GPU results.
This checkpoint has two launchers:
- Single GPU:
scripts/evaluate/full50-single.sh - Eight GPUs:
scripts/evaluate/full50-8gpu.sh
Set CUDA_VISIBLE_DEVICES to allocated GPUs and start with
--workers-per-gpu 1. Use --max-episodes to control the global trial count.
Memory sizing and complete commands
explain the 32-GB/80-GB starting configurations, restart and result merging.
This checkpoint completed a full 500-step H200 simulator episode with
21.38 GiB peak process memory, including rendering.
Use 1 worker on 32 GB or 2 workers on 80 GB. A 24-GB card is the
minimum practical target for the measured setup; 32 GB leaves more task-dependent
rendering headroom. These capacity recommendations use H200 measurements.
A concurrent Full50 run
also completed four 500-step episodes on two H200s with two workers per GPU,
peaking at 42.77 GiB per GPU. Reducing episode count does not reduce
per-worker VRAM.
All five memory measurements and active CPU CI are available with the code.
For a small eight-GPU run on eight allocated devices:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
bash scripts/evaluate/full50-8gpu.sh \
--checkpoint "$ROBODEMO_HOST_ASSETS/checkpoints/RoboDemo-full50/pytorch_model.pt" \
--data-root "$ROBODEMO_DATA_ROOT" \
--workers-per-gpu 1 --max-episodes 8 \
--output outputs/full50-8gpu-smoke
This schedules eight episodes total. Omit --max-episodes for the complete
plan. The launcher is released; native eight-GPU execution remains to be
validated. Measured configurations.
Minimal inference examples include a real human video, robot observations/states/actions, and a separate query-conditioned retrieval workflow.
Model Interface
The evaluator provides a 224-by-224 RGB observation, language instruction, proprioceptive state and a task-matched reference video. The policy predicts 30-step action chunks with 16 active simulator action dimensions. Reference memory is initialized for each episode; execution memory updates from actual observations and previously applied actions.
Files
| File | Contents |
|---|---|
pytorch_model.pt |
Complete policy state dictionary |
policy.yaml |
Inference configuration for this checkpoint |
normalization.json |
Observation and action normalization |
NOTICE.md, licenses/ |
Upstream component attribution and licenses |
License And Citation
This model is distributed under Apache 2.0. Component notices and their
licenses are retained in NOTICE.md and licenses/. Benchmark data and
simulator assets are provided by Imitator Game. This checkpoint is released
for research evaluation; physical-robot deployment requires a compatible
controller and separate validation.
@misc{anonymous2027robodemo,
title = {RoboDemo: Reconciling Demonstration Intent and Execution
Progress for Long-Horizon Manipulation},
author = {Anonymous Authors},
year = {2027},
note = {Manuscript under review}
}
The Full50 two-call policy example and raw-media retrieval example have both completed end to end. See the validation record for H200 memory-budget measurements and example outputs.