Generalist Robot Policy Evaluation in Simulation with NVIDIA Isaac Lab-Arena and LeRobot
This blog was co-authored with Steven Palma, Jade Choghari of Hugging Face
Robots must perceive unstructured environments, reason and plan under uncertainty, and execute actions safely in real time on physical systems. To do this, they need to develop an understanding of the physical world through three forms of computational intelligence: training, simulation, and on-robot inference. Each stage relies on distinct hardware and software, and each plays a critical role in the end-to-end robotics development pipeline.
Robotics is the fastest growing segment on Hugging Face, with NVIDIA’s open robotics models and datasets leading platform downloads. NVIDIA and Hugging Face are integrating NVIDIA’s open Isaac and GR00T technologies, including robot simulation and learning frameworks, models, and hardware systems, across these three computers into the LeRobot library.
This will accelerate open-source physical AI development by bringing NVIDIA's 2 million robotics developers and Hugging Face’s more than 13 million AI builders together.
Developers get access to open-source pre-trained Isaac GR00T N vision language action (VLA) models, physical AI datasets, evaluation frameworks like NVIDIA Isaac Lab-Arena, and hardware platforms such as the Reachy 2 humanoid which will run on NVIDIA Jetson Thor.
In this blog, we show how to evaluate VLA policies using Isaac Lab-Arena in LeRobot EnvHub, making robot environments easier to author, share, and reuse across the community.
NVIDIA Isaac Lab-Arena and LeRobot Integration
Isaac Lab-Arena is an open source framework for efficient and scalable robotic policy evaluation in simulation, with the evaluation and task layers designed in close collaboration with Lightwheel.
Hugging Face’s LeRobot EnvHub feature enables developers to share simulation environments, and easily load them for training, evaluation, or teleoperation, directly from the LeRobot framework.
Isaac Lab-Arena is now integrated into Hugging Face’s LeRobot Environment Hub, making it easier for developers to build, experiment with, and collaborate on robotics simulation. Developers can now easily prototype complex and diverse simulation environments using Isaac Lab-Arena, register these environments on LeRobot EnvHub and seamlessly use them to train and evaluate robot policies, such as GR00T N, Pi, SmolVLA, among other policies available in LeRobot. All of this happens within the LeRobot ecosystem, providing streamlined access to powerful simulation and training tools.
How to Evaluate a VLA on Isaac Lab-Arena Environments available on LeRobot
Prerequisites
- Hardware requirements for Isaac Lab-Arena are shared with Isaac Sim, and are detailed here.
- NVIDIA driver compatible with IsaacSim 5.1.0
- Linux (Ubuntu 22.04 / 24.04)
Setup
# 1. Create conda environment
conda create -y -n lerobot-arena python=3.11
conda activate lerobot-arena
conda install -y -c conda-forge ffmpeg=7.1.1
# 2. Install Isaac Sim 5.1.0
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com
# Accept NVIDIA EULA (required)
export ACCEPT_EULA=Y
export PRIVACY_CONSENT=Y
# 3. Install IsaacLab 2.3.0
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
git checkout v2.3.0
./isaaclab.sh -i
cd ..
# 4. Install Isaac Lab-Arena
git clone https://github.com/isaac-sim/IsaacLab-Arena.git
cd IsaacLab-Arena
git checkout release/0.1.1
pip install -e .
cd ..
# 5. Install LeRobot
git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e .
cd ..
# 6. Install additional dependencies
pip install onnxruntime==1.23.2 lightwheel-sdk==1.0.1 vuer[all]==0.0.70 qpsolvers==4.8.1
pip install numpy==1.26.0 # Isaac Sim 5.1 depends on numpy==1.26.0, this will be fixed in next release
Evaluate SmolVLA
In the first stage, install SmolVLA on your machine following the command below. Please also remember to install numpy to 1.26.0
pip install -e ".[smolvla]"
pip install numpy==1.26.0 # revert to numpy version is 1.26
When the installation is completed, we can run the LeRobot evaluation on Isaac Lab-Arena with the command below
lerobot-eval \
--policy.path=nvidia/smolvla-arena-gr1-microwave \
--env.type=isaaclab_arena \
--env.hub_path=nvidia/isaaclab-arena-envs \
--rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
--policy.device=cuda \
--env.environment=gr1_microwave \
--env.embodiment=gr1_pink \
--env.object=mustard_bottle \
--env.headless=false \
--env.enable_cameras=true \
--env.video=true \
--env.video_length=10 \
--env.video_interval=15 \
--env.state_keys=robot_joint_pos \
--env.camera_keys=robot_pov_cam_rgb \
--trust_remote_code=True \
--eval.batch_size=1
If you want to go more in-depth and also evaluate PI0.5 follow the documentation on the LeRobot documentation.
How to create and register new Isaac Lab-Arena Environments on LeRobot EnvHub
While the previous section demonstrated a sample workflow using Isaac Lab-Arena sample environments already available on EnvHub, developers are encouraged to share custom environments with the community.
- Create your own environments and task suites on Isaac Lab-Arena, and easily swap embodiments, objects and scenarios by following instructions here.
- Register your Isaac Lab-Arena based environments on LeRobot EnvHub by following instructions here.
Lightwheel Robocasa and LIBERO Task Suites on Isaac Lab-Arena
Lightwheel has adopted the Isaac Lab-Arena framework to create and open-source 250+ tasks through the Lightwheel-RoboCasa-Tasks and Lightwheel-LIBERO-Tasks suites - LeRobot developers can now seamlessly use them thanks to this integration work.
Once a policy has been evaluated in simulation, it can be deployed on robot systems such as Reachy 2, powered by NVIDIA Jetson Thor⚡
Get Started
Here are some additional resources to explore how to work with Isaac Lab-Arena and LeRobot:
- Collect and prepare data: Capture real-world data using the SO-101 robot within a simulated Isaac Lab environment, and post-train policies using open-source datasets on Hugging Face, including the Lightwheel Tasks Dataset.
- Work with open VLA models: Download open GR00T N models and review the LeRobot GR00T N 1.5 policy documentation. Watch the technical livestream with the Hugging Face team for a deep dive into GR00T N 1.5 and LeRobot integration.
- Train and evaluate at scale: Isaac Lab-Arena pre-alpha is open source, and we invite you to help guide its future design and development. Build environments, benchmark policies, and contribute reusable environments through the LeRobot Environment Hub. Get started with the GitHub repository, technical blog, and documentation.
- Get started easily with NVIDIA Brev: Use NVIDIA Brev to quickly provision GPU instances for NVIDIA Isaac Lab workloads, and watch the Brev technical livestream for guidance on running Isaac Lab efficiently.
*This blog was co-authored with Steven Palma, Jade Choghari of Hugging Face
