7. Datasets
RoboHive offers extensive support for a diverse range of datasets known as RoboSet. These datasets are designed to facilitate pre-training and offline learning research for robotics. RoboSet was created to fill the gap that there is no large-scale robotics datasets in a uniform format that is suitable for robot learning.
All data can be downloaded here and is made available using a MIT license.
Each instance of the dataset is a robot trajectory, capturing essential information at each time step. These trajectories comprise observations, actions, rewards, RGB visuals from multiple camera views, and other relevant environmental information.
RoboSet supports two types of trajectories: human teleoperation trajectories and expert trajectories. Human teleoperation trajectories are collected through human teleoperation using Puppet, while the expert trajectories are rollouts from a trained task-specific NPG policy for the target task.
RoboSet spans across a diverse set of manipulation environments. Please refer to the table below for the specific number of instances available in each environment.
| Domain | # Trajs | # Tasks | World | Visuals | Source |
|---|---|---|---|---|---|
| Real kitchen | 31000 | 40 | Real | 4 cam | Human TeleOp |
| Franka kitchen | 600 | 4 | Sim | 4 cam | Human TeleOp |
| Hand Manipulation Suite | 25 | 4 | Sim | 3 cam | Human TeleOp |
| Franka kitchen | 75 | 5 | Sim | 4 cam | Expert Policy |
| Hand Manipulation Suite | 75 | 4 | Sim | 3 cam | Expert Policy |
| D'Kitty | 75 | 3 | Sim | 4 cam | Expert Policy |
| File Name | Description | Example Video |
|---|---|---|
| Kitchen(human) | This dataset includes a comprehensive collection of real-world trajectories collected via human teleoperators on table-top manipulation settings with a focus to fuel diversity and generalization in real world robot learning. | Video |
| HMS(expert) | This dataset includes expert trajectories generated on each Hand Manipulation environment using the policies from the original DAPG repo. Each environment has 25 recorded trajectories. | Video |
| HMS(human) | This dataset includes human teleoperation trajectories on each Hand Manipulation environment. These are obtained by replaying the trajectories provided in the original DAPG repo on corresponding RoboHive environements. | Video |
| FK1-v4(expert) | This dataset includes expert trajectories generated on each kitchen environment using open-sourced NPG policies from here. Each environment has 25 recorded trajectories. | Video |
| FK1-v4(human) | This dataset includes human teleoperation trajectories replayed using trajectories from the original Relay repo. The trajectories are play data, which complete a few tasks in a row. This dataset has in total 603 trajectories. | Video |
Please see the performance of baseline methods trained on each dataset on our Baseline page here.
RoboSet is created by researchers and students at the University of Washington, the University of Texas at Austin, Carnegie Mellon University, and Meta AI.