CLVR Jaco Play Dataset is a dataset focused on the field of remote control robots. It was released by a research team from the University of Southern California and KAIST in 2023. It provides 1,085 clips of the remote control robot Jaco 2 with corresponding language annotations. This dataset is a very valuable resource for scientists and developers studying fields such as robot remote control, natural language processing, and human-computer interaction.

The dataset is collected at a frequency of 10Hz. Its structure is as follows:

• The observations are divided into 5 attributes,
  • front_cam_ob : Observation results from the third-person camera
  • mount_cam_ob: Observe from the installed camera
  • ee_cartesian_pos_ob: End effector Cartesian position. ee_cartesian_pos_ob[0:3] corresponds to the position and ee_cartesian_pos_ob[3:7] corresponds to the orientation in quarter format
  • ee_cartesian_vel_ob: Cartesian velocity of the end effector. ee_cartesian_pos_ob[0:3] corresponds to position change, ee_cartesian_pos_ob[3:6] corresponds to direction change in roll, pitch and yaw format
  • joint_pos_ob: The joint positions of the Jaco arm (we only use the last 2 elements corresponding to the gripper joints)
• action: The first 3 elements are Cartesian increments, and the 4th element is a label from {0, 1, 2}, representing {open gripper, do not move gripper, close gripper}
• terminal: Each skill has 1 at the end
• hint: Natural language description of the target
• award: 1 at the end of each skill