Visualizing the pointclouds saved in a trajectories hdf5 file

[jmorgan-bdai]

Hi,

Is there a convenient way to visualize the pointclouds (or rgbd's) saved in a trajectory? Something like the open3d visualizer (https://www.open3d.org/html/python_api/open3d.visualization.Visualizer.html).

Thanks

[StoneT2000]

Currently there is not a convenient way through maniskill to visualize trajectory data (a contribution for that would be great and I think it would be a useful tool). It is best if you use a tool like open3d or trimesh to load the PCD data or just use matplotlib plt.imshow to render the RGB and depth data. Documentation on the trajectory file formats are here: https://maniskill.readthedocs.io/en/latest/user_guide/datasets/demos.html#trajectory-data-hdf5.

For docs on the observation dictionary formats, we detail all of them based on a given obs_mode here: https://maniskill.readthedocs.io/en/latest/user_guide/concepts/observation.html

Usually something like

h5_file["traj_0"]["obs"]["sensor_data"]["base_camera"]["rgb"] is probably what you want. Most environments have a camera called base_camera.

Otherwise we do have a trajectory replay tool but the only visualization supported is to save videos or open the viewer and watch the replay.

[StoneT2000]

Oh I just remember actually our trajectory replay tool does support recording the visual sensor data into a video actually. All environments have a render mode called "sensors" which make env.render() return a rgb array showing all the visual sensor outputs.

python -m mani_skill.trajectory.replay_trajectory \
  --traj-path path/to/trajectory.h5 \
  --use-env-states --render-mode="sensors" --save-video

probably works, and will save a video visualizing the registered cameras for policy observations in the environment to the same folder the trajectory is in.