Problem with using depth image with nvblox

[Car38639]

Hi, I would like to ask some questions about using depth images with nvblox.

So I tried realsense_reacher.py and it works fine. Then I add scene obstacles into world_cfg, with part of the obstacles being in front of the camera_pos, and the result is like this:

It seems that the camera detects both realsense data (physical world) and scene obstacles(virtual world) and sets up voxels based on them.

I want to make world_model sets up voxels only based on the real world objects(from depth camera), while motion_gen calculates collisions based on both depth image obstacles and scene obstacles.

I’m looking for possible solutions, and notice there’s a description in the official document “Using with Depth Camera”:

“In addition, nvblox enables decaying occupancy in layers, which we leverage to account for dynamic obstacles. ……This allows us to load static world objects in a base layer, dynamic obstacles in a dynamic layer, and interaction objects in a high resolution map.”

I can’t figure out how exactly this can be achieved. Could you give some guidance or examples of separate obstacles to different layers?

Or if there’s a way I can set up camera_frame to ignore the scene obstacles?

[balakumar-s]

The voxel visualizer queries cuRobo's collision world which will include static obstacles and obstacles sensed by realsense. Internally, it will run a for loop over different world representations to find collisions.

The nvblox part of the world only reads the realsense camera, so it's working as you mention. However, the current implementation of the voxel visualizer in the isaac sim examples doesn't differentiate between nvblox obstacles and mesh/cuboid obstacles.