This repository provides an H.264 image encoder and decoder that leverages the specialized hardware in NVIDIA GPUs and the Jetson platform. The isaac_ros_h264_encoder package can compress an image into H.264 data using the NVENC on the Jetson platform. The isaac_ros_h264_decoder package can decode the H.264 data into original images using the NVDEC on x86 platform with NVIDIA GPUs.
Image compression reduces the data footprint of images when written to storage or transmitted between computers. A 1080p camera at 30fps produces 177MB/s of data; image compression reduces this by approximately 10 times to 17MB/s of data reducing the throughput needed to send this to another computer or write out to storage; a one minute 1080p camera recording is reduced from ~10GB to ~1GB. This compression is provided by dedicated hardware acceleration (NvEnc) separate from other hardware engines such as the GPU.
A common use case for image compression during the development of robots is to capture camera images to storage. This captured data is processed offline from the robot to produce training datasets for AI models, test datasets for perception functions, and test data for open-loop re-simulation of software in development with real data. The compression parameters used are tuned to minimize visual quality reduction from lossy compression for AI model and perception function development. Compression reduces the amount of data written to storage, the time required to offload the recording, and footprint of the data at rest in a data lake.
Compression can be used with event data recorders that capture camera images to storage when an events of interest occurs, often due to failures on the robot. This provides visual information to assist in the debugging of the event, or to improve perception and robot functions.
H.264 is an efficient and popular compression algorithm with broad support across many platforms. The output of the isaac_ros_h264_encoder package run on Jetson can then be decoded with hardware acceleration using the isaac_ros_h264_decoder on x86_64 systems or by third-party H.264 decoder packages on non-NVIDIA powered platforms.
Check your requirements against package input limitations.
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
The following are the benchmark performance results of the prepared pipelines in this package, by supported platform:
| Pipeline | AGX Orin | Orin Nano | x86_64 w/ RTX 3060 Ti |
|---|---|---|---|
| H.264 encoder node | 170 fps 17.4ms |
N/A | N/A |
| H.264 decoder node | N/A | N/A | 400 fps 2.3ms |
These data have been collected per the methodology described here.
- Isaac ROS Compression
Update 2022-10-19: Initial release of isaac_ros_h264_encoder and isaac_ros_h264_decoder
isaac_ros_h264_encoder is designed and tested to be compatible with ROS2 Humble running on Jetson.
isaac_ros_h264_decoder is designed and tested to be compatible with ROS2 Humble running on x86_64 system with an NVIDIA GPU.
| Platform | Hardware | Software | Notes |
|---|---|---|---|
| Jetson | Jetson Orin Jetson Xavier |
JetPack 5.0.2 | For best performance, ensure that the power settings are configured appropriately. |
| x86_64 | NVIDIA GPU | Ubuntu 20.04+ CUDA 11.6.1+ |
To simplify development, we recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.
Note: All Isaac ROS quick start guides, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
This quickstart shows an example of how to use the isaac_ros_h264_decoder with a pre-recorded rosbag, which contains compressed H.264 images generated from isaac_ros_h264_encoder with two argus cameras as the input source. You will be able to visualize the decoded images after the last step.
Warning: step 7 & 8 must be performed on an
x86_64platform with an NVIDIA GPU.
-
Set up your development environment by following the instructions here.
-
Clone this repository and its dependencies under
~/workspaces/isaac_ros-dev/src.cd ~/workspaces/isaac_ros-dev/src
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_compression
-
Pull down a rosbag of sample data:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_compression && \ git lfs pull -X "" -I "resources/rosbags/h264_compressed_sample.bag"
-
Launch the Docker container using the
run_dev.shscript:cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
-
Inside the container, build and source the workspace:
cd /workspaces/isaac_ros-dev && \ colcon build --symlink-install && \ source install/setup.bash
-
(Optional) Run tests to verify complete and correct installation:
colcon test --executor sequential -
Run the following launch files to run the demo of this
isaac_ros_h264_decoder:ros2 launch isaac_ros_h264_decoder isaac_ros_h264_decoder_rosbag.launch.py rosbag_path:=/workspaces/isaac_ros-dev/src/isaac_ros_compression/resources/rosbags/h264_compressed_sample.bag
-
Open a second terminal and attach to the container:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
-
Visualize and validate the output of the package:
ros2 run image_view image_view --ros-args -r image:=/left/image_uncompressed
ros2 run image_view image_view --ros-args -r image:=/right/image_uncompressed
To continue exploring the Compression packages, check out the following suggested examples:
- Tutorial with RealSense and H.264 data recording
- Tutorial with Argus-compatible camera and H.264 data recording
- Tutorial with rosbag playback and software-based decoder
To customize your development environment, reference this guide.
# Must be run on a Jetson platform.
ros2 launch isaac_ros_h264_encoder isaac_ros_h264_encoder.launch.py input_width:=<"your input image width"> input_height:=<"your input image height">| ROS Parameter | Type | Default | Description |
|---|---|---|---|
input_width |
uint32_t |
1920 |
The width of the input image. |
input_height |
uint32_t |
1200 |
The height of the input image. |
qp |
uint32_t |
20 |
The encoder constant QP value. |
hw_preset |
uint32_t |
3 |
The encoder hardware preset type. The value can be an integer from 0 to 3, representing Ultrafast, Fast, Medium and, Slow, respectviely. |
profile |
uint32_t |
0 |
The profile to be used for encoding. The value can be an integer from 0 to 2, representing Main, Baseline, and High, respectviely. |
config |
std::string |
lossy |
A preset combination of qp, hw_preset, and profile. The value can be lossy or custom. When lossy, the default value of these three parameters will be used. Only when this field is custom will the custom values will be used. |
| ROS Topic | Type | Description |
|---|---|---|
image_raw |
sensor_msgs/Image | Raw input image. |
Limitation: All input images are required to have height and width that are both an even number of pixels.
| ROS Topic | Interface | Description |
|---|---|---|
image_compressed |
sensor_msgs/CompressedImage | H.264 compressed image. |
- The input image resolution must be the same as the dimension you provided, and the resolution must be no larger than
1920x1200. - The input image should be in
rgb8orbgr8format, and it will be converted tonv12format before being sent to the encoder.
- The encoder will perform All-I frame encoding and output the H.264 compressed data.
# Must be run on a x86_64 platform.
ros2 launch isaac_ros_h264_decoder isaac_ros_h264_decoder.launch.py input_width:=<"your original image width"> input_height:=<"your original image height">| ROS Parameter | Type | Default | Description |
|---|---|---|---|
input_width |
uint32_t |
1920 |
The width of the original image |
input_height |
uint32_t |
1200 |
The height of the original image |
| ROS Topic | Interface | Description |
|---|---|---|
image_compressed |
sensor_msgs/CompressedImage | The H.264 compressed image |
| ROS Topic | Type | Description |
|---|---|---|
image_uncompressed |
sensor_msgs/Image | The uncompressed image with RGB8 encoding. |
- The input resolution must be no larger than
1920x1200and must be the same as the original image resolution.
- The
isaas_ros_h264_decoderpackage outputs a uncompressed image with the same resolution as the original image. The output image will be inRGB8format.
For solutions to problems with Isaac ROS, check here.
| Date | Changes |
|---|---|
| 2022-10-19 | Initial release |