Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Added FFMPEG for Blackfly #279

Merged
merged 7 commits into from
Nov 23, 2024
Merged

Feature: Added FFMPEG for Blackfly #279

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cf0100e
Split camera yaml from camera compression and added ffmpeg manual lau…
hilary-luo Nov 20, 2024
651c990
Added a camera ffmpeg tuning tutorial
hilary-luo Nov 20, 2024
d8aef56
Updated blackfly default yaml to match code
hilary-luo Nov 22, 2024
868c1cf
Added ffmpeg to blackfly yaml section
hilary-luo Nov 22, 2024
7cd25b4
Added RViz tutorial
hilary-luo Nov 22, 2024
adce802
Explain ffmpeg test setup further
hilary-luo Nov 22, 2024
fa67eb8
Rename the camera compression tutorial
hilary-luo Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 44 additions & 6 deletions docs_versioned_docs/version-ros2humble/components/yaml/sensors/flir_blackfly.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ camera:
# image_height: 1080
# offset_x: 0
# offset_y: 0
# binning_x: 1
# binning_y: 1
binning_x: 2
binning_y: 2
frame_rate_enable: true
frame_rate_auto: Off
frame_rate: 40.0
frame_rate: 30
```
</td>
</tr>
Expand Down Expand Up @@ -61,7 +61,7 @@ The resulting output of the command should appear as:
ATTR{serial}=="013686A9"
```

Then, convert attirbute serial hexadecimal value to decimal representation and set it to the `serial_number` parameter. Convert values from hexadecimal to decimal representation using the following terminal command, where `HEX` needs to be swapped with the hexadecimal value:
Then, convert attribute serial hexadecimal value to decimal representation and set it to the `serial_number` parameter. Convert values from hexadecimal to decimal representation using the following terminal command, where `HEX` needs to be swapped with the hexadecimal value:
```bash
echo $((16#HEX))
```
Expand All @@ -71,6 +71,44 @@ For the example above, the hexadecimal serial number `013686A9` would be convert
#### Resolution
The `image_height` and `image_width` parameters define the area of the raw image to crop. It does not downsample the image. Use the `offset_x` and `offset_y` parameters to change the origin of the cropped area. These parameters would be best used to define an area of interest.

To reduce the size of the image using downsamlping, use the `binning_x` and `binning_y` parameters. These parameters define the number of pixels that will be binned together in each axis. If both `binning_x` and `binning_y` are set to `2`, then the resolution of the image will be halved.
To reduce the size of the image using downsampling, use the `binning_x` and `binning_y` parameters. These parameters define the number of pixels that will be binned together in each axis. If both `binning_x` and `binning_y` are set to `2`, then the resolution of the image will be quartered.

Reducing the resolution of the image improves the perfomance of the camera driver and network transit of image packets.
Reducing the resolution of the image improves the performance of the camera driver and network transit of image packets.

#### Image Compression
By default, the camera feed is also available in compressed formats including `compressed` (JPEG) and `ffmpeg` (H.264). Each of these topics is made available by an image transport plugin. Which plugins are active can be controlled by adding the following ROS parameters to the Blackfly section of the `robot.yaml`:

```yaml
ros_parameters:
flir_blackfly:
serial_number: ''
image_debayer:
image_mono:
enable_pub_plugins:
- image_transport/compressed
- image_transport/ffmpeg
- image_transport/raw
image_color:
enable_pub_plugins:
- image_transport/compressed
- image_transport/ffmpeg
- image_transport/raw
```

#### FFMPEG Compression Settings
To modify the FFMPEG compression settings, add the following ROS parameters to the Blackfly section of the `robot.yaml`:

```yaml
ros_parameters:
flir_blackfly:
serial_number: ''
image_debayer:
ffmpeg_image_transport:
qmax: 40
preset: superfast
tune: zerolatency
bit_rate: 1000000
gop_size: 15
```

For documentation on these parameters see the [ffmpeg_image_transport GitHub repo](https://github.com/ros-misc-utilities/ffmpeg_image_transport).
167 changes: 167 additions & 0 deletions docs_versioned_docs/version-ros2humble/ros/config/camera_compression.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
title: Camera Data Compression
sidebar_label: Camera Data Compression
sidebar_position: 7
toc_min_heading_level: 2
toc_max_heading_level: 4
---

Using the [`image_transport`](https://github.com/ros-perception/image_common), [`image_transport_plugins`](https://github.com/ros-perception/image_transport_plugins) and [`ffmpeg_image_transport`](https://github.com/ros-misc-utilities/ffmpeg_image_transport) packages, it is possible to compress and stream images/video using ROS messages. This is important in order to enable the transmission of large images or video streams over wireless networks with limited bandwidth. For a walkthrough of a case study see [Video over WiFi](../tutorials/video_over_wifi.mdx). The supported compression methods are outlined below:

| Image Transport Name | Topic Name | Compression Method | Purpose | Comments |
| :------------------- | :---------------- | :--------------------- |:------------------ |:--------- |
| Compressed | `compressed` | JPEG (default) or PNG | RGB or Mono Images | Use JPEG for higher compression, use PNG for lossless |
| Compressed Depth | `compressedDepth` | PNG (or RVL in Jazzy+) | Depth Images | Both PNG and RVL are lossless |
| FFMPEG | `ffmpeg` | H.264 or H.265 | Video | Requires decoding in order to view the data |
| Theora | `theora` | Theora | Video | Not recommended because the subscriber must receive the very first published message in order to decompress the stream |

For example, for the color image topic from the first camera in `robot.yaml`, the default image transport topics will appear as shown below:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/compressed
/a200_0000/sensors/camera_0/color/ffmpeg
```

:::note
The same publisher handles all of these message types. Limit the number of subscribers to minimize the load on the camera driver node.
:::


#### Using Image Transport

Compressed, FFMPEG and Theora types are not standard ROS 2 datatypes, therefore special consideration must be made when interacting with them. Nodes that use `image_transport` have the ability to directly subscribe to or publish to the compressed / encoded topics using the image transport plugins. Many ROS 2 camera drivers offer the compressed topics by default.

However, for any node that does not support these plugins by default, a separate compression (encoding) or decompression (decoding) "republisher" node must be run. Decompressing the images into standard `sensor_msgs/Image` messages allow viewing or processing them using standard tools such as RViz. However, these "republisher" nodes are inefficient because they introduce another subscriber and publisher, and this can cause significant load on the computer networking depending on the size of image and networking configuration. Supporting image transport and image transport plugins directly is much more efficient, avoiding the need for copying and transferring the uncompressed data. This should be done in any situation where the performance of "republisher" nodes is insufficient. Examples on how to directly use image transport can be found in the [Image Transport Tutorials Repo](https://github.com/ros-perception/image_transport_tutorials).

The `clearpath_sensors` package has launch files to start "republisher" nodes to convert to and from these compressed transport types. This allows the usage of compressed topics to optimize networking without requiring custom development. To use these launch files, ensure that the Clearpath sensor package is installed on the computer that will be performing the decompression / decoding:

```bash
sudo apt install ros-humble-clearpath-sensors
```

:::note
These launch files simply wrap the `image_transport` `republish` node and set default parameters. This `republish` node can be run directly as an alternative to installing the full Clearpath sensors package and using the launch files.
:::

#### Compressed

The `compressed` image transport compresses images using the JPEG format at 80% quality by default. JPEG is a static image compression format that compresses each video frame independently and does not rely on any previous frame.

##### Encoding Node {#encode-compressed}

Use the launch file `image_raw_to_compressed.launch.py` to compress `sensor_msgs/Image` messages into compressed messages. Note that this is only required if the image publisher does not already use image transport plugins.

Assume that the following image topic needs to be compressed:
```bash
/a200_0000/sensors/camera_0/color/image
```

Then, compress this image topic by launching the following:
```bash
ros2 launch clearpath_sensors image_raw_to_compressed.launch.py in_raw:=/a200_0000/sensors/camera_0/color/image out_compressed:=/a200_0000/sensors/camera_0/color/compressed
```

Now, the following topics will be available:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/compressed
```

##### Decoding Node {#decode-compressed}

To reverse this process on the receiving computer, use the `image_compressed_to_raw.launch.py` to decompress back to `sensor_msgs/Image` messages:
```bash
ros2 launch clearpath_sensors image_compressed_to_raw.launch.py in_compressed:=/a200_0000/sensors/camera_0/color/compressed out_raw:=/a200_0000/sensors/camera_0/decompressed/image
```

Now, the following topics will be available:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/compressed
/a200_0000/sensors/camera_0/decompressed/image
```

#### FFMPEG

The FFMPEG image transport uses H.264 compression by default. In the context of real-time video transmission, this compression technique uses a combination of keyframes which contain the full data from the video frame, and predictive frames which encode only the changes compared to the previous frame. This provides very high levels of compression for video feeds.

##### Encoding Node {#encode-ffmpeg}

Use the launch file `image_raw_to_ffmpeg.launch.py` to compress `sensor_msgs/Image` messages into FFMPEG messages. Note that this is only required if the image publisher does not already use image transport plugins.

Assume that the following image topic needs to be compressed:
```bash
/a200_0000/sensors/camera_0/color/image
```

Then, compress this image topic by launching the following:
```bash
ros2 launch clearpath_sensors image_raw_to_ffmpeg.launch.py in_raw:=/a200_0000/sensors/camera_0/color/image out_ffmpeg:=/a200_0000/sensors/camera_0/color/ffmpeg
```

There are several optional parameters that can be passed in order to adjust the compression:

| Parameter | Default Value |
|-----------|---------------|
| `encoding`| `libx264` |
| `qmax` | `40` |
| `preset` | `superfast` |
| `tune` | `zerolatency` |
| `bit_rate`| `1000000` |
| `gop_size`| `15` |

For documentation on what each of these parameters does see the [ffmpeg_image_transport GitHub repo](https://github.com/ros-misc-utilities/ffmpeg_image_transport).

Now, the following topics will be available:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/ffmpeg
```

##### Decoding Node {#decode-ffmpeg}

To reverse this process on the receiving computer, use the `image_ffmpeg_to_raw.launch.py` to decode back to `sensor_msgs/Image` messages:
```bash
ros2 launch clearpath_sensors image_ffmpeg_to_raw.launch.py in_ffmpeg:=/a200_0000/sensors/camera_0/color/ffmpeg out_raw:=/a200_0000/sensors/camera_0/decoded/image
```

Now, the following topics will be available:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/ffmpeg
/a200_0000/sensors/camera_0/decoded/image
```

#### Theora

Theora is an older format that also uses keyframes and predictive frames, like the H.264 format, to compress video streams. The Theora image transport encodes images into a video stream with an 800 kbps bitrate by default.

The first message after launching the encoder transmits the header packet. This header packet includes all of the encoding information required to decode the subsequent video packets. Therefore, if that first message is dropped or missed, then decoding the video packets back to images is not possible.

Therefore, it is important to have the decoder node running before launching the encoder node, such that the former will not miss the header packet.

:::warning
Missing the header packet and being unable to decode the video packets is a known, long-standing issue with the `theora` image transport plugin. See this [issue](https://github.com/ros-perception/image_transport_plugins/issues/4) for more information.
:::

Assume that the following image topic needs to be compressed:
```bash
/a200_0000/sensors/camera_0/color/image
```

First, launch the decoder:
```bash
ros2 launch clearpath_sensors image_theora_to_raw.launch.py in_theora:=/a200_0000/sensors/camera_0/color/theora out_raw:=/a200_0000/sensors/camera_0/decoded/image
```

Then, launch the encoder:
```bash
ros2 launch clearpath_sensors image_raw_to_theora.launch.py in_raw:=/a200_0000/sensors/camera_0/color/image out_theora:=/a200_0000/sensors/camera_0/color/theora
```

As long as the first header packet is communicated without loss, the decoder will be able to decode and publish the raw image data. The following topics will then be available:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/theora
/a200_0000/sensors/camera_0/decoded/image
```
108 changes: 11 additions & 97 deletions docs_versioned_docs/version-ros2humble/ros/config/yaml/sensors/cameras.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,97 +15,20 @@ Cameras are sensors that publish:

This broad definition allows us to label a variety of different cameras under a single banner. Stereo cameras, for example, could publish multiple images, disparity maps, and even pointclouds. Other cameras have in built inertial measurement units that publish `sensor_msgs/Imu`. Ultimately, all imaging devices fall into this category.

For further details on how to work with large camera data, see [Camera Data Compression](../../camera_compression.mdx)

## Compressed and Theora Topics
Using the `image_transport` package and the [image_transport_plugins](https://github.com/ros-perception/image_transport_plugins), it is possible to send compressed and stream images using ROS 2 messages.
## Supported Cameras

For example, the color image topic will have multiple versions:
```bash
/a200_0000/sensors/camera_0/color/image
/a200_0000/sensors/camera_0/color/compressed
/a200_0000/sensors/camera_0/color/theora
```

:::note

The same publisher handles all three of these message types. Limit the number of subscribers to minimize the load on the camera driver node.

:::

Compressed and theora types are not standard ROS 2 types, and therefore, it is required to decompress the images into standard `sensor_msgs/Image` messages before attempting to view them using standard tools such as RViz.

The `clearpath_sensors` package has launch files to convert from and to these compressed transport types. Therefore, make sure to have the package installed on the computer that will be doing the decompressing:
```bash
sudo apt install ros-humble-clearpath-sensors
```

### Compressed
Compressed image types are not serialized. These are images compressed using the JPEG format at 80% quality.

Use the launch file `image_raw_to_compressed.launch.py` to compress `sensor_msgs/Image` messages into compressed messages.

Assume we have the following image topic we would like to compress:
```bash
/a200_0000/sensors/camera_0/raw/image
```

Then, we can compress this image by launching the following:
```bash
ros2 launch clearpath_sensors image_raw_to_compressed.launch.py in_raw:=/a200_0000/sensors/camera_0/raw/image out_compressed:=/a200_0000/sensors/camera_0/raw/compressed
```

Now, we will have the following topics:
```bash
/a200_0000/sensors/camera_0/raw/image
/a200_0000/sensors/camera_0/raw/compressed
```

To reverse this process, use the `image_compressed_to_raw.launch.py` to decompress back to `sensor_msgs/Image`:
```bash
ros2 launch clearpath_sensors image_compressed_to_raw.launch.py in_compressed:=/a200_0000/sensors/camera_0/raw/compressed out_raw:=/a200_0000/sensors/camera_0/decompressed/image
```

Now, we will have the following topics:
```bash
/a200_0000/sensors/camera_0/raw/image
/a200_0000/sensors/camera_0/raw/compressed
/a200_0000/sensors/camera_0/decompressed/image
```

### Theora
Theora topics encode images into a video stream. By default, these streams have an 800 kb/s rate.

Theora messages are not serialized, however, the first message after launching the encoder transmits the header packet. This header packet includes all of the encoding information required to decode the subsequent video packets. Therefore, if that first message is dropped or missed, then decoding the video packets back to images is not possible.

Therefore, it is important to have the decoder node running before launching the encoder node, such that the former will not miss the header packet.

:::warning
Missing the header packet and being unable to decode the video packets is a known, long-standing issue with the `theora` image transport plugin. See this [issue](https://github.com/ros-perception/image_transport_plugins/issues/4) for more information.
:::

Assuming we have the following topic that we would like to stream:
```bash
/a200_0000/sensors/camera_0/raw/image
```

First, launch the decoder:
```bash
ros2 launch clearpath_sensors image_theora_to_raw.launch.py in_theora:=/a200_0000/sensors/camera_0/raw/theora out_raw:=/a200_0000/sensors/camera_0/decoded/image
```
### Intel Realsense
<IntelRealsense/>

Then, launch the encoder:
```bash
ros2 launch clearpath_sensors image_raw_to_theora.launch.py in_raw:=/a200_0000/sensors/camera_0/raw/image out_raw:=/a200_0000/sensors/camera_0/raw/theora
```
### Flir BlackflyS
<FlirBlackfly/>

As long as the first header package is communicated without drops, the decoder will be able to decode and publish the raw image data:
```bash
/a200_0000/sensors/camera_0/raw/image
/a200_0000/sensors/camera_0/raw/theora
/a200_0000/sensors/camera_0/decoded/image
```
### Stereolabs Zed
<StereolabsZed/>

## Republishers
## Post Processing Republishers {#republishers}
All cameras publish to the `color/image` topic of type `sensor_msgs/Image`. However, we understand that sometimes this image may require downsampling, rectification, or cropping. To facilitate post processing images, we have included a method to easily add image processing nodes, leveraging composable nodes to maximize efficiency.

:::note
Expand All @@ -127,7 +50,7 @@ Configuring a republisher can be done by setting its node parameters using the `
### Rectify
The `rectify` node takes the `camera_info` topic as a source of calibration parameters and applies an interpolation to rectify the raw images.

To add a the rectify republisher add the following entries to the `camera` entry and its corresponding `ros_parameters` section.
To add a rectify republisher, add the following entries to the `camera` entry and its corresponding `ros_parameters` section.
```yaml
cameras:
- model: CAMERA_MODEL
Expand Down Expand Up @@ -158,7 +81,7 @@ All UPPERCASE entries must be replaced based on the specific camera being used.
### Resize
The `resize` node uses the input image and resizes it.

To add a resize republisher add teh following entreis to the `camera` entry and its corresponding `ros_parameters` section.
To add a resize republisher, add the following entries to the `camera` entry and its corresponding `ros_parameters` section.
```yaml
cameras:
- model: CAMERA_MODEL
Expand Down Expand Up @@ -195,12 +118,3 @@ The `interpolation` parameters must be one of the following:
- **2**: Cubic.
- **3**: Area. Resampling using pixel area relation.
- **4**: Lanczos4. Lanczos interpolation over 8x8 neighbourhood.

## Intel Realsense
<IntelRealsense/>

## Flir BlackflyS
<FlirBlackfly/>

## Stereolabs Zed
<StereolabsZed/>
Binary file added docs_versioned_docs/version-ros2humble/ros/tutorials/img/rviz_add_button.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ..._versioned_docs/version-ros2humble/ros/tutorials/img/rviz_add_visualization.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs_versioned_docs/version-ros2humble/ros/tutorials/img/rviz_blank.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs_versioned_docs/version-ros2humble/ros/tutorials/img/rviz_laserscan.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs_versioned_docs/version-ros2humble/ros/tutorials/img/rviz_no_model.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs_versioned_docs/version-ros2humble/ros/tutorials/img/view_robot.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs_versioned_docs/version-ros2humble/ros/tutorials/manipulation/_category_.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"label": "Manipulation Demos",
"position": 4
"position": 5
}
2 changes: 1 addition & 1 deletion docs_versioned_docs/version-ros2humble/ros/tutorials/navigation_demos/_category_.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"label": "Navigation Demos",
"position": 3
"position": 4
}
Loading
Loading