realsense_bot

This is a ROS package for Intel realsense D435i with 3-DOF Manipulator robot that can be used for Indoor Mapping and localization of objects in the world frame with an added advantage of the robot's dexterity. The 3-DOF Manipulator is a self-built custom robot where the URDF with the depth sensor is included. The package covers the Rosserial communication with Arduino nodes or I2C with the Jetson Nano to control the robot's Joint States and PCL pipelines required for autonomous mapping/Localization/Tracking of the objects in real-time.

Robot URDF and Actual Model

The Robot's URDF model represents the exact simplified version of the actual model with measured offsets from joint to joint.
The RRR Robot is built using standard servo 3 x Clamps,1 x Long U Mount and 45deg U Mount.

Jetson Nano Standalone Implementation

cd catkin_ws/src
git clone https://github.com/fazildgr8/realsense_bot -b jetson

PC Arduino Implementation

cd catkin_ws/src
git clone https://github.com/fazildgr8/realsense_bot -b master

Manual Joint Control with PCL Perception

Launch the Robot in RVIZ with Manual Joint Control.
This Launch file starts the Robot state publisher which updates the Transformation tree using the Joint states from Joint state publisher. The Robot can be Controlled manually by publishing the Joint states message to the Joint State Controller.

For Jetson Nano Implementation execute the below mentioned

roslaunch realsense_bot robot_visualize_control_jetson.launch

For PC+Arduino Implementation execute the below mentioned

roslaunch realsense_bot robot_visualize_control.launch

Launch the IntelRealsense Camera node to start the PCL Perception
This Launch file starts the IntelRealsense camera node which publishes all the messages w.r.t to the Realsense Camera.

roslaunch realsense2_camera rs_camera.launch filters:=pointcloud align_depth:=true depth_width:=640 depth_height:=480 depth_fps:=30 color_width:=640 color_height:=480 color_fps:=30

Autonomous Object Tracking using YoloV3 and 2D to 3D deprojection with Pose Estimation

Launch the Yolo detector node to produce Object Detection and Bounding Boxes Messages

roslaunch yolo_detect.launch

Launch the node which deprojects the 2D detected object in 3D using rs2_pixel_to_point_deprojection(...) with the Node /yolo_pixel_to_3Dpoint

rosrun realsense_bot yolo_pixel_to_3Dpoint.py

Launch the node which follows and tracks the Object using the 3-DOF Manipulator

rosrun realsense_bot object_follow.py

3-DOF Dexterity based RTAB Mapping with Madwick IMU Filter

Launch the below mentioned seperately to begin RTAB-Mapping with Madwick IMU Filter and RVIZ representation of the Cloud Map.

roslaunch realsense_bot robot_visualize_control.launch
roslaunch realsense_bot realsense_mapping.launch

Robot Setup

• Direct Implementation - The setup for PC with Arduino control uses serial communication to control the servos through the Arduino which is capable producing PWM signals.
• Jetson Nano Standalone Implementation - The setup and installation differs for the Jeston boards for using it standalone as GPIO pins present would not be able to produce PWM signals for the servo motors instead we will be using a Servo Controller board which uses I2C protocol to control the Servos. Added to it RTAB Mapping is currently unstable with Realsense D435i in Jetson Nano due to its instability in publishing PCL data.

Setup Info (PC + Arduino Control)

ROS/Python Library Prerequisites

sudo apt-get install ros-$ROS_DISTRO-realsense2-camera
sudo apt-get install ros-$ROS_DISTRO-realsense2-description
sudo apt install ros-$ROS_DISTRO-image-transport-plugins
sudo apt install ros-$ROS_DISTRO-rtabmap-ros
sudo apt install ros-$ROS_DISTRO-rosserial
sudo apt install ros-$ROS_DISTRO-joint-state-publisher-gui
pip install pyrealsense2
pip install opencv-python
cd catkin_ws/src
git clone https://github.com/ccny-ros-pkg/imu_tools -b $ROS_DISTRO
cd ..
catkin_make

Installing darknet_ros package which is the YoloV3 implementation as a ROS Node. On Cuda set up systems make sure to modify the CMakeLists.txt within the darknet_ros package to support compute capability of your systems. For details please go through darknet_ros Official Package

cd catkin_ws/src
git clone https://github.com/ccny-ros-pkg/imu_tools -b $ROS_DISTRO
git clone --recursive git@github.com:leggedrobotics/darknet_ros.git
cd ..
catkin_make

Arduino

The Arduino code file required for flashing can be found in arduino/ros_servo_3dof.ino
Connect the three Servo's signal pin the to 9 10 11 pins of the Arduino. The pin configuration can be modified within ros_servo_3dof.ino.

Setup Info - Jetson Nano

[To Be Updated]

Jetson Nano