PointPainting-Semantic-Segmentation

My Pytorch implementation of PointPainting Paper for realtime pointcloud semantic segmentation painting (labeling each point with a class) based on semantic segmentation maps using BiSeNetv2

Project

• BiSeNetv2 model trained on KITTI dataset
• Implementation of the pointpainting fusion algorithm

Update, 1 June, 2022

• Added Python ONNX Runtime inference for bisenet @ bisenet_onnx.py
• Added Python TensorRT inference for bisenet @ tensorrt_bisenet.py
• Added C++ TensorRT & CUDA inference @ tensorrt_cpp

Demo Video

Download the checkpoint

Download from Drive Place it in "BiSeNetv2/checkpoints"
Important Note The file you will download will have the name "BiseNetv2_150.pth.tar", don't unzip it .. just rename it to be "BiseNetv2_150.pth"

Run Demo

python3 demo.py --image_path PATH_TO_IMAGE --pointcloud_path PATH_TO_POINTCLOUD --calib_path PATH_TO_CALIB --weights_path PATH_TO_MODEL_WEIGHTS

# note that default arguments are set to one of uploaded kitti samples so you can run it as
python3 demo.py

# add --mode 3d to see a 3d visualization of painted pointcloud
python3 demo.py --mode 3d

Run Demo on Kitti Videos

Kitti Provides sequential videos for testing, Download them from Kitti Videos by downloading video data(left & pointclouds)(download from [synced+rectified data]) and calibs (download from [calibration]) in the selected video

# PATH_TO_VIDEO is path contains 'image_02' & 'velodyne_points' together
# PATH_TO_CALIB is path contains calib files ['calib_cam_to_cam', '', '']
# mode 2d to visualize image+bev .. 3d to visualize 3d painted pointcloud
python3 demo_video.py --video_path PATH_TO_VIDEO --calib_path PATH_TO_CALIB --mode 3d

BiSeNetv2

Realtime semantic segmentation on images

Thanks to https://github.com/CoinCheung/BiSeNet for the implementation trained on CityScapes datasets. I used it and finetuned it on KITTI dataset using Pytorch

Training on KITTI dataset

cd BiSeNetv2
python3 train.py

I trained it on Colab and provided the notebook

Test on KITTI Semantic

cd BiSeNetv2
python3 test.py

KITTI Dataset

Semantic KITTI dataset contains 200 images for training & 200 for testing
Download it from KITTI website

# visualize dataset on tensorboard
python3 visualization.py --tensorboard

# PATH_TO_TENSORBOARD_FOLDER is path "BiSeNetv2/checkpoints/tensorboard/"
tensorboard --logdir PATH_TO_TENSORBOARD_FOLDER

Folder structure

├── BiSeNetv2
    ├── checkpoints
		├── BiseNetv2_150.pth 	# path to model
	    ├── tensorboard 		# path to save tensorboard events
    ├── data 					# path to kitti semantic dataset
        ├── KITTI
          ├── testing
              ├── image_2
          ├── training
            ├── image_2
            ├── instance
            ├── semantic
            ├── semantic_rgb

    ├── utils
      ├── label.py 				# label information (colors/ids/names)
      ├── utils.py 				# utils functions
    ├── train.py
    ├── test.py

├── Kitti_sample				# 2 images & pointclouds & calib for testing (by demo.py)
├── KittiCalibration.py 		# Stores Calibration file matrices
├── KittiVideo.py 				# Kitti Video Reader
├── bev_utils.py 				# BEV algorithms
├── demo.py 					# demo to test on 1 sample (Kitti_sample)
├── demo_video.py 				# demo to test on Kitti Videos
├── pointpainting.py 			# implementation of PointPainting
├── visualizer.py 				# visualizer using opend3d & opencv

References

• My Review https://docs.google.com/document/d/1AtpbLfCl_uL5BpwlYDgpdM_2WtCIucDrRomFjSp6bhg/edit?usp=sharing

• Semantic Seg Overview :https://medium.com/beyondminds/a-simple-guide-to-semantic-segmentation-effcf83e7e54 .. https://medium.com/swlh/understanding-multi-scale-representation-learning-architectures-in-convolutional-neural-networks-a71497d1e07c

• Conv Types (Atrous, Transposed) https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d

• DeepLabv3 https://towardsdatascience.com/review-deeplabv3-atrous-convolution-semantic-segmentation-6d818bfd1d74

• Survay: Image Segmentation using Deep Learning https://arxiv.org/pdf/2001.05566.pdf

• Survay Article https://medium.com/swlh/image-segmentation-using-deep-learning-a-survey-e37e0f0a1489

• BiseNet (understanding spatial & context information) (good) https://prince-canuma.medium.com/spatial-path-context-path-3f03ed0c0cf5 .. https://medium.datadriveninvestor.com/bisenet-for-real-time-segmentation-part-i-bf8c04afc448

• Receptive field https://blog.christianperone.com/2017/11/the-effective-receptive-field-on-cnns/