Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps

This is the official PyTorch implementation of the publication:

A. Simoni, S. Pini, G. Borghi, R. Vezzani
Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps
In IEEE Robotics and Automation Letters (RA-L), 2022

Knowing the exact 3D location of workers and robots in a collaborative environment enables several real applications, such as the detection of unsafe situations or the study of mutual interactions for statistical and social purposes. In this paper, we propose a non-invasive and light-invariant framework based on depth devices and deep neural networks to estimate the 3D pose of robots from an external camera. The method can be applied to any robot without requiring hardware access to the internal states. We introduce a novel representation of the predicted pose, namely Semi-Perspective Decoupled Heatmaps (SPDH), to accurately compute 3D joint locations in world coordinates adapting efficient deep networks designed for the 2D Human Pose Estimation. The proposed approach, which takes as input a depth representation based on XYZ coordinates, can be trained on synthetic depth data and applied to real-world settings without the need for domain adaptation techniques. To this end, we present the SimBa dataset, based on both synthetic and real depth images, and use it for the experimental evaluation. Results show that the proposed approach, made of a specific depth map representation and the SPDH, overcomes the current state of the art.

Getting Started

We tested the approach with Python 3.6.13, PyTorch 1.8.1, TorchVision 0.9.1 and CudaToolkit 10.2.

Environment

Create your Python environment and install the required packages with the following commands:

conda create -n <env_name>
conda activate <env_name>
pip install -r requirements.txt

Install PyTorch>=1.8.1 and TorchVision>=0.9.1.

SimBa dataset

For downloading the dataset, please use the links on the dataset webpage.

SPDH

Training

Run the following script:

python train_spdh.py --config <path-to-config> --results_dir <path-to-save-log-and-weights>

Testing with pre-trained weights

Original paper experiments can be reproduced as follows:

1. Download config and pre-trained weights from Model Zoo.
2. Run the testing script as follows:

python test_spdh.py --config <path-to-config> --resume <path-to-weights>

Authors

• Alessandro Simoni - alexj94
• Stefano Pini - stefanopini
• Guido Borghi - gdubrg
• Roberto Vezzani - robervez

Citation

If you find this repository useful for your research, please cite the following paper:

@article{9837426,
  author={Simoni, Alessandro and Pini, Stefano and Borghi, Guido and Vezzani, Roberto},
  journal={IEEE Robotics and Automation Letters}, 
  title={Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation From Depth Maps}, 
  year={2022},
  volume={7},
  number={4},
  pages={11569-11576},
  doi={10.1109/LRA.2022.3193225}
}

License

This project is licensed under the MIT License - see the LICENSE file for detail