[CVPR 2023] PF-Track: End-to-end Vision-centric 3D MOT with Minimal ID-Switches

Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, Yu-Xiong Wang

Introduction

This is the official implementation of "Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking." Our PF-Track illustrates significant advantages in:

• Dramatically less ID-Switches: PF-Track has 90% less ID-Switches compared to previous methods. So far, PF-Track is also SOTA in ID-Switches on nuScenes.
• End-to-end perception and prediction: PF-Track emulates an end-to-end framework.
• Easy integration with detection heads: PF-Track can cooperate with various DETR-style 3D detection heads.

Please click the gif below to check our full demo and reach out to Ziqi Pang if you are interested. Our method seamlessly address occlusions and hand-over between cameras.

If you find our code or paper useful, please cite by:

@inproceedings{pang2023standing,
  title={Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking},
  author={Pang, Ziqi and Li, Jie and Tokmakov, Pavel and Chen, Dian and Zagoruyko, Sergey and Wang, Yu-Xiong},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023}
}

Getting Started

Please follow our documentation step by step. For the convenience of developers and researchers, we also add notes for developers to better convey the implementations of PF-Track and accelerate your adaptation of our framework. If you like my documentation and help, please recommend our work to your colleagues and friends.

1. Pretrained models and data files.
2. Environment Setup.
3. Preprocessing nuScenes.
4. Training.
5. Inference.

Guide for Developers and Researchers

It literally took us THREE MONTHS to implement the baseline because designing the end-to-end tracking and prediction framework is challenging. Therefore, we write the following documents to help you better understand our design choices, read the code, and adapt them to your own tasks and datasets.

1. System Overview: An ABC Guide to End-to-end MOT. (Please skim through it even if you know end-to-end MOT well, because we clarify several implementation details that are non-trivial.)
2. Visualization tools.
3. Integration with various detection heads.

Acknowledgements

We thank the contributors to the following open-source projects. Our project is impossible without the inspirations from these excellent researchers and engineers.

• 3D Detection. MMDetection3d, DETR3D, PETR.
• Multi-object tracking. MOTR, MUTR3D, SimpleTrack.
• End-to-end motion forecasting. FutureDet.

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.