pytorch-deepFEPE

This repo is the official implementation of the paper:

Deep Keypoint-Based Camera Pose Estimation with Geometric Constraints.

You-Yi Jau*, Rui Zhu*, Hao Su, Manmohan Chandraker (*equal contribution)

IROS 2020

See our arxiv, paper and video for more details.

Installation

Clone repo and submodules

# please clone without git lfs and download from google drive if possible
# "This repository is over its data quota. Purchase more data packs to restore access."
export GIT_LFS_SKIP_SMUDGE=1
git clone https://github.com/eric-yyjau/pytorch-deepFEPE.git
git pull --recurse-submodules

Requirements

python == 3.6
pytorch >= 1.1 (tested in 1.3.1)
torchvision >= 0.3.0 (tested in 0.4.2)
cuda (tested in cuda10)

conda create --name py36-deepfepe python=3.6
conda activate py36-deepfepe
pip install -r requirements.txt
pip install -r requirements_torch.txt # install pytorch

Pull and link Superpoint model

install superpoint module

git clone https://github.com/eric-yyjau/pytorch-superpoint.git
cd pytorch-superpoint
git checkout module_20200707
# install
pip install --upgrade setuptools wheel
python setup.py bdist_wheel
pip install -e .

if ascii error

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

Dataset

Data preprocessing

cd deepFEPE_data

Follow the instructions in the README.
Process KITTI dataset for training or testing.
Process ApolloScape dataset for testing.

Config file

Key items

dump_root: '[your dumped dataset]' # apollo or kitti
if_qt_loss: true # use pose-loss and F-loss (true) or only F-loss (false)
if_SP: true (use superpoint instead of SIFT)/ false (SIFT)
# deepF
retrain: true (new deepF model)/ false (use pretrained deepF)
train: true (train the model)/ false (freeze weights)
pretrained: [Path to pretrained deepF]
# superpoint
retrain_SP: true (new superpoint model)/ false (use pretrained_SP)
train_SP: true (train SP model)/ false (freeze SP weights)
pretrained_SP: [Path to pretrained SP]

Run the code - Training

Prepare the dataset. Use training commands following steps 1 to 3 (skip step 0). Visualize training and validation with Tensorboard.

Training commands

KITTI

python deepFEPE/train_good.py train_good deepFEPE/configs/kitti_corr_baseline.yaml test_kitti --eval

ApolloScape

python deepFEPE/train_good.py train_good deepFEPE/configs/apollo_train_corr_baseline.yaml test_apo --eval

0) Baseline: Train deepF model with SIFT features

config file: deepFEPE/configs/kitti_corr_baseline.yaml
config:
- set dump_root to [your dataset path]
- set if_SP to False
run training script

1) Prepare SuperPoint model

Follow the instruction in pytorch-superpoint.
Or use the pretrained models.
git lfs file: deepFEPE/logs/superpoint_models.zip
- KITTI: deepFEPE/logs/superpoint_kitti_heat2_0/checkpoints/superPointNet_50000_checkpoint.pth.tar
- Apollo: deepFEPE/logs/superpoint_apollo_v1/checkpoints/superPointNet_40000_checkpoint.pth.tar

2) Train deepF model with SuperPoint pretrained models

config:
- Set if_SP to True.
- Add pretrained_SP path.

3) End-to-end training

config:
- Set train to True
- Set train_SP to True
- Set the pretrained paths in pretrained and pretrained_SP.
- Set if_qt_loss: true for pose-loss.

Run the code - Testing

KITTI: use kitti_corr_baselineEval.yaml

python deepFEPE/train_good.py eval_good deepFEPE/configs/kitti_corr_baselineEval.yaml eval_kitti --test --eval # use testing set (seq 09, 10)

ApolloScape: use apollo_train_corr_baselineEval.yaml

python deepFEPE/train_good.py eval_good deepFEPE/configs/apollo_train_corr_baselineEval.yaml eval_apo --test --eval

You can use our pretrained models for testing. Just put the paths in the config file.

Our pretrained models:

Download through git-lfs or Google drive

git lfs ls-files # check the files
git lfs fetch 
git lfs pull # get the files
git pull

(Refer to the config.yml and checkpoints/ in the folder)

KITTI models (trained on KITTI):
- git lfs file: deepFEPE/logs/kitti_models.zip
- SIFT Baselines:
  - baselineTrain_deepF_kitti_fLoss_v1
  - baselineTrain_sift_deepF_kittiPoseLoss_v1
  - baselineTrain_sift_deepF_poseLoss_v0
- SuperPoint baselines:
  - baselineTrain_kittiSp_deepF_kittiFLoss_v0
  - baselineTrain_kittiSp_deepF_kittiPoseLoss_v1
- DeepFEPE:
  - baselineTrain_kittiSp_deepF_end_kittiFLoss_freezeSp_v1
  - baselineTrain_kittiSp_deepF_end_kittiFLossPoseLoss_v1_freezeSp
  - baselineTrain_kittiSp_kittiDeepF_end_kittiPoseLoss_v0
Apollo models (trained on Apollo, under apollo/):
- git lfs file: deepFEPE/logs/apollo_models.zip
- SIFT baselines:
  - baselineTrain_sift_deepF_fLoss_apolloseq2_v1
  - baselineTrain_sift_deepF_poseLoss_apolloseq2_v0
  - baselineTrain_sift_deepF_apolloFLossPoseLoss_v0
- SuperPoint baselines:
  - baselineTrain_apolloSp_deepF_fLoss_apolloseq2_v0
  - baselineTrain_apolloSp_deepF_poseLoss_apolloseq2_v0
- DeepFEPE:
  - baselineTrain_apolloSp_deepF_fLoss_apolloseq2_end_v0_freezeSp_fLoss
  - baselineTrain_apolloSp_deepF_poseLoss_apolloseq2_end_v0
  - baselineTrain_apolloSp_deepF_fLossPoseLoss_apolloseq2_end_v0_freezeSp_fLoss

Run the code - batch testing and evaluation

Run the evaluation

cd deepFEPE/
python run_eval_good.py --help

update your dataset path in configs/kitti_corr_baselineEval.yaml and configs/apollo_train_corr_baselineEval.yaml
set the model names in deepFEPE/run_eval_good.py

def get_sequences(...):
    kitti_ablation = { ... }
    apollo_ablation = { ... }

check if the models exist

# kitti models on kitti dataset
python run_eval_good.py --dataset kitti --model_base kitti --exper_path logs --check_exist
# kitti models on apollo dataset
python run_eval_good.py --dataset apollo --model_base kitti --exper_path logs --check_exist

# apollo models
python run_eval_good.py --dataset kitti --model_base apollo --exper_path logs/apollo --check_exist

run the evaluation (dataset should be ready)

python run_eval_good.py -dataset kitti --model_base kitti --exper_path logs --runEval

Read out the results

open jupyter notebook
read the sequences from the config file: table_trans_rot_kitti_apollo.yaml

base_path: '/home/yoyee/Documents/deepSfm/logs/' # set the base path for checkpoints
seq_dict_test:
    Si-D.k: ['eval_kitti', 'DeepF_err_ratio.npz', '07/29/2020']

print out the numbers based on the setting in the config file

jupyter notebook
# navigate to `notebooks/exp_process_table.ipynb`

Download the results

git lfs ls-files # check the files
git lfs fetch 
git lfs pull # get the files

unzip

cd deepFEPE/logs/results/ 
unzip 1107.zip 
unzip 1114.zip 
unzip new_1119.zip

put to the same folder

ln -s 1114/* . # do the same for the other folders

Then, you can follow the instructions in Read out the results.
Trajectory results are in deepFEPE/logs/results/trajectory.zip.

Evaluate visual odometry

jupyter notebook
# navigate to `notebooks/exp_process_table.ipynb`

convert the relative_pose_body to absoluted poses
export the poses to two files for sequence 09 and 10.
use kitti-odom-eval for evaluation.

Tensorboard

tensorboard --logdir=runs/train_good

visualization

Ex: Train SIFT + DeepF

check loss: gt should always be 1
check R, t inlier ratio:

rotation: q, angle: 0.01 to 180
translation: t, angle: 0.01 to 180
# (higher the better)
# pick the checkpoints through inlier ration

Citations

Please cite the following papers.

DeepFEPE

@INPROCEEDINGS{2020_jau_zhu_deepFEPE,
  author={Y. -Y. {Jau} and R. {Zhu} and H. {Su} and M. {Chandraker}},
  booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  title={Deep Keypoint-Based Camera Pose Estimation with Geometric Constraints}, 
  year={2020},
  volume={},
  number={},
  pages={4950-4957},
  doi={10.1109/IROS45743.2020.9341229}}

SuperPoint

@inproceedings{detone2018superpoint,
  title={Superpoint: Self-supervised interest point detection and description},
  author={DeTone, Daniel and Malisiewicz, Tomasz and Rabinovich, Andrew},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops},
  pages={224--236},
  year={2018}
}

DeepF

@inproceedings{ranftl2018deep,
  title={Deep fundamental matrix estimation},
  author={Ranftl, Ren{\'e} and Koltun, Vladlen},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  pages={284--299},
  year={2018}
}

Credits

This implementation is developed by You-Yi Jau and Rui Zhu. Please contact You-Yi for any problems.

License

DeepFEPE is released under the MIT License.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

pytorch-deepFEPE

Installation

Clone repo and submodules

Requirements

Pull and link Superpoint model

Dataset

Config file

Run the code - Training

Training commands

0) Baseline: Train deepF model with SIFT features

1) Prepare SuperPoint model

2) Train deepF model with SuperPoint pretrained models

3) End-to-end training

Run the code - Testing

Our pretrained models:

Run the code - batch testing and evaluation

Run the evaluation

Read out the results

Download the results

Evaluate visual odometry

Tensorboard

Citations

Credits

License

Files

README.md

Latest commit

History

README.md

File metadata and controls

pytorch-deepFEPE

Installation

Clone repo and submodules

Requirements

Pull and link Superpoint model

Dataset

Config file

Run the code - Training

Training commands

0) Baseline: Train deepF model with SIFT features

1) Prepare SuperPoint model

2) Train deepF model with SuperPoint pretrained models

3) End-to-end training

Run the code - Testing

Our pretrained models:

Run the code - batch testing and evaluation

Run the evaluation

Read out the results

Download the results

Evaluate visual odometry

Tensorboard

Citations

Credits

License