KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

This repo covers the data generation, training and inference of kovis visual servos introduced in KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation (IROS 2020) by En Yen Puang, Keng Peng Tee and Wei Jing.

Inference uses UR5, realsense D435 camera, ROS and urx.

[ Paper ] [ Video ]

Installation

We recommend using virtual python environment like Conda with python3.

1. Install PyTorch
2. Install other packages

pip install -r requirements.txt 

3. (Optinal) Install ROS, rospkg and RealSense camera driver
4. (Optinal) install urx for UR robot

pip install git+https://github.com/enyen/python-urx

Usage

1. Generate training data in pyBullet:

cd KOVIS_VisualServo
# example for generating dataset for pick-mug task
python oja_pick.py cfg/dataset_pick_mug.yaml

If no monitor is connected, render without shadow by replacing line 19 with

pyb.connect(pyb.DIRECT)  # pyb.connect(pyb.GUI)

2. Training servo in pyTorch:

# example for training for pick-mug task
python train_servo.py cfg/train_pick_mug.yaml

3. Running on robot:
   • launch realsense camera with both infra cameras enabled
   • turn off realsense laser using rqt_reconfigure

from inference_oja import Interface
rob = Interface()

# reach
# TODO: move tcp to close to object

# pick
rob.servo('pick_mug', 10, [0.01, 0.01, 0.01, 0.05, 0, 0], [0.1, 5])
rob.set_gripper(1)

# continue
# TODO: move object away

Citation

Please cite our paper if you use this code.

  @inproceedings{puang2020kovis,
    title={KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation},
    author={Puang, En Yen and Tee, Keng Peng and Jing, Wei},
    booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    pages={7527--7533},
    year={2020},
    organization={IEEE}
  }

License

This code is under GPL-3.0 License. Contact

puangenyen at gmail . com

to discuss other license agreement.