Fork specific Overview:

This fork of the original work is intended as a repository for Team 4's project in the class KI in der Produktion.

Instructions for starting:

1. Download and extract your chosen dataset to the folder data/

2. Build the dockerfile using the following command:

     Option 1

   $ /build_docker.sh

     Option 2 (windows -- works (requirements: Docker Desktop installed))

   $ docker compose build

3. Start the container from the root directory of this repository:

     Option 1

   $ ./start_docker.sh

     Option 2

   $ docker compose up

   (better start the container on windows from the Docker Desktop GUI)

4. (Optional) If using the cornell dataset and depth images haven't been created from pcd:

$ python -m utils.dataset_processing.generate_cornell_depth data/

5. To train the model:

$ python train_network.py --dataset <cornell or jacquard> --dataset-path data/ --description training_cornell

Training Commands

GRConvNet with RFB, 1 ResBlock

train the modifies model with rfb block and one residual block in the embedding/ bottleneck like in paper "lightweight cnn with gaussian based grasping representation for robotic grasping detection"

Cornell:

$ python train_network.py --dataset cornell --dataset-path utils/data/cornell --network grconvnet3_rfb --epochs 10 --description training_cornell_rfb

Graspnet:

python train_network.py --dataset graspnet --dataset-path data/ --network grconvnet3_rfb --epochs 10 --description training_cornell_rfb

GRConvNet with RFB, MDAF, 1 ResBlock (without concat of deep/ shallow features)

train the modified model with rfb block, multi dimensional fusion and one residual block in the embedding/ bottleneck like in paper "lightweight cnn with gaussian based grasping representation for robotic grasping detection"

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network grconvnet3_1rfb_1mdaf_1residual --epochs 10 --description training_cornell_1rfb_1mdfa_1residual

GRConvNet with RFB, 3 ResBlocks, 2 MDAF

train the modified model with rfb block, 3 RES-Blocks, 2 multi dimensional fusion and concatenation of shallow and deep features in upsampling (part of "lightweight cnn with gaussian based grasping representation for robotic grasping detection")

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network grconvnet3_rfb_mdaf_multi_lightweight --epochs 10 --description training_cornell_rfb_mdfa_multi_lightweight

Lightweight Model

train the lightweight model with ResBlocks, RFB, MDAF, MaxPool ("lightweight cnn with gaussian based grasping representation for robotic grasping detection")

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network lightweight --epochs 50 --description training_cornell_lightweight

Lightweight Model without RFB

train the lightweight model with ResBlocks, MDAF, MaxPool but without RFB ("lightweight cnn with gaussian based grasping representation for robotic grasping detection")

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network lightweight_without_rfb --epochs 50 --description lightweight_basis_without_rfb

Lightweight Model without MaxPooling

train the lightweight model with ResBlocks, 2x MDAF, MaxPool but without RFB ("lightweight cnn with gaussian based grasping representation for robotic grasping detection")

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network lightweight_without_maxpooling --epochs 50 --description lightweight_basis_without_maxpooling

Basic GRConvNet with 1 ResBlock

Train the Basic GRConvNet Model but with only 1 Residual Block to check the impact of the Residual Blocks

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network grconvnet2_1resblock --epochs 50 --description training_GRConvNet_Basic_with_only_1_ResBlock

Basic GRConvNet Model with 2x MaxPooling

Train the Basic GRConvNet Model with 2x MaxPooling during Downsampling to check the impact of MaxPooling

python train_network.py --dataset cornell --dataset-path utils/data/cornell --network grconvnet2_maxpooling --epochs 50 --description training_GRConvNet_Basic_with_2_MaxPooling_during_Downsampling

1. View logs with tensorboard

tensorboard --logdir=logs

Then open the following url in your browser: http://localhost:6006/

1. Visualize Training and Evaluation pictures: set the following flag at the end of your command line when calling the train_network.py or the evaluation.py!

$ --vis

---

Antipodal Robotic Grasping

We present a novel generative residual convolutional neural network based model architecture which detects objects in the camera’s field of view and predicts a suitable antipodal grasp configuration for the objects in the image.

This repository contains the implementation of the Generative Residual Convolutional Neural Network (GR-ConvNet) from the paper:

Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network

Sulabh Kumra, Shirin Joshi, Ferat Sahin

arxiv | video

If you use this project in your research or wish to refer to the baseline results published in the paper, please use the following BibTeX entry:

@inproceedings{kumra2020antipodal,
  author={Kumra, Sulabh and Joshi, Shirin and Sahin, Ferat},
  booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  title={Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network}, 
  year={2020},
  pages={9626-9633},
  doi={10.1109/IROS45743.2020.9340777}}

Requirements

• numpy
• opencv-python
• matplotlib
• scikit-image
• imageio
• torch
• torchvision
• torchsummary
• tensorboardX
• pyrealsense2
• Pillow

Installation

• Checkout the robotic grasping package

$ git clone https://github.com/skumra/robotic-grasping.git

• Create a virtual environment

$ python3.6 -m venv --system-site-packages venv

• Source the virtual environment

$ source venv/bin/activate

• Install the requirements

$ cd robotic-grasping
$ pip install -r requirements.txt

Datasets

This repository supports both the Cornell Grasping Dataset and Jacquard Dataset.

Cornell Grasping Dataset

1. Download the and extract Cornell Grasping Dataset.
2. Convert the PCD files to depth images by running python -m utils.dataset_processing.generate_cornell_depth <Path To Dataset>

Jacquard Dataset

1. Download and extract the Jacquard Dataset.

Model Training

A model can be trained using the train_network.py script. Run train_network.py --help to see a full list of options.

Example for Cornell dataset:

python train_network.py --dataset cornell --dataset-path <Path To Dataset> --description training_cornell

Example for Jacquard dataset:

python train_network.py --dataset jacquard --dataset-path <Path To Dataset> --description training_jacquard --use-dropout 0 --input-size 300

Model Evaluation

The trained network can be evaluated using the evaluate.py script. Run evaluate.py --help for a full set of options.

Example for Cornell dataset:

python evaluate.py --network <Path to Trained Network> --dataset cornell --dataset-path <Path to Dataset> --iou-eval

Example for Jacquard dataset:

python evaluate.py --network <Path to Trained Network> --dataset jacquard --dataset-path <Path to Dataset> --iou-eval --use-dropout 0 --input-size 300

Run Tasks

A task can be executed using the relevant run script. All task scripts are named as run_<task name>.py. For example, to run the grasp generator run:

python run_grasp_generator.py

Run on a Robot

To run the grasp generator with a robot, please use our ROS implementation for Baxter robot. It is available at: https://github.com/skumra/baxter-pnp