Skip to content

The official code implementation for SynTable - A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes

License

Notifications You must be signed in to change notification settings

ngzhili/SynTable

Repository files navigation

SynTable - A Synthetic Data Generation Pipeline for Cluttered Tabletop Scenes

This repository contains the official implementation of the paper "SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes".

Zhili Ng*, Haozhe Wang*, Zhengshen Zhang*, Francis Eng Hock Tay, Marcelo H. Ang Jr. *equal contributions

[arXiv] [Website] [Dataset] [Demo Video] [Modified UOAIS-v2]

Dataset DOI

teaser

SynTable is a robust custom data generation pipeline that creates photorealistic synthetic datasets of Cluttered Tabletop Scenes. For each scene, it includes metadata such as

  • RGB image of scene
  • depth image of Scene
  • scene instance segmentation masks
  • object amodal (visible + invisible) rgb
  • object amodal (visible + invisible) masks
  • object modal (visible) masks
  • object occlusion (invisible) masks
  • object occlusion rate
  • object visible bounding box
  • tabletop visible masks
  • background visible mask (background excludes tabletop and objects)
  • occlusion ordering adjacency matrix (OOAM) of objects on tabletop

Installation

  1. Install NVIDIA Isaac Sim 2022.1.1 version on Omniverse

  2. Change Directory to isaac_sim-2022.1.1 directory

    cd '/home/<username>/.local/share/ov/pkg/isaac_sim-2022.1.1/tools'
  3. Clone the repo

    git clone https://github.com/ngzhili/SynTable.git
  4. Install Dependencies into isaac sim's python

    • Get issac sim source code directory path in command line.
      SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
      echo $SCRIPT_DIR
    • Get isaac sim's python path
      python_exe=${PYTHONEXE:-"${SCRIPT_DIR}/kit/python/bin/python3"}
      echo $python_exe
    • Run isaac sim's python
      $python_exe
    • while running isaac sim's python in bash, install pycocotools and opencv-python into isaac sim's python
      import pip
      package_names=['pycocotools', 'opencv-python'] #packages to install
      pip.main(['install'] + package_names + ['--upgrade'])
  5. Copy the mount_dir folder to your home directory (anywhere outside of isaac sim source code)

    cp -r SynTable/mount_dir /home/<username>

Adding object models to nucleus

  1. You can download the .USD object models to be used for generating the tabletop datasets here.

  2. Upload the downloaded syntable_nucleus folder into Omniverse Nucleus into /Users directory.

  3. Ensure that the file paths in the config file are correct before running the generate dataset commands.

Generate Synthetic Dataset

Note: Before generating the synthetic dataset, please ensure that you uploaded all object models to isaac sim nucleus and their paths in the config file is correct.

  1. Change Directory to Isaac SIM source code
    cd /home/<username>/.local/share/ov/pkg/isaac_sim-2022.1.1
  2. Run Syntable Pipeline (non-headless)
    ./python.sh SynTable/syntable_composer/src/main1.py --input */parameters/train_config_syntable1.yaml --output */dataset/train --mount '/home/<username>/mount_dir' --num_scenes 3 --num_views 3 --overwrite --save_segmentation_data

Types of Flags

Flag Description
--input Path to input parameter file.
--mount Path to mount symbolized in parameter files via '*'.
--headless Will not launch Isaac SIM window.
--nap Will nap Isaac SIM after the first scene is generated.
--overwrite Overwrites dataset in output directory.
--output Output directory. Overrides 'output_dir' param.
--num-scenes Number of scenes in dataset. Overrides 'num_scenes' param.
--num-views Number of views to generate per scene. Overrides 'num_views' param.
--save-segmentation-data Saves visualisation of annotations into output directory. False by default.

Generated dataset

  • SynTable data generation pipeline generates dataset in COCO - Common Objects in Context format.

Folder Structure of Generated Synthetic Dataset

.
├── ...
├── SynTable-Sim                  # Generated dataset
│   ├── data                      # folder to store RGB, Depth, OOAM 
│   │   └── mono
│   │       ├── rgb
│   │       │   ├── 0_0.png       # file naming convention follows sceneNum_viewNum.png
│   │       │   └── 0_1.png 
│   │       ├── depth
│   │       │   ├── 0_0.png  
│   │       │   └── 0_1.png 
│   │       └── occlusion order
│   │           ├── 0_0.npy  
│   │           └── 0_1.npy 
│   ├── parameters                # parameters used for generation of annotations
│   └── train.json                # Annotation COCO.JSON
└── ...

Visualise Annotations

  1. Create python venv and install dependencies
    python3.8 -m venv env
    source env/bin/activate
    pip install -r requirements.txt
    
  2. Visualise sample annotations (creates a visualise_dataset directory in dataset directory, then saves annotation visualisations there)
    python ./visualize_annotations.py --dataset './sample_data' --ann_json './sample_data/annotation_final.json'
    

Sample Visualisation of Annotations

sample_annotations1 sample_annotations2

References

We have heavily modified the Python SDK source code from NVIDA Isaac Sim's Replicator Composer.

Citation

If you find our work useful for your research, please consider citing the following BibTeX entry:

@misc{ng2023syntable,
      title={SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes}, 
      author={Zhili Ng and Haozhe Wang and Zhengshen Zhang and Francis Tay Eng Hock and Marcelo H. Ang Jr au2},
      year={2023},
      eprint={2307.07333},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}