GEM: Glass-Segmentor

Jing Hao, Moyun Liu, Jinrong Yang, Kuo Feng Hung.

This repository is the official implementation of the GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models. Our code is based on MaskDINO.

Todo list

• Release code and GEM-Tiny checkpoint

• Release S-GSD dataset

• Release multiple pre-trained models

• Release GEM-Base model

---

Features

• A simple but accurate segmentation framework, named GEM, for glass surface segmentation.
• Automatically construct a large-scale synthesized glass surface dataset with precise mask annotation, termed S-GSD.
• Surpasses the previous state-of-the-art methods by a large margin (IoU +2.1%).

Installation

See Mask DINO.

Getting Started

See Inference Demo with Pre-trained Model

See Results.

See Preparing Datasets for GEM.

See Getting Started.

---

Results

Model	Pre-trained dataset	IoU	F_β	MAE	BER	FPS	download
GEM-Tiny | config	S-GSD-1x	0.755	0.852	0.038	8.39	16.09	BaiduDisk
GEM-Tiny | config	S-GSD-5x	0.757	0.855	0.035	8.54	16.09	BaiduDisk
GEM-Tiny | config	S-GSD-10x	0.764	0.866	0.034	8.62	16.09	BaiduDisk
GEM-Tiny | config	S-GSD-20x	0.770	0.865	0.032	8.21	16.09	BaiduDisk
GEM-Base | config	S-GSD-1x	0.766	0.873	0.031	9.44	11.55	BaiduDisk
GEM-Base | config	S-GSD-5x	0.769	0.858	0.032	8.16	11.55	BaiduDisk
GEM-Base | config	S-GSD-10x	0.774	0.868	0.032	8.56	11.55	BaiduDisk
GEM-Base | config	S-GSD-20x	0.774	0.865	0.029	8.35	11.55	BaiduDisk

[07/15/2024] All pre-trained models can also download from Huggingface

[07/18/2024] S-GSD-1x dataset can also download from Huggingface

The download link of S-GSD-5x is blow here, it be divided into three parts:

Part1: https://pan.baidu.com/s/1CL0x8s1LXdIIoOjcr5wirw?pwd=2ff4

Part2: https://pan.baidu.com/s/18uCDKFmzy7vSmWm1IF4JFQ?pwd=9jqb

Part3: https://pan.baidu.com/s/1Z8pePl9Ps3QtrZB7WQ7aoQ?pwd=8fku

The S-GSD-10x and S-GSD-20x are not released because of the large disk storage, if you want to get these two large-scale datasets, feel free to contact me via isjinghao@gmail.com.

---

---

Getting Started

In the above tables, the corresponding model checkpoints can pre-trained models can be downloaded from the link.

If your dataset files are not under this repo, you need to add export DETECTRON2_DATASETS=/path/to/your/data or use Symbolic Link ln -s to link the dataset into this repo before the following command first.

Evalaluate our pretrained models

• You can download our pretrained models and evaluate them with the following commands.

  python train_net.py --eval-only --num-gpus 8 --config-file config_path MODEL.WEIGHTS /path/to/checkpoint_file

Train GEM to reproduce results

• Use the above command without eval-only will train the model. For MobileSAM/SAM backbones, you need to download its weight from () and specify the path of the pretrained backbones with MODEL.WEIGHTS /path/to/pretrained_checkpoint

  python train_net.py --num-gpus 8 --config-file config_path MODEL.WEIGHTS /path/to/checkpoint_file

You can also refer to Getting Started with Detectron2 for full usage.

Citing GEM

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@article{hao2024gem,
  title={GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis},
  author={Hao, Jing and Liu, Moyun and Hung, Kuo Feng},
  journal={arXiv preprint arXiv:2401.15282},
  year={2024}
}

If you find the code useful, please also consider the following BibTeX entry.

@misc{li2022mask,
      title={Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation}, 
      author={Feng Li and Hao Zhang and Huaizhe xu and Shilong Liu and Lei Zhang and Lionel M. Ni and Heung-Yeung Shum},
      year={2022},
      eprint={2206.02777},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

Many thanks to these excellent opensource projects

• Mask DINO
• Segment-Anything