Image synthesis via semantic synthesis [Project Page]

by Yi Wang, Lu Qi, Ying-Cong Chen, Xiangyu Zhang, Jiaya Jia.

Introduction

This repository gives the implementation of our semantic image synthesis method in ICCV 2021 paper, 'Image synthesis via semantic synthesis'.

Our framework

Usage

git clone https://github.com/dvlab-research/SCGAN.git
cd SCGAN/code

To use this code, please install PyTorch 1.0 and Python 3+. Other dependencies can be installed by

pip install -r requirements.txt

Dataset Preparation

Please refer to SPADE for detailed execution.

Testing

1. Downloading pretrained models, then putting the folder containing model weights in the folder ./checkpoints.

2. Producing images with the pretrained models.

python test.py --gpu_ids 0,1,2,3 --dataset_mode [dataset] --config config/scgan_[dataset]_test.yml --fid --gt [gt_path] --visual_n 1

For example,

python test.py --gpu_ids 0,1,2,3 --dataset_mode celeba --config config/scgan_celeba-test.yml --fid --gt /data/datasets/celeba --visual_n 1

3. Visual results are stored at ./results/scgan_[dataset]/ by default.

Pretrained Models (to be updated)

Dataset	Download link
CelebAMask-HQ	Baidu Disk (Code: face) or OneDrive
ADE20K	Baidu Disk (Code: n021) or OneDrive| Visual results (Code: wu7b) or OneDrive
COCO	Baidu Disk (Code: ss4b) or OneDrive| Visual results (Code: i4dw) or OneDrive

Training

Using train.sh to train new models. Or you can specify training options in config/[config_file].yml.

Key operators

Our proposed dynamic computation units (spatial conditional convolution and normalization) are extended from conditionally parameterized convolutions [1]. We generalize the scalar condition into a spatial one and also apply these techniques to normalization.

Citation

If our research is useful for you, please consider citing:

@inproceedings{wang2021image,
  title={Image Synthesis via Semantic Composition},
  author={Wang, Yi and Qi, Lu and Chen, Ying-Cong and Zhang, Xiangyu and Jia, Jiaya},
  booktitle={ICCV},
  year={2021}
}

Acknowledgements

This code is built upon SPADE, Imaginaire, and PyTorch-FID.

Reference

[1] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. In NeurIPS. 2019.

Contact

Please send email to yiwang@cse.cuhk.edu.hk.