Skip to content

Implements VAR+CLIP for text-to-image (T2I) generation

Notifications You must be signed in to change notification settings

daixiangzi/VAR-CLIP

Repository files navigation

VAR-CLIP:
Text-to-Image Generator with Visual Auto-Regressive Modeling

arXiv 

VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling
Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, Xingyu Ren
Institute of Applied Physics and Computational Mathematics, DeepGlint,Shanghai Jiao Tong University

Some example for text-conditional generation:

.

Some example for class-conditional generation:

.

TODO

  • Relased Pre_train model.
  • Relased train code.
  • Relased Arxiv.
  • Training T2I on the ImageNet dataset has been completed.
  • Training on the ImageNet dataset has been completed.

Getting Started

Requirements

pip install -r requirements.txt

Download Pretrain model/Dataset

1. Place the downloaded ImageNet train/val parts separately under train/val in the directory ./imagenet/
2. Download clip/vae pretrain model put on pretrained/

Download ClIP_L14
Download VAE

Training Scripts

# training VAR-CLIP-d16 for 1000 epochs on ImageNet 256x256 costs 4.1 days on 64 A100s
# Before running, you need to configure the IP addresses of multiple machines in the run.py file and data_path
python run.py

demo Scripts

# you can run demo_samle.ipynb get text-conditional generation resulets after train completed.
demo_sample.ipynb

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citations

@misc{zhang2024varclip,
      title={VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling}, 
      author={Qian Zhang and Xiangzi Dai and Ninghua Yang and Xiang An and Ziyong Feng and Xingyu Ren},
      year={2024},
      journal={arXiv:2408.01181},
}

About

Implements VAR+CLIP for text-to-image (T2I) generation

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published