Who is the best painter?

Project of the course ML701 at MBZUAI. Our main motivation was to train a model using AI to learn the style of Claude Monet.

Checking the techniques, we choose to explore state of the art and find a way to compare what had better results. We used as base two repositories:

• StyleGAN: https://github.com/dvschultz/stylegan2-ada-pytorch
• LapStyle: https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/tutorials/lap_style.md

Some of our results can be found at: Tensorboard and Results

With StyleGAN-ADA

With LapStyle

StyleGAN-Ada

Installation

Use the package manager pip to install the requirements. If using computers from the labs, load the module of cuda-11.1, to avoid problems with StyleGAN. Erase cache from nvcc if needed. Please move the folder:

colab-sg2-ada-pytorch 

to:

cd /home/{username_lab}/Documents/

so you can check if it's working by seeing the different python files from original git:

ls /home/{username_lab}/Documents/colab-sg2-ada-pytorch

which nvcc
module load cuda-11.1
which nvcc

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

pip install psutil scipy

pip install click opensimplex requests tqdm pyspng ninja imageio-ffmpeg==0.4.3

pip install opensimplex

pip install tensorboard

pip install ninja opensimplex torch==1.7.1 torchvision==0.8.2

You can do the following step (below) and put the weights in results, or download the zip from:

OneDrive_StyleGAN_ADA or if link not working GoogleDrive_StyleGAN_ADA

and put it inside (pasting the zip and unzip there).

cd /home/{username_lab}/Documents/colab-sg2-ada-pytorch

have in mind to change the {username_lab} like

cd /home/ariana.venegas/Documents/colab-sg2-ada-pytorch

Usage & Demo

Open the Jupyter Notebooks provided for each style. If you need to run without notebooks, after being in the directory ('colab-sg2-ada-pytorch/stylegan2-ada-pytorch'). Change the username. This is the configuration of how we started our 3rd experiment:

cd /home/ariana.venegas/Documents/colab-sg2-ada-pytorch/stylegan2-ada-pytorch/ 

python train.py --nkimg=0 --snap=1 --gpus=1 --cfg='24gb-gpu' --metrics=fid50k_full --outdir=./results --data='/home/ariana.venegas/Documents/colab-sg2-ada-pytorch/stylegan2-ada-pytorch/datasets/Monet_folder.zip' --resume='/home/ariana.venegas/Documents/colab-sg2-ada-pytorch/stylegan2-ada-pytorch/pretrained/wikiart.pkl' --augpipe='bg' --initstrength=0 --gamma=50 --mirror=True --mirrory=False --nkimg=0

To generate images wait til 4th kimg (13 min):

python train.py --nkimg=0 --snap=1 --gpus=1 --cfg='24gb-gpu' --metrics=fid50k_full --outdir=./results --data='/home/ariana.venegas/Documents/colab-sg2-ada-pytorch/stylegan2-ada-pytorch/datasets/Monet_folder.zip' --resume='/home/ariana.venegas/Documents/colab-sg2-ada-pytorch/stylegan2-ada-pytorch/results/ID03-network-snapshot-002496.pkl' --augpipe='bg' --initstrength=0 --gamma=50 --mirror=True --mirrory=False --nkimg=0

Check results folder for the image.

Tracking with Tensorboard

Go to the following path:

cd /home/{where you clone the git}/

And run the following command to see the dashboard:

tensorboard --logdir ./

License

Copyright © 2021, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License.

Attribution to Derrick Schultz.

LapStyle

Installation

Use the package manager pip to install the requirements. If using computers from the labs, load the module of cuda-11.2, to avoid problems with LapStyle. Erase cache from nvcc if needed.

which nvcc
module load cuda-11.2
which nvcc

For smoother installation create a conda environment

conda create -n LapEnv
conda activate LapEnv

Install Jupyter Notebook if not already installed

conda install jupyter notebook

Install PaddlePaddle Framework

pip install paddlepaddle-gpu==2.2.0.post112 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html

Set correct path to avoid any errors, replace {username_lab} with your username

export LD_LIBRARY_PATH='/home/{username_lab}/.conda/envs/LapEnv/lib'

Test if Paddle Paddle was installed successfully (It should take less than 1 minute to run), run a python instance

python

Inside the python instance run the following instructions

import paddle
paddle.utils.run_check()

A modified version of the PaddleGAN is already included in this repo

Just, ensure you are inside the "PaddleGAN" folder if not move

cd PaddleGAN

Install the PaddleGAN requirements,

pip install -v -e . 
pip install -r requirements.txt

If the previous commands did not work, run this one

python setup.py develop

The dataset is already included in the repo, go back to "NewLapStyle" folder and into the "dataset" folder.

cd ..

The original dataset can be found here. The dataset split is 70/30.

Unzip both files and three folders should appear

unzip “*.zip”

Go back to PaddleGAN directory

cd ../PaddleGAN

To train the model from scratch run the following commands: (Skip this section to run a pretrained model) Replace {{Previously Generated Folder by Draft}}. (1)Train the Draft Network of LapStyle under 128*128 resolution:

python -u tools/main.py --config-file configs/lapstyle_draft.yaml

(2) Train the Revision Network of LapStyle under 256*256 resolution:

python -u tools/main.py --config-file configs/lapstyle_rev_first.yaml --load 'output_dir/{{Previously Generated Folder by Draft}}/iter_30000_checkpoint.pdparams'

(3) You can train the second Revision Network under 512*512 resolution:

python -u tools/main.py --config-file configs/lapstyle_rev_second.yaml --load 'output_dir/{{Previously Generated Folder by First Rev}}/iter_30000_checkpoint.pdparams'

To change the style image go to the configs folder and change the name of the "style_root" property for train and test sections in the 3 config files.

To run pretrained model download the weights from: One Drive Link and put it inside (paste the zip and unzip there). There should be 3 folders with 1 pdparams file each

cd NewLapStyle/PaddleGAN/output_dir

Set up WandB monitoring

Create a WandB account https://wandb.ai/ Follow the terminal instructions

pip install wandb
wandb login

Now you are ready to run the model!!

Usage

Change the parameter validate/save_img in the configuration file to true to save the output image. To test the trained model, you can directly test the "lapstyle_rev_second", since it also contains the trained weight of previous stages:

python tools/main.py --config-file configs/lapstyle_rev_second.yaml --evaluate-only --load 'output_dir/LongerVarSecond iter_30000_checkpoint.pdparams'

The image will be outputed in the following folder

cd NewLapStyle/PaddleGAN/output_dir

Demo

python applications/tools/lapstyle.py --content_img_path '../dataset/photo_jpg_train/46e84039a1.jpg' --style_image_path '../dataset/monet_jpg/82991e742a.jpg'

You can replace the content and style image:

python applications/tools/lapstyle.py --content_img_path ${PATH_OF_CONTENT_IMG} --style_image_path ${PATH_OF_STYLE_IMG}

Dependencies

Below you can find a list of dependencies:

• PaddlePaddle
• PaddleGAN
• tqdm
• PyYAML>=5.1
• scikit-image>=0.14.0
• scipy>=1.1.0
• opencv-python
• imageio==2.9.0
• imageio-ffmpeg
• librosa
• numba==0.53.1
• easydict
• munch
• natsort
• cudatoolkit

Important files

Lapstyle_model.py

This is the model file of LapStyle, the loss functions are selected here, as well, the generators, how the forward pass and backpropagation is performed. This file was modified to be able to calculate the FID while the model is running. The functions test_iter were added to test the model for the selected metrics (currently only running FID) and to evaluate the networks. One new function per network.

Found in the following folder: NewLapStyle/PaddleGAN/ppgan/models/lapstyle_model.py

lapstyle_predictor.py

Handles how the model loads the pretrained weights and saving the images in the output file. Last but not least, it handles all the laplacian pyramid functions.

Found in the following folder: NewLapStyle/PaddleGAN/ppgan/apps/lapstyle_predictor.py

Trainer.py

The most important file, where the following things happen:

• Training loop is executed
• Metrics are measured
• Log is printed
• WandB monitoring was added
• FID calculation for LapStyle was added

Found in the following folder: NewLapStyle/PaddleGAN/ppgan/engine/trainer.py

Fid.py

Using pretrained InceptionV3 weights trained on ImageNet, this file calculates de FID between the style image and the stylized image. File was modified to work with LapStyle, as it was originally done for cyclegan.

Found in the following folder: NewLapStyle/PaddleGAN/ppgan/metrics/fid.py

Config Files

In these files the iterations, dataset location, batch size, optimizer images and weights for content and style are selected.

Found in the following folder: NewLapStyle/PaddleGAN/configs

License

Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved

PaddleGAN is released under the Apache 2.0 license.