Intelligent-RS

   

This is the model repository of YNU's deep learning principles and platform course assignments, which mainly use remote sensing image datasets to achieve classification, colorization and super-resolution.

The NWPU-RESISC45 dataset, which is publicly available from the Northwestern Polytechnical University, is the main dataset used in this project. In this dataset, there are a total of 45 categories of remote sensing images, and each category contains 700 remote sensing images, all of which are 256x256 in size.

At the same time, the project is divided by network architecture, and the separate network architecture folder is also a relatively complete code for that network, which can be taken down and run directly.

Other related code repositories

Frontend Code Repository: Intelligent-RS_frontend

Backend Code Repository: Intelligent-RS_backend

Get Started

git clone https://github.com/JackieLin2004/Intelligent-RS.git
cd ./Intelligent-RS/

Requirements

pip install torch==1.12.0
pip install torchvision==0.13.0
pip install pillow==9.4.0
pip install matplotlib==3.5.1
pip install seaborn==0.13.2
pip install scikit-learn==1.2.1
pip install scikit-image==0.24.0

Environments

You can install the environment by the following method:

pip install -r requirements.txt

Or you can execute it if you want the full conda environment:

conda env create -f environment.yaml

0. Preparation

First, you need to put the NWPU-RESISC45 dataset into the folder in the following format:

/dataset/NWPU-RESISC45/subfolder

Though it is not necessary for the categorization task and the super-resolution reconstruction task.

But it's needed for colorization tasks, and it is recommended to run the code under the following folder first:

/data_prepare/

1. Image Classification

This repository utilizes five classical convolutional neural networks for image classification and also experiments with the Transformer architecture for image classification.

Convolutional Neural Networks include network architectures such as AlexNet, VGGNet, GoogLeNet, ResNeXt, and DenseNet, while Transformer architectures include Swin Transformer.

1.1 Preparing the Dataset

Using AlexNet as an example, if you want to use these networks for classification model training, you first need to place the appropriate dataset:

/dataset/NWPU-RESISC45/

1.2 Run the Training Script

/AlexNet/train.py

1.3 Classification Projections

/AlexNet/predictAll.py

1.4 Charting Indicators

/AlexNet/draw_indicators.ipynb

1.5 Indicator Charts for Various Models

1.6 Classification Model Comparison

Networks & Metrics	AlexNet	VGGNet	GoogLeNet	ResNeXt	DenseNet	Swin Transformer
Accuracy	0.864	0.920	0.905	0.938	0.929	0.884
Loss	0.545	0.367	0.417	0.374	0.316	0.456
Precision	0.867	0.922	0.910	0.939	0.931	0.886
Recall	0.864	0.920	0.905	0.938	0.929	0.884
F1 Score	0.864	0.920	0.905	0.938	0.929	0.884
AUC	0.997	0.998	0.998	0.999	0.998	0.997

2. Image Super-Resolution

In this section, two network architecture implementations are used in this project. The former is a traditional convolutional approach and the latter is a generative adversarial network.

For convolutional methods, we used SRResNet.

And for generative adversarial networks, we used SRGAN, ESRGAN. in addition, we made a little improvement to SRGAN, which is tentatively called ISRGAN here.

2.1 Create Data List

Using SRResNet as an example, first you create the data list:

/SRResNet/create_data_list.py

Then the json file will be obtained:

/SRResNet/data/test_images.json and train_images.json

2.2 Run the Training Script

/SRResNet/train.py

2.3 Evaluation of Test Sets

/SRResNet/evaluate.ipynb

2.4 Prediction of a Single Image

/SRResNet/test.py

2.5 Comparison of Data on Indicators

	SRResNet	SRGAN	ISRGAN	ESRGAN
PSNR	34.524	30.628	30.169	30.629
SSIM	0.935	0.891	0.863	0.907
Time	0.008	0.008	0.011	0.015

For SRResNet, the loss varies as shown in Fig:

For SRGAN, the loss varies as shown in Fig:

For ISRGAN, the loss varies as shown in Fig:

For ESRGAN, the loss varies as shown in Fig:

Obviously, as can be seen from the above graph of the loss curve of GAN, the loss of adversarial neural network is not stable, especially the generative loss and discriminative loss are doing fierce confrontation between each other. We do not directly judge the effect by the loss.

2.6 Effective Demonstration

For ISRGAN, we replaced its original ordinary convolution module with a residual module, which is visually better than the original one, although there is no improvement in the metrics.

For ESRGAN, although the super-segmentation is good and clear, it loses a lot of details of the remote sensing image.

However, as can be seen from the loss plot, the overall trend of ISRGAN is smoother and better.

3. Image Colorization

In this section, for the image colorization task, we again proceeded with two ideas. One is CNN architecture to extract color features and the other is GAN architecture to generate images.

The Colorization1 and Colorization2 folders then correspond to the implementation of the CNN architecture and the GAN architecture, respectively.

3.1 Prepared Work

Using Colorization1 as an example, first you should do some prepared work:

/Colorization1/prework.ipynb

Then the target file will be obtained:

/dataset/colorization_/

In the case of the Colorization2 model there is no need for preprocessing, as this is already done in Preparation.

3.2 Run the Training Script

/Colorization1/train.py

3.3 Evaluation of Test Sets

/Colorization1/evaluate.ipynb

3.4 Prediction of a Single Image

/Colorization1/test.py

3.5 Comparison of Data on Indicators

	Colorization1	Colorization2
PSNR	30.733	30.980
SSIM	0.865	0.977
Time	0.0042	0.0265

For Colorization1, the loss varies as shown in Fig:

For Colorization2, the loss varies as shown in Fig:

3.6 Effective Demonstration

Comparing the colorized models of the two architectures, it can be seen that the CNN architecture performs better for green information, while the GAN architecture is more realistic for the overall performance.

If you are interested in this project, feel free to fork and star!