Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network

• PDF: arXiv or sciencedirect

Overview of our method

Results download

 Our results on DHF1K, HollyWood-2, UCF and DIEM can be downloaded by Google Drive or Baidu wangpan. Which consists of 4 training settings with the training set(s) from (i)DHF1K, (ii)HollyWood-2, (iii)UCF sports, (iv)DHF1K + HollyWood-2 + UCF sports.

settings/datasets	DHF1K	HollyWood-2	UCF	DIEM
setting(i)
setting(ii)
setting(iii)
setting(iv)

Preparation

Datasets download

 How to get million pictures is the first barrier in video saliency prediction task. Thanks to @wenguanwang proposed, pre-processed and published some pictures. DHF1K, HollyWood-2, UCF can be downloaded by Google Drive in his repo.

 For convenience, we clone and upload the duplicate of HollyWood-2 to Baidu wangpan to download. All data belongs to the original author, sharing the duplicate is only for academic development, if there is any infringement, please contact me.

 We adopt here to pre-processed the DIEM datasets. Following STRA-Net, the testing sets contain 20 selected videos which including first 300 frames per video, and some frames without labels are eliminated. Click here to download in Google Drive.

Models download

 We use the VGG16 pretrained weights from PyTorch official version here, and we remove the last few layers. Click here to download in Google Drive, click here to download in Baidu wangpan.

 The trained model in setting(iv): click here to download in Google Drive, click here to download in Baidu wangpan.

Experiments platform

 OS: ubuntu 16.04

 RAM: 64G

 CPU: Intel i7-8700

 GPU: Nvidia RTX 2080Ti * 2

 Language: Python 3

Enviroment dependencies

 Due to the compilation of DCN need earlier version PyTorch and torch.trapz() function need newer, so our dependencies are listed:

 Training and testing phase: PyTorch 1.0.1.post2, torchvision, Pillow, numpy, scipy and other dependencies.

 Eval phase: PyTorch 1.2 or newer, Pillow, numpy, scipy, tkinter and other dependencies.

First

• We use DCN-V2 and modify something in dcn_v2.py, you need replace file and re-compile it.

Test

• Get or download the dataset.
• Download pretrained model in Google Drive.
• Modify the config.py and run test.py.

Val

• Modify the config.py and run eval.py.

Train

• Get or download the dataset.
• Download VGG16 pretrained weights in Google Drive. Actually is from PyTorch offical model weights, expect for deleting the last serveral layers.
• Modify the config.py and run main.py.

Notes

• There is something wrong about the share of BaiduPan, contact me if want.

Schedule

• Create github repo (2019.12.29)
• Release arXiv pdf (2020.01.05)
• Release all results (2020.01.09)
• Add preparation (2020.01.13)
• Test and Train code (2021.06.04)