init commit

twke18 · Aug 14, 2018 · 5c6048e · 5c6048e
commit 5c6048e
Show file tree

Hide file tree

Showing 47 changed files with 29,763 additions and 0 deletions.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,106 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# dotenv
+.env
+
+# virtualenv
+.venv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# cache dir
+snapshots/
+*.swp
+*.so
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,24 @@
+Adaptive Affinity Fields for Semantic Segmentation
+
+The MIT License
+
+Copyright (c) Tsung-Wei Ke (UC Berkeley/ICSI), Jyh-Jing Hwang (UC Berkeley/ICSI), 
+Ziwei Liu (UC Berkeley/ICSI), and Stella X. Yu (UC Berkeley/ICSI).
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
diff --git a/README.md b/README.md
@@ -0,0 +1,96 @@
+# Adaptive-Affinity-Field for Semantic Segmentation
+
+By [Tsung-Wei Ke*](https://www1.icsi.berkeley.edu/~twke/), Jyh-Jing Hwang*, [Ziwei Liu](https://liuziwei7.github.io/), 
+and [Stella X. Yu](http://www1.icsi.berkeley.edu/~stellayu/) (* equal contribution)
+
+<img src="misc/architecture.png" width="720">
+
+Semantic segmentation has made much progress with increasingly powerful pixel-wise classifiers and incorporating structural 
+priors via Conditional Random Fields (CRF) or Generative Adversarial Networks (GAN). We propose a simpler alternative that 
+learns to verify the spatial structure of segmentation during training only. Unlike existing approaches that enforce semantic 
+labels on individual pixels and match labels between neighbouring pixels, we propose the concept of Adaptive Affinity Fields 
+(AAF) to capture and match the semantic relations between neighbouring pixels in the label space. We use adversarial learning 
+to select the optimal affinity field size for each semantic category. It is formulated as a minimax problem, optimizing our 
+segmentation neural network in a best worst-case learning scenario. AAF is versatile for representing structures as a collection 
+of pixel-centric relations, easier to train than GAN and more efficient than CRF without run-time inference. Our extensive evaluations 
+on PASCAL VOC 2012, Cityscapes, and GTA5 datasets demonstrate its above-par segmentation performance and robust generalization across 
+domains. 
+
+AAF is published in ECCV 2018, see [our paper](https://arxiv.org/abs/1803.10335) for more details.
+
+### License
+AAF is released under the MIT License (refer to the LICENSE file for details).
+
+### Citation
+If you use this code for your research, please cite our paper [Adaptive Affinity Fields for Semantic Segmentation](https://arxiv.org/abs/1803.10335).
+
+```
+@article{aaf2018,
+  title={Adaptive Affinity Fields for Semantic Segmentation},
+  author={Tsung-Wei Ke*, Jyh-Jing Hwang*, Ziwei Liu, and Stella X. Yu.},
+  journal={ECCV},
+  year={2018}
+}
+```
+
+## Prerequisites
+
+1. Linux
+2. Python2.7 or Python3 (>=3.5)
+3. Cuda 8.0 and Cudnn 6
+
+## Required Python Packages
+
+1. tensorflow 1.4 (for versions >= 1.6 might cause OOM error)
+2. numpy
+3. scipy
+4. tqdm
+5. PIL
+6. opencv
+
+## Data Preparation
+
+* [PASCAL VOC 2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/)
+* [Cityscapes](https://www.cityscapes-dataset.com/)
+
+## ImageNet Pre-Trained Models
+
+Download ResNet101.v1 from [Tensorflow-Slim](https://github.com/tensorflow/models/tree/master/research/slim).
+
+## Training
+
+* Baseline Models:
+```
+python pyscripts/train/train.py
+```
+
+* Affinity
+```
+python pyscripts/train/train_aff.py
+```
+
+* AAF
+```
+python pyscripts/train/train_aaf.py
+```
+
+## Inference
+
+* Single-Scale Input only
+```
+python pyscripts/test/evaluate.py
+```
+
+* Multi-Scale Inputs and Left-Right Flipping (opencv is required)
+```
+python pyscripts/test/evaluate_msc.py
+```
+
+## Benchmarking
+
+* mIoU
+```
+python pyscripts/utils/benchmark_by_mIoU.py
+```
+
+See our [bash script examples](/bashscripts/) for the corresponding input arguments.
diff --git a/bashscripts/voc12/train_pspnet.sh b/bashscripts/voc12/train_pspnet.sh
@@ -0,0 +1,114 @@
+#!/bin/bash
+# Training Parameters
+BATCH_SIZE=8
+TRAIN_INPUT_SIZE=336,336
+WEIGHT_DECAY=5e-4
+ITER_SIZE=1
+NUM_STEPS=30000
+NUM_CLASSES=21
+# Testing Parameters
+TEST_INPUT_SIZE=480,480
+TEST_STRIDES=320,320
+TEST_SPLIT=val
+# saved model path
+SNAPSHOT_DIR=snapshots/voc12/pspnet/p336_bs8_lr1e-3_it30k
+# Procedure pipeline
+IS_TRAIN_1=1
+IS_TEST_1=1
+IS_BENCHMARK_1=1
+IS_TRAIN_2=1
+IS_TEST_2=1
+IS_BENCHMARK_2=1
+
+export PYTHONPATH=`pwd`:$PYTHONPATH
+DATAROOT=/path/to/data
+
+
+# Stage1 Training
+if [ ${IS_TRAIN_1} -eq 1 ]; then
+  python3 pyscripts/train/train.py\
+    --snapshot-dir ${SNAPSHOT_DIR}/stage1\
+    --restore-from snapshots/imagenet/trained/resnet_v1_101.ckpt\
+    --data-list dataset/voc12/train+.txt\
+    --data-dir ${DATAROOT}/VOCdevkit/\
+    --batch-size ${BATCH_SIZE}\
+    --save-pred-every 10000\
+    --update-tb-every 50\
+    --input-size ${TRAIN_INPUT_SIZE}\
+    --learning-rate 1e-3\
+    --weight-decay ${WEIGHT_DECAY}\
+    --iter-size ${ITER_SIZE}\
+    --num-classes ${NUM_CLASSES}\
+    --num-steps $(($NUM_STEPS+1))\
+    --random-mirror\
+    --random-scale\
+    --random-crop\
+    --not-restore-classifier\
+    --is-training
+fi
+
+# Stage1 Testing
+if [ ${IS_TEST_1} -eq 1 ]; then
+  python3 pyscripts/test/evaluate.py\
+    --data-dir ${DATAROOT}/VOCdevkit/\
+    --data-list dataset/voc12/${TEST_SPLIT}.txt\
+    --input-size ${TEST_INPUT_SIZE}\
+    --strides ${TEST_STRIDES}\
+    --restore-from ${SNAPSHOT_DIR}/stage1/model.ckpt-${NUM_STEPS}\
+    --colormap misc/colormapvoc.mat\
+    --num-classes ${NUM_CLASSES}\
+    --ignore-label 255\
+    --save-dir ${SNAPSHOT_DIR}/stage1/results/${TEST_SPLIT}
+fi
+
+if [ ${IS_BENCHMARK_1} -eq 1 ]; then
+  python3 pyscripts/utils/benchmark_by_mIoU.py\
+    --pred-dir ${SNAPSHOT_DIR}/stage1/results/${TEST_SPLIT}/gray/\
+    --gt-dir ${DATAROOT}/VOCdevkit/VOC2012/segcls/\
+    --num-classes ${NUM_CLASSES}
+fi
+
+# Stage2 Training
+if [ ${IS_TRAIN_2} -eq 1 ]; then
+  python3 pyscripts/train/train.py\
+    --snapshot-dir ${SNAPSHOT_DIR}/stage2\
+    --restore-from ${SNAPSHOT_DIR}/stage1/model.ckpt-30000\
+    --data-list dataset/voc12/train.txt\
+    --data-dir ${DATAROOT}/VOCdevkit/\
+    --batch-size ${BATCH_SIZE}\
+    --save-pred-every 10000\
+    --update-tb-every 50\
+    --input-size ${TRAIN_INPUT_SIZE}\
+    --learning-rate 1e-4\
+    --weight-decay ${WEIGHT_DECAY}\
+    --iter-size ${ITER_SIZE}\
+    --num-classes ${NUM_CLASSES}\
+    --num-steps $(($NUM_STEPS+1))\
+    --random-mirror\
+    --random-scale\
+    --random-crop\
+    --is-training
+fi
+
+# Stage2 Testing
+if [ ${IS_TEST_2} -eq 1 ]; then
+  python3 pyscripts/test/evaluate_msc.py\
+    --data-dir ${DATAROOT}/VOCdevkit/\
+    --data-list dataset/voc12/${TEST_SPLIT}.txt\
+    --input-size ${TEST_INPUT_SIZE}\
+    --strides ${TEST_STRIDES}\
+    --restore-from ${SNAPSHOT_DIR}/stage2/model.ckpt-${NUM_STEPS}\
+    --colormap misc/colormapvoc.mat\
+    --num-classes ${NUM_CLASSES}\
+    --ignore-label 255\
+    --flip-aug\
+    --scale-aug\
+    --save-dir ${SNAPSHOT_DIR}/stage2/results/${TEST_SPLIT}
+fi
+
+if [ ${IS_BENCHMARK_2} -eq 1 ]; then
+  python3 pyscripts/utils/benchmark_by_mIoU.py\
+    --pred-dir ${SNAPSHOT_DIR}/stage2/results/${TEST_SPLIT}/gray/\
+    --gt-dir ${DATAROOT}/VOCdevkit/VOC2012/segcls/\
+    --num-classes ${NUM_CLASSES}
+fi