This is the official repository for the paper Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels. (Accepted by AAAI 2021).
@inproceedings{chen2021robustness,
title={Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels.},
author={Chen, Pengfei and Ye, Junjie and Chen, Guangyong and Zhao, Jingwei and Heng, Pheng-Ann},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2021}
}
Under diagonally-dominant class-conditional label noise, the main conclusions are as follows.
- A classifier maximizing its accuracy on the noisy distribution is guaranteed to maximize the accuracy on clean distribution.
- We can obtain an optimal classifier by maximizing training accuracy on sufficiently many noisy samples.
- A noisy validation set is reliable.
Regarding the the critical demand of model selection in scenarios like hyperparameter-tuning and early stopping, we theoretically prove that a noisy validation set is reliable. We empritically verify the utility of a noisy validation set by showing the impressive performance of a very simple method Noisy best Teacher and Student (NTS).
- Python 3.6+
- PyTorch 1.2+
- torchvision 0.4+
- pillow 5.0+
- numpy 1.17+
| File | Usage |
|---|---|
| hyperparameter.txt | containing detailed hyperparameters |
| command.txt | containing commands for running the training |
| train_cifar10_ce.py | training on CIFAR-10 with normal cross-entropy (NT and NS) |
| train_cifar10_ct.py | training on CIFAR-10 with Co-teaching (NT and NS) |
| train_cifar10_dmi.py | training on CIFAR-10 with DMI (NT and NS) |
| train_cifar10_gce.py | training on CIFAR-10 with GCE (NT and NS) |
| train_cifar100_ce.py | training on CIFAR-10 with normal cross-entropy (NT and NS) |
| train_cifar100_ct.py | training on CIFAR-100 with Co-teaching (NT and NS) |
| train_cifar100_dmi.py | training on CIFAR-100 with DMI (NT and NS) |
| train_cifar100_gce.py | training on CIFAR-100 with GCE (NT and NS) |
| train_clothing1m_ce.py | training on Clothing1M with normal cross-entropy (NS) |
| train_clothing1m_dividemix.py | training on Clothing1M with DivideMix (NT) |
Please refer to hyperparameter.txt and command.txt for detailed hyperparameters and commands.
**CE CIFAR-10 Teacher**
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.2 --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.3 --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.4 --save_model
**CE CIFAR-100 Teacher**
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.2 --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.3 --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.4 --save_model
**GCE CIFAR-10 Teacher**
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.2 --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.3 --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.4 --save_model
**GCE CIFAR-100 Teacher**
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.2 --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.3 --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.4 --save_model
**Co-T CIFAR-10 Teacher**
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.2 --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.4 --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.6 --e_warm 60 --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.2 --tau 0.1 --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.3 --tau 0.15 --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.4 --tau 0.2 --e_warm 60 --save_model
**Co-T CIFAR-100 Teacher**
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.2 --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.4 --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.6 --e_warm 60 --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.2 --tau 0.2 --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.3 --tau 0.3 --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.4 --tau 0.4 --e_warm 60 --save_model
**DMI CIFAR-10 teacher** (DMI requires a pretrained model, e.g., using CE, to initialize)
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.2 --init_path cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.3 --init_path cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.4 --init_path cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model
**DMI CIFAR-100 Teacher**
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.2 --init_path cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.3 --init_path cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.4 --init_path cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --save_model
**CE CIFAR-10 Student**
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.2 --teacher_path cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.3 --teacher_path cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.4 --teacher_path cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model
**CE CIFAR-100 Student**
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.2 --teacher_path cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.3 --teacher_path cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.4 --teacher_path cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --save_model
**GCE CIFAR-10 Student**
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path gce_cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path gce_cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path gce_cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.2 --teacher_path gce_cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.3 --teacher_path gce_cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.4 --teacher_path gce_cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model
**GCE CIFAR-100 Student**
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path gce_cifar100_uniform0.2_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path gce_cifar100_uniform0.4_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path gce_cifar100_uniform0.6_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.2 --teacher_path gce_cifar100_pair0.2_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.3 --teacher_path gce_cifar100_pair0.3_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.4 --teacher_path gce_cifar100_pair0.4_dp0.0_augstandard_seed0_best.pth --save_model
**Co-T CIFAR-10 Student**
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.05 --teacher_path ct_cifar10_uniform0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.1 --teacher_path ct_cifar10_uniform0.4_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.15 --e_warm 60 --teacher_path ct_cifar10_uniform0.6_warm60_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.2 --tau 0.03 --teacher_path ct_cifar10_asym0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.3 --tau 0.04 --teacher_path ct_cifar10_asym0.3_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.4 --tau 0.05 --e_warm 60 --teacher_path ct_cifar10_asym0.4_warm60_dp0.2_augstrong_seed0_best.pth --save_model
**Co-T CIFAR-100 Student**
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.1 --teacher_path ct_cifar100_uniform0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.2 --teacher_path ct_cifar100_uniform0.4_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.3 --e_warm 60 --teacher_path ct_cifar100_uniform0.6_warm60_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.2 --tau 0.1 --teacher_path ct_cifar100_pair0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.3 --tau 0.2 --teacher_path ct_cifar100_pair0.3_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.4 --tau 0.3 --e_warm 60 --teacher_path ct_cifar100_pair0.4_warm60_dp0.2_augstrong_seed0_best.pth --save_model
**DMI CIFAR-10 Student**
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.2 --init_path cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.3 --init_path cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.4 --init_path cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model
**DMI CIFAR-100 Student**
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.2 --init_path cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.3 --init_path cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.4 --init_path cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --save_model
**Teacher and Student: Clothing1M**
python train_clothing1m_dividemix.py --root data/Clothing1M/
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path dividemix_net1.pth
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path dividemix_net2.pth
**Teacher and Student: Clothing1M (Noisy Validation)**
python train_clothing1m_dividemix.py --root data/Clothing1M/ --use_noisy_val
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path nv_dividemix_net1.pth --use_noisy_val
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path nv_dividemix_net2.pth --use_noisy_val