Skip to content

Latest commit

 

History

History
158 lines (129 loc) · 9.14 KB

image_classification.rst

File metadata and controls

158 lines (129 loc) · 9.14 KB
Raw

Image Classification

We provide benchmarks of different domain adaptation algorithms on Digits, Office-31 , Office-Home, VisDA-2017 and DomainNet. Those domain adaptation algorithms includes:

Note

  • Origin means the accuracy reported by the original paper.
  • Avg is the accuracy reported by Trasnfer-Learn.
  • Source Only refers to the model trained with data from the source domain.
  • Oracle refers to the model trained with data from the target domain.

Note

We found that the accuracies of adversarial methods (including DANN, ADDA, CDAN, MCD, BSP and MDD) are not stable even after the random seed is fixed, thus we repeat running adversarial methods on Office-31 and VisDA-2017 for three times and report their average accuracy.

Note

ADDA with gradient reverse layer is frequently benchmarked in the literature. Therefore, we implement this baseline and use ADDAgrl to denote it below.

Digits accuracy

Methods SVHN2MNIST MNIST2USPS USPS2MNIST
Source Only 74.1 82.1 74.6
DANN 90.8 91.7 95.2
DAN 82.1 86.0 89.5
JAN 90.3 84.0 86.8
ADDA 93.3 94.5 98.3
CDAN 93.8 96.0 97.7
MCD 90.6 94.1 97.6
AFN 88.2 88.6 97.2
BSP+DANN 84.2 95.7 97.8
MDD 88.4 94.8 97.8
LDD 93.5 95.5 96.3
MCC 76.6 95.1 94.6

Office-31 accuracy on ResNet-50

Methods Origin Avg A → W D → W W → D A → D D → A W → A
Source Only 76.1 79.5 75.8 95.5 99.0 79.3 63.6 63.8
DANN 82.2 86.1 91.4 97.9 100.0 83.6 73.3 70.4
DAN 80.4 83.7 84.2 98.4 100.0 87.3 66.9 65.2
JAN 84.3 87.0 93.7 98.4 100.0 89.4 69.2 71.0
ADDA / 86.5 91.2 98.5 100.0 84.3 73.7 71.2
ADDAgrl / 87.3 94.6 97.5 99.7 90.0 69.6 72.5
CDAN 87.7 87.7 93.8 98.5 100.0 89.9 73.4 70.4
MCD / 85.4 90.4 98.5 100.0 87.3 68.3 67.6
AFN 85.7 88.6 94.0 98.9 100.0 94.4 72.9 71.1
BSP+DANN 87.7 87.8 92.7 97.9 100.0 88.2 74.1 73.8
MDD 88.9 89.6 95.6 98.6 100.0 94.4 76.6 72.2
MCC 89.4 89.6 94.1 98.4 99.8 95.6 75.5 74.2

Office-Home accuracy on ResNet-50

Methods Origin Avg Ar → Cl Ar → Pr Ar → Rw Cl → Ar Cl → Pr Cl → Rw Pr → Ar Pr → Cl Pr → Rw Rw → Ar Rw → Cl Rw → Pr
Source Only 46.1 58.4 41.1 65.9 73.7 53.1 60.1 63.3 52.2 36.7 71.8 64.8 42.6 75.2
DANN 57.6 65.2 53.8 62.6 74.0 55.8 67.3 67.3 55.8 55.1 77.9 71.1 60.7 81.1
DAN 56.3 61.4 45.6 67.7 73.9 57.7 63.8 66.0 54.9 40.0 74.5 66.2 49.1 77.9
JAN 58.3 65.9 50.8 71.9 76.5 60.6 68.3 68.7 60.5 49.6 76.9 71.0 55.9 80.5
ADDA / 62.5 47.4 63.9 72.6 53.1 62.6 64.3 56.0 49.1 76.3 68.1 56.5 80.3
ADDAgrl / 65.6 52.6 62.9 74.0 59.7 68.0 68.8 61.4 52.5 77.6 71.1 58.6 80.2
CDAN 65.8 68.8 55.2 72.4 77.6 62.0 69.7 70.9 62.4 54.3 80.5 75.5 61.0 83.8
MCD / 67.8 51.7 72.2 78.2 63.7 69.5 70.8 61.5 52.8 78.0 74.5 58.4 81.8
AFN 67.3 68.2 53.2 72.7 76.8 65.0 71.3 72.3 65.0 51.4 77.9 72.3 57.8 82.4
BSP+DANN 64.9 67.6 54.7 67.7 76.2 61.0 69.4 70.9 60.9 55.2 80.2 73.4 60.3 81.2
MDD 68.1 69.7 56.2 75.4 79.6 63.5 72.1 73.8 62.5 54.8 79.9 73.5 60.9 84.5
MCC / 72.4 58.4 79.6 83.0 67.5 77.0 78.5 66.6 54.8 81.8 74.4 61.4 85.6

VisDA-2017 accuracy ResNet-101

Note

  • Origin means the accuracy reported by the original paper.
  • Mean refers to the accuracy average over classes
  • Avg refers to accuracy average over samples.
Methods Origin Mean plane bcycl bus car horse knife mcycl person plant sktbrd train truck Avg
Source Only 52.4 51.7 63.6 35.3 50.6 78.2 74.6 18.7 82.1 16.0 84.2 35.5 77.4 4.7 56.9
DANN 57.4 79.5 93.5 74.3 83.4 50.7 87.2 90.2 89.9 76.1 88.1 91.4 89.7 39.8 74.9
DAN 61.1 66.4 89.2 37.2 77.7 61.8 81.7 64.3 90.6 61.4 79.9 37.7 88.1 27.4 67.2
JAN / 73.4 96.3 66.0 82.0 44.1 86.4 70.3 87.9 74.6 83.0 64.6 84.5 41.3 70.3
ADDA / 79.3 93.6 70.8 83.2 63.5 90.6 93.2 89.0 75.3 88.4 79.3 87.4 37.2 76.4
ADDAgrl / 77.5 95.6 70.8 84.4 54.0 87.8 75.8 88.4 69.3 84.1 86.2 85.0 48.0 74.3
CDAN / 80.1 94.0 69.2 78.9 57.0 89.8 94.9 91.9 80.3 86.8 84.9 85.0 48.5 76.5
MCD 71.9 77.7 87.8 75.7 84.2 78.1 91.6 95.3 88.1 78.3 83.4 64.5 84.8 20.9 76.7
AFN 76.1 75.0 95.6 56.2 81.3 69.8 93.0 81.0 93.4 74.1 91.7 55.0 90.6 18.1 74.4
BSP+DANN 75.9 80.5 95.7 75.6 82.8 54.5 89.2 96.5 91.3 72.2 88.9 88.7 88.0 43.4 76.2
MDD / 82.0 88.3 62.8 85.2 69.9 91.9 95.1 94.4 81.2 93.8 89.8 84.1 47.9 79.8
MCC 78.8 83.6 95.3 85.8 77.1 68.0 93.9 92.9 84.5 79.5 93.6 93.7 85.3 53.8 80.4

DomainNet accuracy on ResNet-101

Methods c->p c->r c->s p->c p->r p->s r->c r->p r->s s->c s->p s->r Avg
Source Only 32.7 50.6 39.4 41.1 56.8 35.0 48.6 48.8 36.1 49.0 34.8 46.1 43.3
DANN 37.9 54.3 44.4 41.7 55.6 36.8 50.7 50.8 40.1 55.0 45.0 54.5 47.2
DAN 38.8 55.2 43.9 45.9 59.0 40.8 50.8 49.8 38.9 56.1 45.9 55.5 48.4
JAN 40.5 56.7 45.1 47.2 59.9 43.0 54.2 52.6 41.9 56.6 46.2 55.5 50.0
ADDA 38.4 54.1 44.1 43.5 56.7 39.2 52.8 51.3 40.9 55.0 45.4 54.5 48.0
CDAN 40.4 56.8 46.1 45.1 58.4 40.5 55.6 53.6 43.0 57.2 46.4 55.7 49.9
MCD 37.5 52.9 44.0 44.6 54.5 41.6 52.0 51.5 39.7 55.5 44.6 52.0 47.5
MDD 42.9 59.5 47.5 48.6 59.4 42.6 58.3 53.7 46.2 58.7 46.5 57.7 51.8
MCC 37.7 55.7 42.6 45.4 59.8 39.9 54.4 53.1 37.0 58.1 46.3 56.2 48.9

Oracle DomainNet accuracy on ResNet-101

Oracle clp inf pnt real skt Avg
/ 78.2 40.7 71.6 83.8 70.6 69.0