diff --git a/docs/baseline_results/carla_mot_results.md b/docs/baseline_results/carla_mot_results.md index 0577c7c11..06354bb74 100644 --- a/docs/baseline_results/carla_mot_results.md +++ b/docs/baseline_results/carla_mot_results.md @@ -2,18 +2,18 @@ This is the baseline evaluation for the multi-object tracking scenario. For single-object tracking, see [carla_video_tracking_results.md](../baseline_results/carla_video_tracking_results.md). -For [dev data](https://github.com/twosixlabs/armory/blob/master/armory/data/adversarial/carla_mot_dev.py), results obtained using Armory v0.16.1. -[Test](https://github.com/twosixlabs/armory/blob/master/armory/data/adversarial/carla_mot_test.py) results obtained using Armory v0.17.2. +Results obtained using Armory v0.18.0. + | Data | Defended | Attack | Attack Parameters | Benign DetA / AssA / HOTA | Adversarial DetA / AssA / HOTA | Test Size | |------|----------|-------------------|--------------------------------|---------------------------|--------------------------------|-----------| -| Dev | no | Adversarial Patch | step_size=0.02, max_iter=100 | 0.49 / 0.62 / 0.55 | 0.18 / 0.57 / 0.32 | 20 | -| Dev | no | Robust DPatch | step_size=0.002, max_iter=1000 | 0.49 / 0.62 / 0.55 | 0.39 / 0.59 / 0.48 | 20 | -| Dev | yes | Robust DPatch | step_size=0.002, max_iter=1000 | 0.34 / 0.53 / 0.42 | 0.24 / 0.51 / 0.34 | 20 | -| Test | no | Adversarial Patch | step_size=0.02, max_iter=100 | 0.43 / 0.51 / 0.46 | 0.19 / 0.45 / 0.29 | 10 | -| Test | no | Robust DPatch | step_size=0.002, max_iter=1000 | 0.43 / 0.51 / 0.46 | 0.31 / 0.46 / 0.37 | 10 | -| Test | yes | Robust DPatch | step_size=0.002, max_iter=1000 | 0.32 / 0.45 / 0.38 | 0.22 / 0.41 / 0.30 | 10 | +| Dev | no | Adversarial Patch | step_size=0.02, max_iter=100 | 0.55 / 0.64 / 0.59 | 0.15 / 0.58 / 0.29 | 20 | +| Dev | no | Robust DPatch | step_size=0.002, max_iter=1000 | 0.55 / 0.64 / 0.59 | 0.42 / 0.61 / 0.50 | 20 | +| Dev | yes | Robust DPatch | step_size=0.002, max_iter=1000 | 0.36 / 0.53 / 0.44 | 0.25 / 0.49 / 0.35 | 20 | +| Test | no | Adversarial Patch | step_size=0.02, max_iter=100 | 0.45 / 0.55 / 0.49 | 0.25 / 0.47 / 0.35 | 10 | +| Test | no | Robust DPatch | step_size=0.002, max_iter=1000 | 0.45 / 0.55 / 0.49 | 0.36 / 0.49 / 0.41 | 10 | +| Test | yes | Robust DPatch | step_size=0.002, max_iter=1000 | 0.31 / 0.44 / 0.37 | 0.22 / 0.39 / 0.29 | 10 | Defended results not available for Adversarial Patch attack because JPEG Compression defense is not implemented in PyTorch and so is not fully differentiable. Note that Robust DPatch is considerably slower than Adversarial Patch. diff --git a/docs/baseline_results/carla_od_results.md b/docs/baseline_results/carla_od_results.md index 00b1db45b..cb14a18c6 100644 --- a/docs/baseline_results/carla_od_results.md +++ b/docs/baseline_results/carla_od_results.md @@ -5,21 +5,21 @@ Single Modality (RGB) Object Detection -| Data | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | +| Data | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | |------|-------------------|------------------------------------|-------------|-----------------------------|----------------------------------|---------------------------------|-----------------------------|------------------|----------------------------------|--------------------------------------|-------------------------------------|---------------------------------|-----------| -| Dev | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.76/0.72 | 0.19/0.22 | 3.97/3.48 | 0.06/0.06 | 0.75/0.71 | 0.68/0.66 | 0.27/0.28 | 4.48/3.65 | 0.06/0.07 | 0.67/0.65 | 31 | -| Dev | Adversarial Patch | learning_rate=0.003, max_iter=1000 | 0.76/0.72 | 0.19/0.22 | 3.97/3.48 | 0.06/0.06 | 0.75/0.71 | 0.54/* | 0.32/* | 22.16/* | 0.05/* | 0.62/* | 31 | -| Test | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.79/0.74 | 0.16/0.25 | 4.10/3.50 | 0.03/0.01 | 0.82/0.75 | 0.72/0.64 | 0.32/0.39 | 4.80/4.0 | 0.03/0.01 | 0.65/0.60 | 20 | -| Test | Adversarial Patch | learning_rate=0.003, max_iter=1000 | 0.79/0.74 | 0.16/0.25 | 4.10/3.50 | 0.03/0.01 | 0.82/0.75 | 0.38/* | 0.40/* | 42.55/* | 0.03/* | 0.57/* | 20 | +| Dev | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.76/0.72 | 0.19/0.22 | 3.97/3.48 | 0.06/0.06 | 0.75/0.71 | 0.68/0.66 | 0.27/0.28 | 4.48/3.65 | 0.06/0.07 | 0.67/0.65 | 31 | +| Dev | Adversarial Patch | learning_rate=0.003, max_iter=1000 | 0.76/0.72 | 0.19/0.22 | 3.97/3.48 | 0.06/0.06 | 0.75/0.71 | 0.54/* | 0.32/* | 22.16/* | 0.05/* | 0.62/* | 31 | +| Test | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.79/0.74 | 0.16/0.25 | 4.10/3.50 | 0.03/0.01 | 0.82/0.75 | 0.72/0.64 | 0.32/0.39 | 4.80/4.0 | 0.03/0.01 | 0.65/0.60 | 20 | +| Test | Adversarial Patch | learning_rate=0.003, max_iter=1000 | 0.79/0.74 | 0.16/0.25 | 4.10/3.50 | 0.03/0.01 | 0.82/0.75 | 0.38/* | 0.40/* | 42.55/* | 0.03/* | 0.57/* | 20 | Multimodality (RGB+depth) Object Detection -| Data | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | +| Data | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | |------|-------------------|--------------------------------------------------------------------------------------|-------------|-----------------------------|----------------------------------|---------------------------------|-----------------------------|------------------|----------------------------------|--------------------------------------|-------------------------------------|---------------------------------|-----------| -| Dev | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.0001, max_iter=2000 | 0.87/0.86 | 0.06/0.04 | 1.23/2.55 | 0.05/0.05 | 0.88/0.91 | 0.76/0.83 | 0.10/0.06 | 5.68/4.87 | 0.05/0.05 | 0.84/0.89 | 31 | -| Dev | Adversarial Patch | depth_delta_meters=3, learning_rate=0.003, learning_rate_depth=0.0001, max_iter=1000 | 0.87/0.86 | 0.06/0.04 | 1.23/2.55 | 0.05/0.05 | 0.88/0.91 | 0.66/0.76 | 0.11/0.10 | 10.74/7.13 | 0.06/0.05 | 0.83/0.85 | 31 | -| Test | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.0001, max_iter=2000 | 0.90/0.89 | 0.03/0.04 | 1.0/1.45 | 0.03/0.02 | 0.94/0.94 | 0.81/0.89 | 0.13/0.06 | 4.75/2.05 | 0.03/0.02 | 0.83/0.91 | 20 | -| Test | Adversarial Patch | depth_delta_meters=3, learning_rate=0.003, learning_rate_depth=0.0001, max_iter=1000 | 0.90/0.89 | 0.03/0.04 | 1.0/1.45 | 0.03/0.02 | 0.94/0.94 | 0.50/0.57 | 0.21/0.14 | 22.55/13.70 | 0.04/0.03 | 0.75/0.83 | 20 | +| Dev | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.0001, max_iter=2000 | 0.87/0.86 | 0.06/0.04 | 1.23/2.55 | 0.05/0.05 | 0.88/0.91 | 0.76/0.83 | 0.10/0.06 | 5.68/4.87 | 0.05/0.05 | 0.84/0.89 | 31 | +| Dev | Adversarial Patch | depth_delta_meters=3, learning_rate=0.003, learning_rate_depth=0.0001, max_iter=1000 | 0.87/0.86 | 0.06/0.04 | 1.23/2.55 | 0.05/0.05 | 0.88/0.91 | 0.66/0.76 | 0.11/0.10 | 10.74/7.13 | 0.06/0.05 | 0.83/0.85 | 31 | +| Test | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.0001, max_iter=2000 | 0.90/0.89 | 0.03/0.04 | 1.0/1.45 | 0.03/0.02 | 0.94/0.94 | 0.81/0.89 | 0.13/0.06 | 4.75/2.05 | 0.03/0.02 | 0.83/0.91 | 20 | +| Test | Adversarial Patch | depth_delta_meters=3, learning_rate=0.003, learning_rate_depth=0.0001, max_iter=1000 | 0.90/0.89 | 0.03/0.04 | 1.0/1.45 | 0.03/0.02 | 0.94/0.94 | 0.50/0.57 | 0.21/0.14 | 22.55/13.70 | 0.04/0.03 | 0.75/0.83 | 20 | a/b in the tables refer to undefended/defended performance results, respectively. @@ -30,26 +30,44 @@ Find reference baseline configurations [here](https://github.com/twosixlabs/armo ## CARLA Overhead OD Dataset -Dev data results obtained using Armory 0.16.6, Test data results obtained using Armory 0.16.1 +Results obtained using Armory 0.18.1. Single Modality (RGB) Object Detection -| Data | Defended | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | -|------|----------|-------------------|------------------------------------|-------------|-----------------------------|----------------------------------|---------------------------------|-----------------------------|------------------|----------------------------------|--------------------------------------|-------------------------------------|---------------------------------|-----------| -| Dev 2.0.0 | no | Adversarial Patch | learning_rate=0.003, max_iter=1000 | 0.65 | 0.29 | 3.1 | 0.03 | 0.68 | 0.05 | 0.80 | 56.1 | 0.01 | 0.19 | 20 | -| Dev 2.0.0 | no | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.65 | 0.29 | 3.1 | 0.03 | 0.68 | 0.43 | 0.40 | 16.9 | 0.03 | 0.57 | 20 | -| Dev 2.0.0 | yes | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.59 | 0.43 | 1.7 | 0.03 | 0.54 | 0.40 | 0.52 | 9.0 | 0.03 | 0.45 | 20 | -| Test 1.0.0 | no | Adversarial Patch | learning_rate=0.003, max_iter=1000 | 0.60 | 0.42 | 3.6 | 0.03 | 0.55 | 0.04 | 0.81 | 54.1 | 0.0 | 0.19 | 15 | +| Data | Split | Defended | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | +|------|---|----------|-------------------|------------------------------------|-------------|-----------------------------|----------------------------------|---------------------------------|-----------------------------|------------------|----------------------------------|--------------------------------------|-------------------------------------|---------------------------------|-----------| +| Dev | dev | no | Adversarial Patch | learning_rate=0.05, max_iter=500, optimizer=Adam | 0.78 | 0.15 | 6.2 | 0.04 | 0.81 | 0.01 | 0.95 | 91.5 | 0.0 | 0.05 | 20 | +| Dev | dev | no | Adversarial Patch Targeted | learning_rate=0.05, max_iter=500, hallucination_per_label=300, optimizer=Adam | 0.78 | 0.15 | 6.2 | 0.04 | 0.81 | 0.44 | 0.42 | 67.2 | 0.03 | 0.55 | 20 | +| Dev | dev | no | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.78 | 0.15 | 6.2 | 0.04 | 0.81 | 0.69 | 0.24 | 7.85 | 0.03 | 0.72 | 20 | +| Dev | dev | yes | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.62 | 0.37 | 3.0 | 0.03 | 0.60 | 0.50 | 0.46 | 9.4 | 0.03 | 0.51 | 20 | +| Test | test_hallucination | no | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.74 | 0.15 | 3.6 | 0.05 | 0.80 | 0.32 | 0.18 | 30.3 | 0.04 | 0.78 | 25 | +| Test | test_disappearance | no | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.74 | 0.25 | 5.36 | 0.03 | 0.72 | 0.63 | 0.34 | 8.12 | 0.02 | 0.64 | 25 | +| Test | test_hallucination | yes | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.61 | 0.4 | 2.6 | 0.04 | 0.56 | 0.41 | 0.41 | 28.8 | 0.04 | 0.55 | 25 | +| Test | test_disappearance | yes | Robust DPatch | learning_rate=0.002, max_iter=2000 | 0.56 | 0.46 | 3.7 | 0.02 | 0.52 | 0.42 | 0.55 | 14.5 | 0.01 | 0.44 | 25 | +| Test | test_hallucination | no | Adversarial Patch | learning_rate=0.05, max_iter=500, optimizer=Adam | 0.74 | 0.15 | 3.6 | 0.05 | 0.80 | 0.0 | 1.0 | 100.0 | 0.0 | 0.0 | 25 | +| Test | test_disappearance | no | Adversarial Patch | learning_rate=0.05, max_iter=500, optimizer=Adam | 0.74 | 0.25 | 5.36 | 0.03 | 0.72 | 0.01 | 0.98 | 99.2 | 0.0 | 0.02 | 25 | +| Test | test_hallucination | no | Adversarial Patch Targeted | learning_rate=0.05, max_iter=500, hallucination_per_label=300, optimizer=Adam | 0.74 | 0.15 | 3.6 | 0.05 | 0.80 | 0.60 | 0.28 | 71.2 | 0.04 | 0.68 | 25 | +| Test | test_disappearance | no | Adversarial Patch Targeted | learning_rate=0.05, max_iter=500, hallucination_per_label=300, optimizer=Adam | 0.74 | 0.25 | 5.4 | 0.03 | 0.72 | 0.44 | 0.48 | 64.6 | 0.02 | 0.50 | 25 | Multimodality (RGB+depth) Object Detection -| Data | Defended | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | -|------|----------|-------------------|-----------------------------------------------------------------------------------------|-------------|-----------------------------|----------------------------------|---------------------------------|-----------------------------|------------------|----------------------------------|--------------------------------------|-------------------------------------|---------------------------------|-----------| -| Dev 2.0.0 | no | Adversarial Patch | depth_delta_meters=3, learning_rate=0.003, learning_rate_depth=0.005, max_iter=1000 | 0.66 | 0.29 | 2.9 | 0.03 | 0.68 | 0.14 | 0.56 | 29.5 | 0.02 | 0.41 | 20 | -| Dev 2.0.0 | yes | Adversarial Patch | depth_delta_meters=3, learning_rate=0.003, learning_rate_depth=0.005, max_iter=1000 | 0.70 | 0.28 | 1.9 | 0.03 | 0.69 | 0.16 | 0.51 | 25.7 | 0.02 | 0.47 | 20 | -| Dev 2.0.0 | no | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.66 | 0.29 | 2.9 | 0.03 | 0.68 | 0.59 | 0.37 | 3.3 | 0.03 | 0.60 | 20 | -| Dev 2.0.0 | yes | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.70 | 0.28 | 1.9 | 0.03 | 0.69 | 0.61 | 0.37 | 1.7 | 0.03 | 0.60 | 20 | -| Test 1.0.0 | no | Adversarial Patch | depth_delta_meters=0.03, learning_rate=0.003, learning_rate_depth=0.0001, max_iter=1000 | 0.58 | 0.39 | 0.8 | 0.03 | 0.58 | 0.19 | 0.72 | 15.8 | 0.01 | 0.23 | 15 | +| Data | Split | Defended | Attack | Attack Parameters | Benign mAP | Benign Disappearance Rate | Benign Hallucination per Image | Benign Misclassification Rate | Benign True Positive Rate | Adversarial mAP | Adversarial Disappearance Rate | Adversarial Hallucination per Image | Adversarial Misclassification Rate | Adversarial True Positive Rate | Test Size | +|------|---|----------|-------------------|-----------------------------------------------------------------------------------------|-------------|-----------------------------|----------------------------------|---------------------------------|-----------------------------|------------------|----------------------------------|--------------------------------------|-------------------------------------|---------------------------------|-----------| +| Dev | dev | no | Adversarial Patch | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam | 0.79 | 0.13 | 4.5 | 0.04 | 0.83 | 0.18 | 0.38 | 39.0 | 0.03 | 0.59 | 20 | +| Dev | dev | yes | Adversarial Patch | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam | 0.80 | 0.14 | 2.8 | 0.03 | 0.83 | 0.21 | 0.39 | 31.2 | 0.02 | 0.59 | 20 | +| Dev | dev | no | Adversarial Patch Targeted | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam, hallucination_per_label=300 | 0.79 | 0.13 | 4.5 | 0.04 | 0.83 | 0.67 | 0.21 | 17.6 | 0.05 | 0.74 | 20 | +| Dev | dev | no | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.79 | 0.13 | 4.5 | 0.04 | 0.83 | 0.74 | 0.20 | 4.2 | 0.04 | 0.77 | 20 | +| Dev | dev | yes | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.80 | 0.14 | 2.8 | 0.03 | 0.83 | 0.78 | 0.21 | 2.65 | 0.03 | 0.76 | 20 | +| Test | test_hallucination | no | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.78 | 0.10 | 3.0 | 0.05 | 0.85 | 0.77 | 0.10 | 4.9 | 0.05 | 0.85 | 25 | +| Test | test_disappearance | no | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.76 | 0.17 | 3.3 | 0.04 | 0.79 | 0.73 | 0.27 | 4.1 | 0.03 | 0.70 | 25 | +| Test | test_hallucination | yes | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.82 | 0.10 | 1.96 | 0.05 | 0.84 | 0.81 | 0.11 | 2.08 | 0.05 | 0.83 | 25 | +| Test | test_disappearance | yes | Robust DPatch | depth_delta_meters=3, learning_rate=0.002, learning_rate_depth=0.003, max_iter=2000 | 0.81 | 0.16 | 2.4 | 0.04 | 0.80 | 0.76 | 0.26 | 2.28 | 0.02 | 0.71 | 25 | +| Test | test_hallucination | no | Adversarial Patch | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam | 0.78 | 0.10 | 3.0 | 0.05 | 0.85 | 0.2 | 0.69 | 92.7 | 0.01 | 0.30 | 25 | +| Test | test_disappearance | no | Adversarial Patch | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam | 0.76 | 0.17 | 3.3 | 0.04 | 0.79 | 0.55 | 0.36 | 6.16 | 0.04 | 0.61 | 25 | +| Test | test_hallucination | yes | Adversarial Patch | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam | 0.82 | 0.10 | 2.0 | 0.05 | 0.84 | 0.05 | 0.51 | 78.9 | 0.03 | 0.46 | 25 | +| Test | test_disappearance | yes | Adversarial Patch | learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam | 0.81 | 0.16 | 2.4 | 0.04 | 0.80 | 0.45 | 0.36 | 12.3 | 0.03 | 0.62 | 25 | +| Test | test_hallucination | no | Adversarial Patch Targeted | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam, hallucination_per_label=300 | 0.78 | 0.10 | 3.0 | 0.05 | 0.85 | 0.73 | 0.17 | 22.8 | 0.05 | 0.79 | 25 | +| Test | test_disappearance | no | Adversarial Patch Targeted | depth_delta_meters=3, learning_rate=0.02, learning_rate_depth=0.0001, max_iter=1000, optimizer=Adam, hallucination_per_label=300 | 0.76 | 0.17 | 3.28 | 0.04 | 0.79 | 0.69 | 0.27 | 15.0 | 0.03 | 0.70 | 25 | Defended results not available for Adversarial Patch attack against single modality because JPEG Compression defense is not implemented in PyTorch and so is not fully differentiable diff --git a/docs/baseline_results/object_detection_poisoning_results.md b/docs/baseline_results/object_detection_poisoning_results.md index 8aa7ae2a5..d95bc980a 100644 --- a/docs/baseline_results/object_detection_poisoning_results.md +++ b/docs/baseline_results/object_detection_poisoning_results.md @@ -13,6 +13,8 @@ Relevant parameters: See [the paper](https://arxiv.org/pdf/2205.14497.pdf) for a detailed description of each metric recorded. +Example configs exist at [scenario_configs/eval7/poisoning](https://github.com/twosixlabs/armory/tree/master/scenario_configs/eval7/poisoning). The relevant triggers can be found in [armory/utils/triggers/](https://github.com/twosixlabs/armory/tree/master/armory/utils/triggers) + # Global Misclassification Attack ## Globe trigger @@ -93,6 +95,80 @@ See [the paper](https://arxiv.org/pdf/2205.14497.pdf) for a detailed description | 30 | 0.493 | 0.447 | 0.494 | 0.447 | 0.077 | 0.230 | 0.015 | +## Skull trigger + +### Undefended + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.536 | 0.470 | - | - | - | - | - | +| 01 | 0.522 | 0.463 | 0.523 | 0.463 | 0.080 | 0.240 | 0.021 | +| 05 | 0.497 | 0.460 | 0.182 | 0.303 | 0.146 | 0.437 | 0.591 | +| 10 | 0.503 | 0.480 | 0.160 | 0.320 | 0.159 | 0.477 | 0.662 | +| 20 | 0.492 | 0.457 | 0.158 | 0.303 | 0.157 | 0.470 | 0.684 | +| 30 | 0.496 | 0.447 | 0.154 | 0.303 | 0.167 | 0.500 | 0.703 | + + +### Random Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.452 | 0.387 | - | - | - | - | - | +| 01 | 0.458 | 0.430 | 0.457 | 0.427 | 0.076 | 0.227 | 0.013 | +| 05 | 0.444 | 0.430 | 0.380 | 0.390 | 0.089 | 0.267 | 0.135 | +| 10 | 0.462 | 0.437 | 0.141 | 0.250 | 0.139 | 0.417 | 0.601 | +| 20 | 0.464 | 0.450 | 0.159 | 0.320 | 0.148 | 0.443 | 0.665 | +| 30 | 0.456 | 0.447 | 0.129 | 0.280 | 0.152 | 0.457 | 0.685 | + + +### Perfect Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.531 | 0.483 | - | - | - | - | - | +| 01 | 0.523 | 0.473 | 0.522 | 0.473 | 0.080 | 0.240 | 0.014 | +| 05 | 0.512 | 0.460 | 0.513 | 0.460 | 0.081 | 0.243 | 0.013 | +| 10 | 0.519 | 0.450 | 0.521 | 0.450 | 0.079 | 0.237 | 0.016 | +| 20 | 0.491 | 0.453 | 0.481 | 0.453 | 0.078 | 0.233 | 0.012 | +| 30 | 0.470 | 0.430 | 0.466 | 0.430 | 0.077 | 0.230 | 0.018 | + + +## Student-driver trigger + +### Undefended + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.523 | 0.470 | - | - | - | - | - | +| 01 | 0.513 | 0.450 | 0.509 | 0.450 | 0.079 | 0.237 | 0.016 | +| 05 | 0.482 | 0.460 | 0.342 | 0.400 | 0.100 | 0.300 | 0.237 | +| 10 | 0.482 | 0.453 | 0.156 | 0.300 | 0.149 | 0.447 | 0.629 | +| 20 | 0.472 | 0.443 | 0.160 | 0.300 | 0.159 | 0.477 | 0.676 | +| 30 | 0.480 | 0.457 | 0.169 | 0.327 | 0.158 | 0.473 | 0.676 | + + +### Random Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.460 | 0.437 | - | - | - | - | - | +| 01 | 0.443 | 0.423 | 0.441 | 0.423 | 0.074 | 0.223 | 0.017 | +| 05 | 0.444 | 0.437 | 0.438 | 0.433 | 0.077 | 0.230 | 0.027 | +| 10 | 0.437 | 0.427 | 0.138 | 0.253 | 0.140 | 0.420 | 0.599 | +| 20 | 0.430 | 0.427 | 0.157 | 0.290 | 0.149 | 0.447 | 0.661 | +| 30 | 0.449 | 0.430 | 0.154 | 0.313 | 0.149 | 0.447 | 0.659 | + + +### Perfect Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.536 | 0.460 | - | - | - | - | - | +| 01 | 0.524 | 0.450 | 0.523 | 0.450 | 0.079 | 0.237 | 0.019 | +| 05 | 0.511 | 0.453 | 0.510 | 0.453 | 0.079 | 0.237 | 0.012 | +| 10 | 0.512 | 0.450 | 0.508 | 0.450 | 0.078 | 0.233 | 0.014 | +| 20 | 0.480 | 0.437 | 0.479 | 0.433 | 0.077 | 0.230 | 0.007 | +| 30 | 0.472 | 0.423 | 0.472 | 0.423 | 0.074 | 0.223 | 0.016 | @@ -183,4 +259,82 @@ Generated box parameters: +## Skull trigger + +### Undefended + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.522 | 0.477 | - | - | - | - | - | +| 01 | 0.507 | 0.463 | - | - | 0.501 | 0.517 | 0.833 | +| 05 | 0.529 | 0.450 | - | - | 0.526 | 0.527 | 0.947 | +| 10 | 0.527 | 0.463 | - | - | 0.543 | 0.547 | 0.948 | +| 20 | 0.533 | 0.477 | - | - | 0.554 | 0.553 | 0.925 | +| 30 | 0.544 | 0.490 | - | - | 0.556 | 0.550 | 0.912 | + + +### Random Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.470 | 0.447 | - | - | - | - | - | +| 01 | 0.466 | 0.437 | - | - | 0.444 | 0.473 | 0.840 | +| 05 | 0.483 | 0.433 | - | - | 0.479 | 0.520 | 0.935 | +| 10 | 0.442 | 0.430 | - | - | 0.453 | 0.537 | 0.952 | +| 20 | 0.477 | 0.443 | - | - | 0.478 | 0.513 | 0.925 | +| 30 | 0.514 | 0.460 | - | - | 0.509 | 0.540 | 0.930 | + + +### Perfect Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.539 | 0.483 | - | - | - | - | - | +| 01 | 0.524 | 0.473 | - | - | 0.441 | 0.223 | 0.002 | +| 05 | 0.514 | 0.467 | - | - | 0.434 | 0.223 | 0.000 | +| 10 | 0.531 | 0.460 | - | - | 0.443 | 0.220 | 0.001 | +| 20 | 0.497 | 0.433 | - | - | 0.422 | 0.213 | 0.002 | +| 30 | 0.473 | 0.433 | - | - | 0.397 | 0.207 | 0.001 | + + + + +## Student-driver trigger + +### Undefended + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.508 | 0.460 | - | - | - | - | - | +| 01 | 0.501 | 0.443 | - | - | 0.477 | 0.497 | 0.812 | +| 05 | 0.489 | 0.447 | - | - | 0.487 | 0.540 | 0.927 | +| 10 | 0.506 | 0.453 | - | - | 0.504 | 0.523 | 0.894 | +| 20 | 0.523 | 0.473 | - | - | 0.520 | 0.557 | 0.892 | +| 30 | 0.516 | 0.467 | - | - | 0.520 | 0.543 | 0.897 | + + +### Random Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.453 | 0.417 | - | - | - | - | - | +| 01 | 0.417 | 0.383 | - | - | 0.399 | 0.410 | 0.736 | +| 05 | 0.453 | 0.423 | - | - | 0.449 | 0.500 | 0.876 | +| 10 | 0.443 | 0.420 | - | - | 0.448 | 0.513 | 0.929 | +| 20 | 0.477 | 0.443 | - | - | 0.491 | 0.550 | 0.940 | +| 30 | 0.456 | 0.437 | - | - | 0.480 | 0.537 | 0.919 | + + +### Perfect Filter + +| Poison Percentage | Benign mAP | Benign AP target | Adv mAP - Clean labels | Adv AP target - Clean labels | Adv mAP - Adv labels | Adv AP target - Adv labels | Attack success rate | +| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | +| 00 | 0.491 | 0.440 | - | - | - | - | - | +| 01 | 0.496 | 0.453 | - | - | 0.400 | 0.217 | 0.009 | +| 05 | 0.500 | 0.453 | - | - | 0.412 | 0.217 | 0.015 | +| 10 | 0.493 | 0.450 | - | - | 0.411 | 0.213 | 0.009 | +| 20 | 0.470 | 0.440 | - | - | 0.382 | 0.210 | 0.017 | +| 30 | 0.456 | 0.403 | - | - | 0.384 | 0.193 | 0.009 | + +