The benchmark
The aim of this benchmark is to have a framework that is able to test the performance of the adversarial examples detection methods under the same attack scenarios. This will help researchers to follow-up the up-to-date progress on the domain. Here, we start with the results published in the review paper; Adversarial Example Detection for DNN Models: A Review and Experimental Comparison.
Fig.1 - Average detection rates (%) for detectors against adversarial examples for each dataset
Results
Note: In this website, we only report the detection rate (DR) and the false positive rate (FPR). Other performance results, like TP, TN, FP, and FN, can be accquired from the genenerated CSV file for each detector (visit the gitub repository).
About
Citation
@article{aldahdooh2022adversarial,
title={Adversarial Example Detection for DNN Models: A Review and Experimental Comparison},
author={Ahmed Aldahdooh and Wassim Hamidouche and Sid Ahmed Fezza and Olivier Deforges},
journal={Artificial Intelligence Review},
year={2022},
publisher={Springer}
}
Authors
Your contribution
We are welcoming your contribution to enrich this benchmark either by adding new detectors’ performance evaluation or by including current detectors’ performance with more attacks and with different baseline classifiers. Please 1)Follow the instruction here 2)Contact us by opening an isuue to include your updates to the code and to the results.
Datasets and Attacks
Datasets
| Dataset | Neural Network Model(s) |
|---|---|
| MNIST |
|
| CIFAR-10 |
|
| SVHN |
|
| Tiny-ImageNet |
|
**Models Description
| Model Name | Description |
|---|---|
| MNIST - Model 1 (98.73) | 2 (CONV(32, 3x3)+ReLU) + MaxPool, 2 (CONV(64, 3x3)+ReLU) + MaxPool, Dense (256) + ReLU + Dropout (0.3), Dense (256) + ReLU, Dense(10) + Softmax |
| CIFAR-10 - Model 1 (89.11) | 2(Conv(64, 3x3) + BatchNorm + ReLU) + MaxPool + Dropout(0.1), 2(Conv(128, 3x3) + BatchNorm + ReLU) + MaxPool + Dropout(0.2), 2(Conv(256, 3x3) + BatchNorm + ReLU) + MaxPool + Dropout(0.3), Conv(512, 3x3) + BatchNorm + ReLU + MaxPool + Dropout(0.4), Dense (512) , Dense(10) + Softmax |
| SVHN - Model 1 (94.98) | 2 (CONV(32, 3x3)+ReLU)+MaxPool, 2 (CONV(64, 3x3)+ReLU)+MaxPool, Dense (512) + ReLU + Dropout (0.3), Dense (128) + ReLU, Dense(10) + Softmax |
| Tiny-ImageNet - Model 1 (64.48) | DenseNet201 |
Attacks
| Scenario | Attack | Norm | (Un)Targeted | Parameters |
|---|---|---|---|---|
| White-box | FGSM | L-inf | U | eps = (8, 16, 32, 64, 80, 128)/255 eps_step = 0.01 |
| BIM | L-inf | U | eps = (8, 16, 32, 64, 80, 128)/255 eps_step = 0.01 iter = eps*255*1.25 |
|
| PGD | L-1 | U | eps = 5, 10, 15, 20, 25 eps_step = 4 iter = 100 |
|
| PGD | L-2 | U | eps = 0.25, 0.3125, 0.5, 1, 1.5, 2 eps_step = 0.01 iter = eps*255*1.25 |
|
| PGD | L-inf | U | eps = (8, 16, 32, 64, 80, 128)/255 eps_step = 0.01 iter = 100 |
|
| CW | L-inf | U | Confidence = 0 iter=200 |
|
| CW-HCA | L-2 | U | eps = (8, 16, 32, 64, 80, 128)/255 tol = 1 num_steps = 100 step_size = 1/255 random_start = False |
|
| DF | L-2 | U | eps = 1e-6 iter = 100 |
|
| Black-box | Square Attack | L-inf | U | eps = 0.3 (mnist), 0.125 (cifar, svhn, tiny) iter = 200 |
| HopSkipJump | L-2 | U | max_eval = 100 init_eval = 10 iter = 40 |
|
| Spatial Transformation | - | U | rotation = 60 (mnist, svhn), 30 (cifar, tiny) translation = 10 (mnist, svhn), 8 (cifar, tiny) |
|
| ZOO | L-2 | U | confidence=0.1 learning_rate=0.01 max_iter=100 |
Baseline classifiers’ accuracies to the normal training data and the tested attacked data.
| Attack | Datasets | ||||
| MNIST | CIFAR | SVHN | Tiny ImageNet | ||
| Clean Data | - | 98.73 | 89.11 | 94.98 | 64.48 |
| White box | FGSM(8) | - | 14.45 | 15.06 | 12.14 |
| FGSM(16) | - | 13.66 | 5.91 | 8.11 | |
| FGSM(32) | 76.97 | 11.25 | - | - | |
| FGSM(64) | 13.76 | - | - | - | |
| FGSM(80) | 8.64 | - | - | - | |
| BIM(8) | - | 1.9 | 1.25 | 0.3 | |
| BIM(16) | - | 0.61 | 0 | 0 | |
| BIM(32) | 21.84 | - | - | - | |
| BIM(64) | 0 | - | - | - | |
| BIM(80) | 0 | - | - | - | |
| PGD-L1(5) | - | 43.45 | - | - | |
| PGD-L1(10) | 65.95 | 10.56 | - | - | |
| PGD-L1(15) | 25.74 | 5.27 | 17.59 | 44.7 | |
| PGD-L1(20) | 4.95 | - | 7.97 | 31.34 | |
| PGD-L1(25) | - | - | 3.73 | 21.97 | |
| PGD-L2(0.25) | - | 13.97 | - | - | |
| PGD-L2(0.3125) | - | 8.19 | 35.5 | - | |
| PGD-L2(0.5) | - | 5.52 | 13.26 | 8.46 | |
| PGD-L2(1) | 70.54 | - | 0.8 | 1.34 | |
| PGD-L2(1.5) | 18.89 | - | - | - | |
| PGD-L2(2) | 0.79 | - | - | - | |
| PGD-L∞(8) | - | 0.78 | 0.8 | 0.02 | |
| PGD-L∞(16) | - | 0.28 | 0 | 0 | |
| PGD-L∞(32) | 19.05 | - | - | - | |
| PGD-L∞(64) | 0 | - | - | - | |
| CW-L∞ | 38.98 | 20.95 | 23.73 | 16.64 | |
| CW-HCA(8) | - | 46.51 | 47.06 | 39.47 | |
| CW-HCA(16) | - | 18.96 | 29.06 | 17.51 | |
| CW-HCA(80) | 43.36 | - | - | - | |
| CW-HCA(128) | 8.64 | - | - | - | |
| DF | 4.96 | 4.8 | 6.12 | 0.52 | |
| Black box | SA | 4.66 | 0 | 0.7 | 0.22 |
| HopSkipJump | 0 | 0 | 0 | 0 | |
| ST | 22.04 | 52.57 | 17.0 | 52.28 | |
Acknowledgment
The project is funded by both Region Bretagne (Brittany region), France, and direction generale de l’armement (DGA).
