Research map / Adversarial robustness and security / Certified and empirical robustness
ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
Pierre Stock, Moustapha Cisse
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| count a prediction as correct when at least 4 of 5 turkers agree (rectified error) instead of ground-truth label only | ResNet-101 · ImageNet val | top-1 error (%): 22.69 -> 9.47 | Improved |
| human rectification of top-1 error | ResNet-18 · ImageNet val | top-1 error (%): 31.39 -> 17.93 | Improved |
| human rectification of top-1 error | DenseNet-161 · ImageNet val | top-1 error (%): 22.85 -> 10.87 | Improved |
| human rectification of top-1 error | DenseNet-121 · ImageNet val | top-1 error (%): 25.53 -> 14.37 | Improved |
| human rectification of top-5 error, showing top-1-misclassified images with top-5 predictions | ResNet-101 · ImageNet val | top-5 error (%): 6.44 -> 1.94 | Improved |
| show top-5-misclassified images instead of top-1-misclassified ones for top-5 rectification | ResNet-101 · ImageNet val | rectified top-5 error (%): 1.94 -> 1.37 | Improved |
| show LIME explanation (top 8 super-pixels) instead of the whole adversarial image to turkers | ResNet-101 under IFGSM · ImageNet val (20 classes, adversarial) | rectified top-1 error (%): 76.78 -> 70.68 | Improved |
| show LIME explanation instead of whole image | ResNet-101 under IFGSM · ImageNet val (20 classes, adversarial) | turker agreement (%): 22.01% to 30.80% | Improved |
| IFGSM adversarial attack (eps 0.001, 10 iterations) | ResNet-101 · ImageNet val | accuracy (%): 77.31% down to 7.96% | Hurt |
| use adversarial selection of prototypes and criticisms instead of MMD-critic | model criticism · Prototype/criticism assignment study | success rate (%): 50.64 -> 57.06 | Improved |
| use adversarial selection of prototypes instead of MMD-critic (prototypes only) | model criticism · Prototype/criticism assignment study | success rate (%): 53.18 -> 55.02 | Improved |
| add criticisms to prototypes (3 plus 3 instead of 6 prototypes), adversarial selection (ablation) | adversarial model criticism · Prototype/criticism assignment study | success rate (%): 55.02 -> 57.06 | Improved |
Closest papers by meaning
- With Friends Like These, Who Needs Adversaries? (NeurIPS 2018 · similarity 0.87)
- Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models (ECCV 2018 · similarity 0.87)
- ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy (ICML 2024 · similarity 0.87)
- Interpreting Adversarially Trained Convolutional Neural Networks (ICML 2019 · similarity 0.87)
- Analyzing Filters Toward Efficient ConvNet (CVPR 2018 · similarity 0.87)
- Concise Explanations of Neural Networks using Adversarial Training (ICML 2020 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
