Research map / Adversarial robustness and security / Black-box and transfer attacks
Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adding Gaussian noise to intermediate hidden features at inference time as a lightweight, training-free defense against score-based and decision-based query attacks, with theoretical analysis linking robustness to gradient-norm ratios.
Problem
Query-based black-box adversarial attacks (score-based and decision-based) can fool image classifiers using only model outputs; the paper seeks a lightweight defense that perplexes such attackers without adversarial training or large accuracy loss.
Approach
Inject independent Gaussian noise into hidden features at a chosen intermediate layer at every query, so the attacker receives stochastic feedback that misleads gradient estimates and random-search or best-radius decisions; the theory shows robustness depends on the ratio of feature-gradient norm to input-gradient norm, which varies during the attack, and that feature-level noise can beat input-le
Key results
- On ImageNet VGG19 with 10000 Square queries, randomized feature defense reaches 22.2% robust accuracy versus 17.8% for randomized input defense at about 2% clean-accuracy drop.
- On CIFAR10 against decision-based attacks, feature defense on VGG19 raises RayS robustness to 15.4% vs 8.1% for input defense, while input defense is better against SignFlip (86.0 vs 76.5).
- Combining the defense with adversarial training on CIFAR10 ResNet20 raises Square robustness from 32.5% (AT alone) to 77.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace input noise with hidden-feature noise at a similar clean-accuracy drop | VGG19 · ImageNet, Square, 10000 queries | robust accuracy: 17.8 -> 22.2 | Helped |
| Replace input noise with hidden-feature noise at a similar clean-accuracy drop | DeiT · ImageNet, Square, 10000 queries | robust accuracy: 67.2 -> 69.1 | Helped |
| Replace input noise with hidden-feature noise at a similar clean-accuracy drop | VGG19 · CIFAR10, Square, 10000 queries | robust accuracy: 27.1 -> 58.4 | Helped |
| Replace input noise with hidden-feature noise | ResNet50 · CIFAR10, RayS | robust accuracy: 12.0 -> 14.4 | Helped |
| Replace input noise with hidden-feature noise | ResNet50 · CIFAR10, SignFlip | robust accuracy: 85.5 -> 82.5 | Hurt |
| Combine feature defense with adversarial training | ResNet20 · CIFAR10, Square, 1000 queries | robust accuracy: 32.5 -> 77.8 | Helped |
| Replace input noise with hidden-feature noise under EOT with M=5 | VGG19 · CIFAR10, Square, QC=1000 | robust accuracy: 24.2 -> 53.0 | Helped |
| Randomize layer 12 only instead of layer 4 only (ablation) | VGG19 feature defense · CIFAR10, Square | robustness: 52.5 -> 63.0 | Helped |
| Randomize layer 1 only instead of layer 11 only (ablation) | ViT feature defense · CIFAR10, Square | robustness: 48.3 -> 77.3 | Helped |
| Increase attack perturbation magnitude for NES from 0.1 to 0.2 (ablation) | VGG19 feature defense · CIFAR10, NES | robustness: 89.6 -> 91.4 | Mixed |
| Increase attack perturbation magnitude across tested values (ablation) | Randomized defenses · CIFAR10, Square, NES, and SignHunt | robustness | Mixed |
Surprising findings
- AAA, which directly optimizes logits to confuse attackers, gives essentially no robustness gain over the base model against decision-based attacks.
- Randomizing deeper hidden layers yields higher gradient-norm ratios and better robustness than input noise at matched clean accuracy.
Limitations
- Trade-off: robustness gains come with a 1-4% clean-accuracy drop, and the defense is weaker against some attacks (e.g., SignFlip on feature vs input defense).
- Theoretical analysis relies on linearization and assumes the randomized mean equals the original model output; decision-based distribution has no closed form.
- Does not cover transfer-based black-box attacks, which the authors defer to future work.
Experimental setup
Datasets: ImageNet CIFAR10
Benchmarks: ImageNet CIFAR10
Metrics: robust accuracy under attack clean accuracy
Baselines: Random input noise defense (RND-style) AAA Base (no defense) Adversarial training
Builds on: Random Noise Defense (RND) Small Noise Defense (SND) Square attack NES RayS
Closest papers by meaning
- Random Noise Defense Against Query-Based Black-Box Attacks (NeurIPS 2021 · similarity 0.92)
- Mind the Gap: Detecting Black-box Adversarial Attacks in the Making through Query Update Analysis (CVPR 2025 · similarity 0.91)
- Black-box Adversarial Attacks with Limited Queries and Information (ICML 2018 · similarity 0.91)
- Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks (NeurIPS 2021 · similarity 0.91)
- Adversarial Robustness via Deformable Convolution with Stochasticity (ICML 2025 · similarity 0.90)
- Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Prior (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.