Research map / Adversarial robustness and security / Black-box and transfer attacks
Mind the Gap: Detecting Black-box Adversarial Attacks in the Making through Query Update Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GWAD, a detector for query-based black-box adversarial attacks in progress that classifies histograms of "Delta Similarity" (cosine similarity between consecutive query updates) with a small MLP, plus a stateful screener variant (GWAD+).
Problem
Detecting query-based black-box adversarial attacks while they are being generated (in the making), robust to adaptive attacks that evade input-space stateful defenses.
Approach
Shifts detection from input-space similarity to update-space patterns: zeroth-order gradient estimation makes consecutive query updates produce characteristic cosine-similarity (Delta Similarity) distributions absent from benign queries; a histogram of these values feeds a classifier.
Key results
- GWAD detects over 99.42% of attack queries across six query-based attacks (HSJA, NES, SimBA, Sign-Opt, Sign-Flip, BA) on ImageNet and CIFAR-10 with 100K queries.
- Against OARS-NES, Blacklight and PIHA reach 97.5% and 84.5% ASR at sigma^2=0.05 while GWAD keeps 100% detection; GWAD FPR is 0.00% on CIFAR-10, Tiny-ImageNet and FLIR ADAS.
- GWAD is evaded by benign injection at high ratios (detection drops after rb>150%), but GWAD+ with a stateful screener keeps over 99.2% detection at rb=1000%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GWAD instead of Blacklight or PIHA for attack detection | CIFAR-10 queries against MobileNet-V2 · CIFAR-10 | detection rate: 23.96% / 38.08% -> 99.56% | Helped |
| Use GWAD for benign-query monitoring | ImageNet queries · ImageNet | false positive rate: 00.03% | Helped |
| Use GWAD instead of Blacklight or PIHA | high-similarity Hollywood Heads sequences · Hollywood Heads high similarity | false positive rate: 25.47% / 26.19% -> 17.99% | Helped |
| Use GWAD instead of Blacklight or PIHA against OARS-NES | OARS-NES · OARS-NES, σ²=0.050 | detection rate: 0.0% / 31.7% -> 100.0% | Helped |
| Use GWAD against OARS adaptive attacks | OARS-NES, OARS-HSJA, OARS-QEBA, and OARS-Square · OARS adaptive attacks | attack success rate: 00%, 00%, 00%, 05% | Helped |
| Inject benign queries between attack queries | GWAD · Irregular batch attacks | attack detection accuracy: perfect attack detection performance until rb ≤150%; detection remains up to 80% for rb ≃200% | Hurt |
| Add the Screener pre-processing step | GWAD · Irregular batch, rb = 1000%, ImageNet | detection rate: 00.00% -> 99.95% | Helped |
| Add the Screener pre-processing step | GWAD · Irregular batch, rb = 1000%, CIFAR-10 | detection rate: 00.12% -> 99.24% | Helped |
Surprising findings
- GWAD achieves 0.00% false positives on CIFAR-10 and Tiny-ImageNet but still has 17.99% FPR on highly similar Hollywood Heads sequences, indicating high-similarity benign input remains challenging.
- SimBA attack recognition on CIFAR-10 is only 5.60% even though detection is 100%, showing attack classification can fail while detection succeeds.
Limitations
- Can be evaded by excessive benign query injection (detection drops beyond rb≈150% without the screener).
- High false positive rate on highly similar single-class benign sequences (17.99% on Hollywood Heads high-similarity subset).
- Attack recognition confuses attacks with similar DS distributions (e.g., Sign-Opt and SimBA).
Experimental setup
Datasets: CIFAR-10 ImageNet Tiny-ImageNet Hollywood Heads FLIR ADAS BIRDSAI
Benchmarks: CIFAR-10 ImageNet Tiny-ImageNet Hollywood Heads FLIR ADAS BIRDSAI
Metrics: detection rate attack recognition accuracy false positive rate attack success rate
Baselines: Blacklight PIHA Stateful Detection
Builds on: Stateful Detection Blacklight PIHA OARS
Closest papers by meaning
- Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks (ICLR 2024 · similarity 0.91)
- Random Noise Defense Against Query-Based Black-Box Attacks (NeurIPS 2021 · similarity 0.89)
- BayesOpt Adversarial Attack (ICLR 2020 · similarity 0.88)
- Detecting Adversarial Data Using Perturbation Forgery (CVPR 2025 · similarity 0.88)
- BRUSLEATTACK: A QUERY-EFFICIENT SCORE- BASED BLACK-BOX SPARSE ADVERSARIAL ATTACK (ICLR 2024 · similarity 0.88)
- Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Prior (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.