Research map / Adversarial robustness and security / Certified and empirical robustness
Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 30 adversarial defenses (purification, adversarial training, certified smoothing) against 14 attacks on 9 no-reference IQA metrics, in adaptive and non-adaptive settings, with a large subjective study.
Problem
Neural no-reference image quality assessment (IQA) metrics are vulnerable to adversarial attacks that inflate scores, and there is no systematic benchmark of defenses for IQA metrics.
Approach
First comprehensive benchmark of IQA-specific defenses; adapts classification-oriented defenses (randomized smoothing variants, median smoothing for regression) to IQA scores, evaluates attacks at weak/medium/strong budgets, and introduces robustness and purification-discrepancy metrics plus a subjective perceptual-quality study.
Key results
- DiffJPEG and JPEG-style compression are among the strongest purification defenses (e.g., DiffJPEG SROCC_adv 0.608 vs 0.413 with no defense in non-adaptive setting).
- Certified methods give the best correlation/robustness combination but are computationally expensive; DP takes about 82 s per image set versus 0.05 ms for Flip.
- In the subjective study (60,000+ responses), Real-ESRGAN achieves perceptual quality surpassing the undefended case, while objective PSNR/SSIM can mislead rankings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DiffJPEG compression | IQA metrics under adversarial attack · Purification benchmark | SROCC_adv: 0.413/0.413 -> 0.608/0.549 | Helped |
| Apply JPEG compression | IQA metrics under non-adaptive attack · Purification benchmark | SROCC_adv: 0.413 -> 0.605 | Helped |
| Apply DiffPure purification | IQA metrics under adaptive attack · Purification benchmark | D_score^(D): 66.68 -> 22.96 | Helped |
| Apply DiffPure purification | IQA metrics · Purification benchmark | SROCC_adv: 0.413/0.413 -> 0.485/0.470 | Hurt |
| Apply DISCO purification | IQA metrics under non-adaptive attack · Purification benchmark | D_score^(D): 66.68 -> 3.51 | Helped |
| Apply DISCO purification | IQA metrics under adaptive attack · Purification benchmark | R_score: 0.56 -> 0.50 | Hurt |
| Use adversarial training instead of purification | IQA metrics · Adaptive attack evaluation | SROCC_adv | Helped |
| Use adversarial training instead of purification | IQA metrics · Adaptive attack evaluation | D_score^(D) | Hurt |
| Train with APGD-SSIM at perturbation budget 2 | Koncept and Linearity · Adaptive attacks, KonIQ-10k | SROCC_adv: 0.413 -> 0.763 | Helped |
| Increase APGD-SSIM training budget from 2 to 4 (ablation) | Koncept and Linearity · Adaptive attacks, KonIQ-10k | SROCC_adv: 0.763 -> 0.625 | Hurt |
| Increase APGD-SSIM training budget from 2 to 8 (ablation) | Koncept and Linearity · Adaptive attacks, KonIQ-10k | SROCC_adv: 0.763 -> 0.582 | Hurt |
| Increase APGD-LPIPS training budget from 2 to 4 (ablation) | Koncept and Linearity · Adaptive attacks, KonIQ-10k | SROCC_adv: 0.651 -> 0.576 | Hurt |
Surprising findings
- Real-ESRGAN ranks top in subjective perceptual quality despite lower objective metric scores.
- DISCO ranks top-3 by robustness in the non-adaptive case but drops sharply under adaptive attacks.
Limitations
- Attacks with multiple or evolving parameters (e.g., boundary attacks) are not handled by the fixed-parameter framework.
- Transferability of attacks across defenses is not evaluated.
- Evaluation uses small image subsets (50 images for black-box and certified, 10 for certified attacks), and ranking methodology is simple.
Experimental setup
Datasets: KonIQ-10k KADID-10k NIPS 2017 Adversarial Learning Development Set AGIQA-3K DIV2K_valid_HR
Benchmarks: KonIQ-10k KADID-10k AGIQA-3K NIPS 2017
Metrics: SROCC_adv SROCC_clear R_score D_score certified radius abstention rate PSNR SSIM MSE L_inf
Baselines: W/o defense Gaussian blur JPEG DiffJPEG DiffPure DISCO Real-ESRGAN MPRNet Classic adversarial training Gradient Norm optimization
Builds on: randomized smoothing denoised randomized smoothing DensePure DiffPure DISCO APGD adversarial training
Closest papers by meaning
- Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization (CVPR 2024 · similarity 0.91)
- Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks (AAAI 2024 · similarity 0.91)
- Backdoor Attacks Against No-Reference Image Quality Assessment Models via a Scalable Trigger (AAAI 2025 · similarity 0.88)
- Adaptive Feature Selection for No-Reference Image Quality Assessment by Mitigating Semantic Noise Sensitivity (ICML 2024 · similarity 0.87)
- IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image- and Video-Quality Metrics (ICML 2024 · similarity 0.86)
- Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.