Research map / Adversarial robustness and security / Attacks on text and graphs
Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks the adversarial robustness of 15 no-reference image- and video-quality metrics against nine adapted attacks (FGSM-family, universal perturbations, perceptually-aware attacks) on six public datasets, and releases a public leaderboard and code.
Problem
Existing quality-metric benchmarks measure correlation with subjective quality and runtime but not adversarial robustness, even though deep NR IQA/VQA metrics can be manipulated to raise scores without improving visual quality.
Approach
Adapts computer-vision adversarial attacks (FGSM, I-FGSM, MI-FGSM, AMI-FGSM, universal perturbations, Korhonen et al., MADC) to score-increasing attacks on NR quality metrics, and quantifies robustness with gain, robustness-score, Wasserstein and energy-distance measures after mapping all metrics to a common domain via neural optimal transport.
Key results
- MANIQA shows the lowest absolute gain (0.104) and relative gain (0.078) among the 15 metrics, while FPR is the least robust with the lowest robustness score (-0.229).
- Iterative attacks (I-FGSM, MI-FGSM, AMI-FGSM) are more efficient against all metrics than FGSM or UAP-based attacks.
- CLIP-IQA shows the best robustness to most iterative attacks, followed by RANK-IQA and MDTVSFA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the tested adversarial attacks | FPR · All tested attacks and test datasets | R_score: R_score: -0.229{ (-0.234, -0.224)} | Hurt |
| Apply the tested adversarial attacks | MANIQA · All tested attacks and test datasets | Abs. gain: 0.104 { (0.101, 0.107)} | No clear effect |
| Use iterative attacks rather than UAP-based attacks | The 15 evaluated NR quality metrics · All test datasets | Attack efficiency | Helped |
| Allow more SSIM loss during attacks | The 15 evaluated NR quality metrics · All attacks | Robustness score | Hurt |
| Allow more SSIM loss during attacks | HYPER-IQA · All attacks | Robustness score | No clear effect |
| Allow more SSIM loss during attacks | PAQ-2-PIQ, VSFA and FPR · All attacks | Robustness score | Hurt |
| Use Optimized-UAP | MANIQA · All test datasets | E_score: -0.390 | Mixed |
| Use Generative-UAP | MANIQA · All test datasets | E_score: -0.174 | Helped |
| Use FGSM | CLIP-IQA · All test datasets | E_score: 0.398 | Mixed |
| Use iterative attacks | CLIP-IQA · All test datasets | Robustness score: I-FGSM 0.836; MI-FGSM 0.821; AMI-FGSM 0.819 | Helped |
| Use MADC and Korhonen attacks | RANK-IQA · All test datasets | Robustness score: MADC 0.383; Korhonen et al. 0.763 | Helped |
| Apply adversarial attacks | META-IQA · All tested attacks and test datasets | R_score: 1.168 { (1.161, 1.176)} | Helped |
Surprising findings
- Robustness to UAP-based attacks and to iterative attacks differs substantially across metrics, so a metric robust to one attack family is not necessarily robust to the other.
- Robust metrics tend to use input cropping or resizing, and META-IQA is robust without input transformation thanks to a small backbone.
Limitations
- Only score-increasing attacks are studied; score-decreasing attacks are excluded.
- Evaluation uses only six public datasets, with just one video dataset in the test set (Derf's collection).
- Only no-reference metrics are benchmarked; full-reference metrics are excluded.
Experimental setup
Datasets: COCO Pascal VOC Vimeo-90k NIPS 2017 Adversarial Learning Development Set Derf's collection
Benchmarks: NIPS 2017 Adversarial Learning Development Set Derf's collection Vimeo-90k test set
Metrics: absolute gain relative gain robustness score Wasserstein score energy distance score SSIM
Baselines: RankIQA MetaIQA WSP CLIP-IQA PAQ-2-PIQ HyperIQA MANIQA TReS
Builds on: FGSM I-FGSM MI-FGSM MADC NIMA
Closest papers by meaning
- Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics (ICML 2025 · similarity 0.91)
- IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image- and Video-Quality Metrics (ICML 2024 · similarity 0.90)
- Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks (CVPR 2025 · similarity 0.87)
- GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models (NeurIPS 2024 · similarity 0.87)
- Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization (CVPR 2024 · similarity 0.87)
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.