Research map / Adversarial robustness and security / Certified and empirical robustness
RP-PGD: Boosting Segmentation Robustness with a Region-and-Prototype Based Adversarial Attack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RP-PGD, a multi-step attack for semantic segmentation that splits pixels into True/Boundary/False regions in a spatial-temporal way plus a prototype-based feature-space loss, and uses it as the underlying attack for adversarial training to improve segmentation robustness.
Problem
Existing segmentation attacks are weak underlying attacks for adversarial training; they fail to yield robustness against strong attacks such as PGD with 100 iterations, giving a false sense of robustness.
Approach
Partition pixels over attack iterations into True, Boundary, and False regions, weight them with iteration-dependent losses to focus on True and Boundary pixels, and add a prototype loss that maximizes cosine similarity between class prototypes computed from True-region features, then use the resulting hybrid attack for adversarial training.
Key results
- On Pascal VOC, RP-PGD3-AT reaches 18.69 MIoU under PGD100 on PSPNet, about 4.2 points above SegPGD3-AT and 5.05 above CosPGD3-AT.
- RP-PGD3-AT achieves comparable or better robustness than SegPGD7-AT while requiring fewer than half the attack iterations.
- RP-PGD-AT achieves the best transfer-based black-box robustness on Pascal VOC and Cityscapes in the reported DeepLabv3 experiment (e.g., 15.10 vs 17.79 for RP-PGD3-AT vs SegPGD3-AT's 13.34 under SegPGD100... see Table 6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RP-PGD7 rather than SegPGD7 for adversarial training | PSPNet · Pascal VOC | MIoU under PGD100: 19.23 -> 23.14 | Helped |
| Use RP-PGD7 rather than SegPGD7 for adversarial training | PSPNet · Cityscapes | MIoU under PGD100: 23.13 -> 26.04 | Helped |
| Use RP-PGD7 rather than SegPGD7 for adversarial training | DeepLabv3 · Pascal VOC | MIoU under PGD100: 23.09 -> 28.10 | Helped |
| Use RP-PGD7 rather than SegPGD7 for adversarial training | DeepLabv3 · Cityscapes | MIoU under PGD100: 25.31 -> 29.87 | Helped |
| Add the Boundary Region to True Region (ablation) | Region-based attack · Pascal VOC | MIoU: 2.65 -> 2.97; 0.65 -> 0.54; 2.97 -> 18.40; 12.84 -> 31.71; 3.26 -> 16.68; 13.38 -> 29.62 | Helped |
| Add the False Region to the True and Boundary Regions (ablation) | Region-based attack · Pascal VOC | MIoU: 2.97 -> 0.61; 12.84 -> 9.83; 3.02 -> 0.51; 13.38 -> 10.35 | Mixed |
| Add prototype attack to the True, Boundary, and False Regions (ablation) | RP-PGD · Pascal VOC | MIoU: 0.61 -> 0.48; 9.83 -> 7.69; 0.51 -> 0.34; 10.35 -> 8.30 | Helped |
| Extract prototypes from True Region only (ablation) | Prototype-based attack · Pascal VOC | MIoU: 10.75 -> 8.74; 28.52 -> 26.17; 9.56 -> 7.57; 26.41 -> 25.09 | Helped |
| Use dynamic region weights from Eq. (4) instead of fixed weights (ablation) | Region-based attack · Pascal VOC | MIoU: 3.60 -> 0.61; 14.31 -> 9.83; 3.47 -> 0.51; 13.92 -> 10.35 | Helped |
Surprising findings
- Proximal attacks can be strong with smaller perturbation but converge slowly (around 500 iterations), making them unsuitable for adversarial training.
- Less than 1% of initially mispredicted pixels were restored to correct predictions over 100 PGD iterations, motivating the False Region treatment.
Limitations
- Proximal-style and other baselines are noted as slower per iteration; evaluation limited to PSPNet and DeepLabv3 with ResNet-50 backbone.
- Hyperparameters (beta=0.75, weight schedule) were selected empirically on Pascal VOC.
Experimental setup
Datasets: Pascal VOC Cityscapes ADE20K
Benchmarks: Pascal VOC Cityscapes
Metrics: MIoU MisRatio
Baselines: PGD SegPGD CosPGD Proximal DDC-AT
Builds on: PGD SegPGD DDC-AT CosPGD DAG
Closest papers by meaning
- CosPGD: an efficient white-box adversarial attack for pixel-wise prediction tasks (ICML 2024 · similarity 0.87)
- Stop Walking in Circles! Bailing Out Early in Projected Gradient Descent (CVPR 2025 · similarity 0.86)
- Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2 (NeurIPS 2025 · similarity 0.84)
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.84)
- TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases (NeurIPS 2024 · similarity 0.84)
- AAKR: Adversarial Attack-based Knowledge Retention for Continual Semantic Segmentation (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.