Research map / Adversarial robustness and security / Attacks on text and graphs

TETRIS: Towards Exploring the Robustness of Interactive Segmentation

AAAI 2024 · Adversarial robustness and security › Attacks on text and graphs · approach: Latency and driving-perception attacks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TETRIS, a high-resolution interactive segmentation benchmark with real-user clicks, and an adversarial click-optimization protocol that measures model robustness via BIoU min/max trajectories.

Problem

Interactive segmentation evaluation relies on mostly low-resolution datasets and on IoU, which is insensitive to boundary quality, and does not account for how sensitive models are to the placement of user clicks.

Approach

Optimize adversarial click positions (greedy, about 10 forward-backward steps per click) to find the worst and best click placements, and measure the spread of IoU/Boundary-IoU across them as a robustness metric, on a new high-resolution manually labeled dataset.

Key results

• CFR-ICL (ViT-H) achieves the best BIoU min/max scores on TETRIS (85.91 min, 90.71 max), while SAM-HQ ViT-B has the largest BIoU spread on TETRIS (16.06).
• SimpleClick ViT-H attains the best BIoU-D robustness on several datasets, with TETRIS BIoU-D of 3.25 among the lowest.
• Adversarial click optimization runs about 10 forward-backward passes per click (about 10 s per input on a V100 at 400x400), an order of magnitude more costly than plain evaluation.

Surprising findings

• Small perturbations of click coordinates and local optimization substantially change model performance, showing high sensitivity to click placement.
• The spread of quality after a second negative click is slightly smaller than after a second positive click, even though the distance from the baseline click is greater.

Limitations

• Greedy, fixed-iteration click optimization does not guarantee global extremes and can yield trajectories that violate expected ordering.
• Adversarial evaluation is an order of magnitude more expensive than standard evaluation without optimization.
• TETRIS-things images come from a single source (Unsplash), which may introduce biases; annotations are subjective manual labels.

Experimental setup

Datasets: GrabCut Berkeley DAVIS COCO-MVal TETRIS

Benchmarks: GrabCut Berkeley DAVIS COCO-MVal TETRIS

Metrics: BIoU (AuC@10) IoU BIoU spread (D)

Baselines: RITM SimpleClick CDNet GPCIS CFR-ICL SAM SAM-HQ MobileSAM

Builds on: Boundary IoU RITM SimpleClick

Closest papers by meaning

• RClicks: Realistic Click Simulation for Benchmarking Interactive Segmentation (NeurIPS 2024 · similarity 0.92)
• NTClick: Achieving Precise Interactive Segmentation With Noise-tolerant Clicks (CVPR 2025 · similarity 0.88)
• AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.87)
• Order-aware Interactive Segmentation (ICLR 2025 · similarity 0.87)
• Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.87)
• MFP: Making Full Use of Probability Maps for Interactive Image Segmentation (CVPR 2024 · similarity 0.86)