Research map / Detection and segmentation / Open-set and few-shot detection
RClicks: Realistic Click Simulation for Benchmarking Interactive Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects 475K real-user clicks via crowdsourcing, trains a SegNeXt-based clickability model to sample realistic clicks, and uses it to build the RClicks benchmark for interactive segmentation robustness.
Problem
Standard interactive segmentation evaluation simulates user clicks with a center-of-largest-error heuristic that does not reflect real user click patterns, so reported performance may overestimate real-world annotation quality and robustness.
Approach
Model user clicks as a probability density (clickability map) conditioned on image, target mask and error mask, trained on crowdsourced real clicks; sample clicks from grouped probability-mass intervals (clicking groups) to evaluate both average quality and robustness across click patterns.
Key results
- Baseline click strategy underestimates real-world annotation time by 5% up to 29% (ΔSB).
- Robustness across clicking groups varies annotation time from 3% up to 79% (ΔGR); no method is best on both performance and robustness on all datasets.
- Clickability model achieves KS 0.55, PL1 0.40, WD 0.08 on TETRIS validation, outperforming uniform, distance transform and saliency baselines; it runs at 23 ms per image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use realistic clickability-model clicks instead of baseline-strategy clicks | Interactive segmentation methods · RClicks datasets | NoC20@90 annotation time: 5\% up to 29\% | Hurt |
| Sample clicks from different clickability groups rather than a single click pattern | Interactive segmentation methods · RClicks datasets | NoC20@90 annotation time across groups: 3\% up to 79\% | Mixed |
| Train the clickability model using Object CutOut rather than Text Description display-mode clicks (ablation) | Click collection for TETRIS first rounds · TETRIS first round clicks | PL1: 0.258 -> 0.242 | Helped |
| Train the clickability model using Object CutOut rather than Text Description display-mode clicks (ablation) | Click collection for TETRIS first rounds · TETRIS first round clicks | KS: 0.37 -> 0.58 | Helped |
| Train the clickability model using Object CutOut rather than Text Description display-mode clicks (ablation) | Click collection for TETRIS first rounds · TETRIS first round clicks | WD: 0.051 -> 0.042 | Helped |
| Use σ=5 for clickability-map construction instead of σ=0.01 (ablation) | Clickability model · TETRIS validation | KS: 0.48 -> 0.55 | Helped |
| Use σ=5 for clickability-map construction instead of σ=0.25 (ablation) | Clickability model · TETRIS validation | KS: 0.46 -> 0.55 | Helped |
| Use σ=5 for clickability-map construction instead of σ=0.5 (ablation) | Clickability model · TETRIS validation | KS: 0.49 -> 0.55 | Helped |
| Use σ=5 for clickability-map construction instead of σ=1 (ablation) | Clickability model · TETRIS validation | KS: 0.51 -> 0.55 | Helped |
| Use σ=5 for clickability-map construction instead of σ=2 (ablation) | Clickability model · TETRIS validation | KS: 0.53 -> 0.55 | Helped |
| Use σ=5 for clickability-map construction instead of σ=10 (ablation) | Clickability model · TETRIS validation | KS: 0.51 -> 0.55 | Helped |
| Use σ=5 for clickability-map construction instead of σ=20 (ablation) | Clickability model · TETRIS validation | KS: 0.51 -> 0.55 | Helped |
Surprising findings
- Baseline strategy clicks mostly outperform clicks from the realistic groups, so standard benchmarks overestimate real-world performance.
- High-performing methods such as SimpleClick ViT-H and CFR-ICL are less robust than SAM-like methods on some datasets.
Limitations
- Clickability model trained only on the TETRIS subset; may not generalize to specialized domains such as medical or satellite images.
- Click data comes from one crowdsourcing platform and may reflect platform-specific user biases.
- Only point clicks are considered; scribbles, contours, boxes and voice inputs are not evaluated.
Experimental setup
Datasets: GrabCut Berkeley DAVIS COCO-MVal TETRIS SBD COCO+LVIS SA-1B SA-V
Benchmarks: RClicks GrabCut Berkeley DAVIS COCO-MVal TETRIS
Metrics: NoC@90 (NoC20@90) NoF20@90 IoU-AuC Sample mean and std ΔSB ΔGR ΔHH NSR KS PL1
Baselines: uniform distribution distance transform saliency map (TranSalNet) GPCIS CDNet RITM AdaptClick SimpleClick CFR-ICL MobileSAM SAM SAM-HQ
Builds on: SegNeXt Conv1S RITM SimpleClick TranSalNet
Closest papers by meaning
- TETRIS: Towards Exploring the Robustness of Interactive Segmentation (AAAI 2024 · similarity 0.92)
- NTClick: Achieving Precise Interactive Segmentation With Noise-tolerant Clicks (CVPR 2025 · similarity 0.89)
- AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.87)
- Order-aware Interactive Segmentation (ICLR 2025 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
- MFP: Making Full Use of Probability Maps for Interactive Image Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.