Research map / Label-efficient and robust learning / Recognition and long-tail
Plug-and-Play PPO: An Adaptive Point Prompt Optimizer Making SAM Greater
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PPO, a plug-and-play DRL-based optimizer that refines SAM point prompts on a dual-space (DINOv2 feature and physical distance) heterogeneous graph without per-task training.
Problem
SAM's class-agnostic segmentation depends heavily on point prompt quality, and automatically generated point prompts often have redundant or erroneous points with poor distribution.
Approach
Builds a dual-space heterogeneous graph of prompt candidates using DINOv2 patch feature distances and physical distances, and trains a Q-learning DRL agent (add/delete/move node actions) with a distance-based reward to select an optimized, sparse set of positive/negative prompts that are fed to SAM; the agent is trained once on GBMSD and transferred without retraining.
Key results
- FM-PPO reaches 91.9 mDSC on FSS-1000 and 73.7 on VOC2012, versus 91.6 and 69.0 for Matcher, without in-domain training.
- On Kvasir, FM-PPO improves mDSC from 27.2 to 70.3 over its unoptimized FM initial prompts.
- Matcher-PPO improves Matcher prompts on Kvasir from 46.2 to 77.6 mDSC and on ISIC from 73.7 to 85.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize FM point prompts with PPO | SAM · ISIC | mDSC (%): 48.5 -> 75.4 | Helped |
| Optimize FM point prompts with PPO | SAM · Kvasir | mDSC (%): 27.2 -> 70.3 | Helped |
| Optimize FM point prompts with PPO | SAM · VOC2012 | mDSC (%): 34.7 -> 73.7 | Helped |
| Optimize FM point prompts with PPO (ablation) | SAM · FSS-1000 | mDSC (%): 72.1 -> 91.9 | Helped |
| Optimize feature space only (α=1, β=0) (ablation) | SAM prompt optimization · FSS-1000 | mDSC (%): 72.1 -> 81.3 | Helped |
| Optimize physical space only (α=0, β=1) (ablation) | SAM prompt optimization · FSS-1000 | mDSC (%): 72.1 -> 73.5 | Helped |
| Optimize both feature and physical spaces (α=1, β=1) (ablation) | SAM prompt optimization · FSS-1000 | mDSC (%): 72.1 -> 86.6 | Helped |
| Increase physical-space weight to β=5 (α=1) (ablation) | PPO prompt optimization · FSS-1000 | mDSC (%): 86.6 -> 82.4 | Hurt |
| Increase feature-space weight to α=5 (β=1) (ablation) | PPO prompt optimization · FSS-1000 | mDSC (%): 86.6 -> 91.9 | Helped |
| Optimize CS prompts with PPO | SAM · ISIC | mDSC (%): 57.7 -> 68.6 | Helped |
| Optimize CS prompts with PPO | SAM · Kvasir | mDSC (%): 69.1 -> 79.8 | Helped |
| Optimize Matcher prompts with PPO | SAM · Kvasir | mDSC (%): 46.2 -> 77.6 | Helped |
Surprising findings
- Increasing the physical-space weight (beta=5) reduces performance, indicating accuracy of prompts matters more than sparsity.
Limitations
- Limited when initial prompts do not cover the target (e.g., CS pseudo-labels missing the target).
- Relies on the quality of initial prompts from FM or CS methods.
Experimental setup
Datasets: GBMSD FSS-1000 VOC2012 ISIC2018 Kvasir
Benchmarks: FSS-1000 VOC2012 ISIC2018 Kvasir
Metrics: mDSC mHD
Baselines: HSNet VAT SegGPT PerSAM Matcher CS
Builds on: Segment Anything Model DINOv2 Matcher Deep Q-network
Closest papers by meaning
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.86)
- SAM-Aware Graph Prompt Reasoning Network for Cross-Domain Few-Shot Segmentation (AAAI 2025 · similarity 0.86)
- Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.85)
- AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning (CVPR 2024 · similarity 0.85)
- PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.