Research map / Detection and segmentation / Open-set and few-shot detection
AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlignSAM, which keeps SAM frozen and trains a PPO-based RL prompting agent plus a semantic recalibration module (CLIP-Surgery text attention or visual-similarity branch) to automatically generate point prompts for adapting SAM to blur, shadow, glass, saliency, and semantic segmentation.
Problem
SAM is class-agnostic and depends on manual prompts, so adapting it to diverse downstream segmentation tasks (explicit and implicit semantics) without costly fine-tuning or gradient access is hard.
Approach
Formulate automatic point prompting as a sequential decision process: an actor-critic PPO agent selects patch-center points iteratively (reward +1/-1 for foreground/background), with a semantic recalibration module that labels prompts using CLIP-Surgery text attention (explicit tasks) or a learned visual branch (implicit tasks), while SAM stays frozen.
Key results
- On blur detection (CUHK) AlignSAM reaches 68.47 mIoU vs 55.84 for PerSAM and 55.44 for SAMed.
- On Pascal VOC 2012 AlignSAM achieves 62.09 mIoU vs 59.27 for Painter and 53.02 for PerSAM.
- Removing RL drops blur-detection mIoU by 8.72 points, and removing multi-step interaction drops it by 21.55 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AlignSAM instead of the strongest reported baseline | SAM on CUHK blur detection · CUHK | mIoU: 55.44 -> 68.47 | Helped |
| use AlignSAM instead of the strongest reported baseline | SAM on SBU shadow detection · SBU | mIoU: 19.52 -> 30.78 | Helped |
| use AlignSAM instead of the strongest reported baseline | SAM on MSD glass detection · MSD | mIoU: 41.35 -> 45.44 | Helped |
| use AlignSAM instead of the strongest reported baseline | SAM on DUTS saliency detection · DUTS | MAE: 0.104 -> 0.082 | Helped |
| use AlignSAM instead of the strongest reported baseline | SAM on PASCAL VOC 2012 · PASCAL VOC 2012 | mIoU: 59.27 -> 62.09 | Helped |
| add reinforcement learning (ablation) | AlignSAM on CUHK · CUHK | mIoU: 59.75 -> 68.47 | Helped |
| add reinforcement learning (ablation) | AlignSAM on SBU · SBU | mIoU: 25.62 -> 30.78 | Helped |
| add reinforcement learning (ablation) | AlignSAM on MSD · MSD | mIoU: 33.41 -> 45.44 | Helped |
| add semantic recalibration module (ablation) | AlignSAM on CUHK · CUHK | mIoU: 66.89 -> 68.47 | Helped |
| add semantic recalibration module (ablation) | AlignSAM on SBU · SBU | mIoU: 21.29 -> 30.78 | Helped |
| add semantic recalibration module (ablation) | AlignSAM on PASCAL VOC 2012 · PASCAL VOC 2012 | mIoU: 27.73 -> 62.09 | Helped |
| add multi-step interaction (ablation) | AlignSAM on CUHK · CUHK | mIoU: 46.92 -> 68.47 | Helped |
Surprising findings
- Removing the semantic recalibration module on Pascal VOC collapses mIoU to 27.73, far below the RL-free variant.
Limitations
- Requires multiple interaction steps (T=15) at inference, increasing cost
- Explicit vs implicit branch must be chosen manually per task
- Evaluated on a limited set of segmentation tasks with only 50 training samples
Experimental setup
Datasets: CUHK SBU MSD DUTS PASCAL VOC 2012
Benchmarks: CUHK SBU MSD DUTS PASCAL VOC 2012
Metrics: mIoU F-measure BER E-measure MAE
Baselines: SAMed SEEM Painter PerSAM MSA
Builds on: Segment Anything Model PPO CLIP-Surgery PerSAM
Closest papers by meaning
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment (NeurIPS 2026 · similarity 0.85)
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.85)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.85)
- Plug-and-Play PPO: An Adaptive Point Prompt Optimizer Making SAM Greater (CVPR 2025 · similarity 0.85)
- Prompt-based Visual Alignment for Zero-shot Policy Transfer (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.