Research map / Detection and segmentation / Open-set and few-shot detection
Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenSAM, a training-free test-time adaptation method that turns a single generic text prompt ("the camouflaged animal") into image-specific visual prompts for SAM via cross-modal chain-of-thought prompting with BLIP-2 and a spatial CLIP, refined by progressive mask generation, for weakly-supervised camouflaged object detection.
Problem
Weakly-supervised camouflaged object detection needs per-image sparse annotations (points/scribbles), and SAM requires instance-specific manual prompts that are ambiguous and not always available in practice.
Approach
Replace manual prompts with automatically generated visual prompts: BLIP-2 keywords from multiple chains of thought (foreground and background) are scored by a spatial CLIP with k-k-v self-attention to form consensus heatmaps, which become SAM point prompts; progressive mask generation reweights the image with the heatmap over several iterations, with all parameters fixed.
Key results
- On COD10K, GenSAM with the generic prompt reaches F-beta 0.681, E-phi 0.838, S-alpha 0.775, outperforming point-supervised SAM-P (0.694/0.796/0.765 on F-beta/E/S with lower E and S) on E-phi and S-alpha and matching scribble-level results.
- On CAMO, GenSAM achieves S-alpha 0.719 and E-phi 0.775 versus CLIP Surgery+SAM at 0.573 and 0.666.
- GenSAM improves over CLIP Surgery+SAM on polyp (ETIS) and shadow (SBU) segmentation with the generic prompts, e.g., SBU S-alpha 0.529 vs 0.442.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenSAM's generic task prompt and image-specific prompt generation instead of CLIP Surgery+SAM | SAM for camouflaged object segmentation · COD10K | Eφ: 0.672 -> 0.838 | Helped |
| add BLIP2 keyword generation (ablation) | GenSAM · COD10K | Fβ: 0.448 -> 0.556 | Helped |
| add foreground chain consensus (ablation) | GenSAM with BLIP2 keywords · COD10K | Fβ: 0.556 -> 0.569 | Helped |
| add Progressive Mask Generation (ablation) | GenSAM with foreground chain · COD10K | Fβ: 0.569 -> 0.616 | Helped |
| add k-k-v self-attention (ablation) | GenSAM with PMG · COD10K | Fβ: 0.616 -> 0.660 | Helped |
| add background chain consensus (ablation) | GenSAM with k-k-v self-attention · COD10K | Fβ: 0.660 -> 0.681 | Helped |
| use 1 chain instead of 3 (ablation) | GenSAM · COD10K | Fβ: 0.681 -> 0.671 | Hurt |
| use 5 chains instead of 3 (ablation) | GenSAM · COD10K | Fβ: 0.681 -> 0.683 | Helped |
| use upsample factor 8 instead of 2 (ablation) | GenSAM heatmap prompt · COD10K | Fβ: 0.681 -> 0.496 | Hurt |
| use threshold 0.80 instead of 0.90 (ablation) | GenSAM heatmap-to-point prompts · COD10K | Fβ: 0.681 -> 0.549 | Hurt |
| use MaxBox instead of MaxIOUBox (ablation) | GenSAM post-processing · COD10K | Fβ: 0.681 -> 0.639 | Hurt |
| use the previous mask as prompt instead of MaxIOUBox (ablation) | GenSAM post-processing · COD10K | Fβ: 0.681 -> 0.666 | Hurt |
Surprising findings
- Removing the iterative/keyword components does not uniformly help: on CHAMELEON the full model's MAE (0.090) is not the best, which the authors attribute to the small dataset size.
Limitations
- Performance on the hardest dataset (COD10K F-beta and MAE) still trails scribble and point supervised methods in some metrics.
- Relies on the quality of BLIP-2 keywords and CLIP heatmaps; noisy keywords can produce misleading point prompts.
- Iterative test-time adaptation adds inference cost over a single pass.
Experimental setup
Datasets: CHAMELEON CAMO COD10K
Benchmarks: CHAMELEON CAMO COD10K ETIS SBU
Metrics: MAE adaptive F-measure mean E-measure structure measure
Baselines: SAM SAM-P SAM-S WSSA SCWS TEL SCOD CLIP Surgery+SAM
Builds on: Segment Anything Model CLIP Surgery BLIP-2 generated knowledge prompting
Closest papers by meaning
- Endow SAM with Keen Eyes: Temporal-spatial Prompt Learning for Video Camouflaged Object Detection (CVPR 2024 · similarity 0.91)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.87)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.87)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.87)
- SAM-Aware Graph Prompt Reasoning Network for Cross-Domain Few-Shot Segmentation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.