Research map / Detection and segmentation / Open-set and few-shot detection
Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Perception Tuning (SPT), which adapts SAM for promptable industrial anomaly segmentation via a self-draft decoder (coarse mask then refine) and a visual-relation-aware adapter, outperforming LoRA/DoRA/Adapter baselines.
Problem
Zero-shot SAM and prompt-based anomaly segmentation suffer from domain shift on industrial images, and conventional PEFT methods do not adequately address perception challenges during adaptation to anomaly images under varied prompts.
Approach
A self-draft tuning strategy adds a draft decoder (initialized from the mask decoder) that produces a coarse anomaly mask, which is re-encoded as a dense prompt for the refine decoder; a visual-relation-aware adapter injects cosine-similarity-based inter-region relations into the two-way transformer features of the decoders.
Key results
- SPT with DoRA (ViT-B) reaches 68.5 mIoU / 63.1 mBIoU averaged over four prompt modes, versus 66.5 / 60.9 for DoRA and 52.0 / 47.0 for zero-shot SAM.
- Average gain over zero-shot SAM is over 15% in mIoU and mBIoU across six datasets.
- Ablation with ViT-B LoRA variant: SDT alone gives 67.6 / 62.2 and SDT plus VRA-Adapter gives 68.4 / 62.8 versus PEFT-only 66.5 / 61.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SPT with DoRA instead of DoRA alone | SAM ViT-B · Six benchmark datasets, Avg. (four prompt types) | mIoU: 66.5 -> 68.5 | Helped |
| Apply SPT with DoRA instead of DoRA alone | SAM ViT-B · Six benchmark datasets, Avg. (four prompt types) | mBIoU: 60.9 -> 63.1 | Helped |
| Add SDT to the PEFT baseline (ablation) | SAM ViT-B with LoRA · Box-level and point-level prompts, Avg. | mIoU: 66.5 -> 67.6 | Helped |
| Add SDT to the PEFT baseline (ablation) | SAM ViT-B with LoRA · Box-level and point-level prompts, Avg. | mBIoU: 61.0 -> 62.2 | Helped |
| Add VRA-Adapter to SDT (ablation) | SAM ViT-B with LoRA · Box-level and point-level prompts, Avg. | mIoU: 67.6 -> 68.4 | Helped |
| Add VRA-Adapter to SDT (ablation) | SAM ViT-B with LoRA · Box-level and point-level prompts, Avg. | mBIoU: 62.2 -> 62.8 | Helped |
| Add SDT without PEFT tuning (ablation) | Zero-shot SAM ViT-B · Box-level and point-level prompts, Avg. | mIoU: 52.0 -> 53.0 | Helped |
| Add SDT without PEFT tuning (ablation) | Zero-shot SAM ViT-B · Box-level and point-level prompts, Avg. | mBIoU: 47.0 -> 48.4 | Helped |
| Use two draft decoders instead of one (ablation) | SPT with LoRA · Box-level and point-level prompts, Avg. | mIoU: 68.4 -> 68.1 | No clear effect |
| Use two draft decoders instead of one (ablation) | SPT with LoRA · Box-level and point-level prompts, Avg. | mBIoU: 62.8 -> 62.9 | No clear effect |
| Replace the separate draft decoder with the mask decoder applied twice (ablation) | SPT with LoRA · Box-level and point-level prompts, Avg. | mIoU: 68.4 -> 67.0 | Hurt |
| Replace the separate draft decoder with the mask decoder applied twice (ablation) | SPT with LoRA · Box-level and point-level prompts, Avg. | mBIoU: 62.8 -> 61.6 | Hurt |
Limitations
- Training dataset is proprietary and not released, limiting reproducibility
- Evaluated only on industrial anomaly datasets with prompts derived from ground truth masks
- Automatic-prompt evaluation uses human-crafted prompts known to be sensitive across models and datasets
Experimental setup
Datasets: in-house industrial anomaly training set (~15,000 images)
Benchmarks: MVTec AD VisA MTD KSDD2 BTAD MPDD
Metrics: mIoU mBIoU
Baselines: zero-shot SAM LoRA DoRA Adapter SAMAdapter AdaptFormer NOLA BitFit
Builds on: SAM LoRA DoRA Adapter
Closest papers by meaning
- MetaUAS: Universal Anomaly Segmentation with One-Prompt Meta-Learning (NeurIPS 2024 · similarity 0.87)
- Self-Supervised Likelihood Estimation with Energy Guidance for Anomaly Segmentation in Urban Scenes (AAAI 2024 · similarity 0.85)
- Unsupervised Continual Anomaly Detection with Contrastively-Learned Prompt (AAAI 2024 · similarity 0.84)
- A Dataset for Semantic Segmentation in the Presence of Unknowns (CVPR 2025 · similarity 0.84)
- Unseen Visual Anomaly Generation (CVPR 2025 · similarity 0.84)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.