Research map / Image generation / Autoregressive and latent generation
AoP-SAM: Automation of Prompts for Efficient Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AoP-SAM, a lightweight CNN Prompt Predictor that uses SAM's image embeddings to output a prompt confidence map, plus test-time adaptive sampling and filtering, to automatically generate point prompts for SAM without fine-tuning SAM.
Problem
Automatic prompt generation for SAM: grid-based automatic mask generation is either too sparse (missing objects) or too dense (redundant masks and latency), and detector-based prompts (OAS with YOLOv8) add heavy computational overhead.
Approach
A lightweight prompt predictor fuses the raw image and SAM's ViT image embedding to produce a prompt confidence map, and a coarse-to-fine test-time Adaptive Sampling and Filtering step removes prompts likely to yield duplicate masks, using previously generated masks as references.
Key results
- On SA-1B with ViT-H SAM, AoP-SAM reaches 70.6 mIoU at 0.122 s latency versus 66.9 mIoU for OAS (Box) at 0.160 s.
- With MobileSAM on SA-1B, AoP-SAM achieves 51.4 mIoU vs 50.7 for OAS (Box), with latency 0.101 s vs 0.191 s and peak memory 4.1 GB vs 7.3 GB.
- On a Jetson Orin Nano edge GPU, AoP-SAM latency is 0.65 s on SA-1B versus 1.16 s for OAS (Box), with peak memory 0.042 GB vs 0.78 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AoP-SAM instead of OAS (Central Point) | MobileSAM with ViT_H image encoder · SA-1B | mIoU: 68.3 -> 70.6 | Helped |
| Use AoP-SAM instead of AMG_D | MobileSAM with ViT_H image encoder · COCO | mIoU: 69.5 -> 70.1 | Helped |
| Use AoP-SAM instead of AMG_D | MobileSAM with ViT_H image encoder · LVIS | mIoU: 71.0 -> 71.9 | Helped |
| Add Adaptive Sampling to the Prompt Predictor (ablation) | AoP-SAM · SA-1B | mIoU: 57.2 -> 72.8 | Helped |
| Add Adaptive Sampling to the Prompt Predictor (ablation) | AoP-SAM · COCO | mIoU: 67.9 -> 70.5 | Helped |
| Add Adaptive Sampling to the Prompt Predictor (ablation) | AoP-SAM · LVIS | mIoU: 60.9 -> 71.7 | Helped |
| Add Adaptive Filtering after Adaptive Sampling (ablation) | AoP-SAM · SA-1B | mIoU: 72.8 -> 71.3 | Hurt |
| Add Adaptive Filtering after Adaptive Sampling (ablation) | AoP-SAM · COCO | mIoU: 70.5 -> 70.1 | Hurt |
| Add Adaptive Filtering after Adaptive Sampling (ablation) | AoP-SAM · LVIS | mIoU: 71.7 -> 71.9 | Helped |
| Increase the sampling smoothing factor from 1 to 4 (ablation) | AoP-SAM · SA-1B | mIoU: 72.4 -> 63.4 | Hurt |
| Increase the confidence intensity threshold from 0.1 to 0.4 (ablation) | AoP-SAM · SA-1B | mIoU: 70.9 -> 66.4 | Hurt |
| Increase the prompt spacing factor from 4 to 7 (ablation) | AoP-SAM · SA-1B | mIoU: 72.7 -> 68.9 | Hurt |
Surprising findings
- Adaptive filtering reduces latency and memory but slightly lowers mIoU on SA-1B for ViT-H (71.3 vs 72.8 without filtering in the ablation table).
Limitations
- Only MobileSAM could run on the edge device because other pre-trained models exhausted edge GPU memory.
- Training details are incomplete (learning rate value is missing from the text) and the training subset size is not specified.
Experimental setup
Datasets: SA-1B
Benchmarks: SA-1B COCO LVIS
Metrics: mIoU inference latency peak memory number of prompts
Baselines: AMG_S AMG_D OAS (Box) OAS (Central Point)
Builds on: Segment Anything Model Automatic Mask Generation Object-Aware Sampling
Closest papers by meaning
- Stable Segment Anything Model (ICLR 2025 · similarity 0.91)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.90)
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.89)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.89)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.