Research map / Detection and segmentation / CNN features
RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces real-time multi-purpose segmentation (interactive, panoptic, video instance segmentation) and RMP-SAM, a single shared dynamic-convolution decoder with asymmetric adapters that is co-trained on COCO, COCO-SAM and YouTube-VIS 2019.
Problem
Existing segmentation foundation models are too heavy for real-time use, and specialized real-time methods handle only one task; no single real-time model covers image panoptic, video instance, and SAM-style interactive segmentation.
Approach
A lightweight encoder with a shared pooling-based dynamic convolution decoder that handles object queries and visual-prompt queries together, followed by asymmetric adapters (dynamic-conv for object queries, cross-attention for prompt queries) to balance multi-task co-training.
Key results
- RMP-SAM with ResNet18 reaches 39.9 PQ on COCO-Panoptic, 38.7 mAP on YouTube-VIS 2019, and 40.3 FPS on an A100, outperforming Mask2Former (35.6 PQ, 31.2 FPS) at similar cost.
- With ResNet50, RMP-SAM reaches 46.9 PQ and 46.2 mAP at 35.1 FPS, versus Mask2Former's 42.9 PQ and 26.6 FPS.
- With ConvNeXt-L, RMP-SAM reports 52.3 PQ, 61.1 mIoU and 55.6 mAP at 8.6 FPS, and 33.2 mAP on COCO instance segmentation with SAM-like prompts at 47M parameters and 35 GFLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RMP-SAM instead of Mask2Former | ResNet-50 models · COCO-Panoptic | PQ: 42.9 -> 46.9 | Helped |
| Use RMP-SAM instead of MaskFormer | ResNet-50 models · COCO-SAM | mIoU: 58.8 -> 57.9 | Hurt |
| Use RMP-SAM instead of Mask2Former | ResNet-50 models · YouTube-VIS 2019 | mAP: 42.1 -> 46.2 | Helped |
| Use RMP-SAM instead of Mask2Former | ResNet-18 models · COCO-Panoptic | PQ: 35.6 -> 39.9 | Helped |
| Use RMP-SAM instead of MaskFormer | ResNet-18 models · COCO-SAM | mIoU: 55.6 -> 52.7 | Hurt |
| Use RMP-SAM instead of Mask2Former | ResNet-18 models · YouTube-VIS 2019 | mAP: 38.6 -> 38.7 | Helped |
| Use RMP-SAM instead of MaskFormer | TopFormer models · COCO-Panoptic | PQ: 31.6 -> 34.6 | Helped |
| Use RMP-SAM instead of MaskFormer | TopFormer models · COCO-SAM | mIoU: 56.1 -> 53.3 | Hurt |
| Use RMP-SAM instead of MaskFormer | TopFormer models · YouTube-VIS 2019 | mAP: 38.2 -> 41.7 | Helped |
| Add video training to image-only training (ablation) | Joint image/video segmentation training · COCO-Panoptic | PQ: 36.6 -> 36.2 | Hurt |
| Add image training to video-only training (ablation) | Joint image/video segmentation training · YouTube-VIS 2019 | mAP: 21.5 -> 36.0 | Helped |
| Add COCO-SAM training to image + video co-training (ablation) | Joint multi-purpose training · COCO-Panoptic | PQ: 36.2 -> 35.7 | Hurt |
Surprising findings
- Decoupled decoders did not improve over a single shared decoder, which the authors attribute to weaker lightweight encoders.
- Joint co-training with image and video data improves video instance segmentation but lowers panoptic segmentation performance.
Limitations
- Struggles with highly overlapping objects and crowded scenes given a limited number of instance kernels.
- Lags the strongest specialized video models (e.g., Tube-Link, DVIS) on single-task video segmentation with the lightweight backbone.
Experimental setup
Datasets: COCO YouTube-VIS 2019 COCO-SAM ADE20K VIPSeg SA-1B
Benchmarks: COCO-Panoptic COCO-SAM YouTube-VIS 2019 VIPSeg ADE20K COCO instance segmentation
Metrics: PQ SQ mIoU mAP VPQ STQ FPS FLOPs parameters 1-IoU
Baselines: Mask2Former MaskFormer kMaX-DeepLab YOSO K-Net SAM-H MobileSAM TinySAM EdgeSAM EfficientSAM FastSAM Semantic-SAM
Builds on: Mask2Former SAM YOSO ViLD kMaX-DeepLab
Closest papers by meaning
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.88)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.87)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.86)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.