Research map / Detection and segmentation / Open-set and few-shot detection
MaskPrompt: Open-Vocabulary Affordance Segmentation with Object Shape Mask Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes open-vocabulary affordance segmentation (OVAS) with a new OVAS-25 benchmark and MaskPrompt, which uses SAM object-shape masks, Alpha-CLIP/BLIP-2 mask captions, and masked instance features to segment affordance parts of unseen categories.
Problem
Affordance segmentation from images struggles with background and adjacent-object pixel interference in cluttered scenes and cannot segment affordance categories unseen during training (open-vocabulary setting).
Approach
Use detector+SAM object shape masks to (1) build masked single-instance images whose features are concatenated with global features, and (2) generate mask-region captions via Alpha-CLIP and BLIP-2 as fine-grained text prompts, then decode class-agnostic affordance masks with a text-conditioned pixel decoder and classify them by cosine similarity to text embeddings.
Key results
- On OVAS-25, MaskPrompt (ResNet-101) reaches 71.26 mIoU, 5.27 points above OV-PARTS (65.99 F1-based comparison in Table 1 reports 81.58 F1 for ours vs 76.06 for OV-PARTS).
- Ablation on OVAS-25: adding mask prompt feature enhancement raises mIoU from 60.51 to 67.41; adding mask text prompts and the pixel decoder bring it to 71.26.
- On IIT-AFF and UMD closed-vocabulary affordance benchmarks, ResNet-101 variant reports 89.46 and 93.83 F1, competitive with ADOSMNet (85.85 / 91.81 F1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaskPrompt instead of OV-PARTS | Open-vocabulary affordance segmentation · OVAS-25 | mIoU: 65.99 -> 71.26 | Helped |
| Use MaskPrompt instead of ADOSMNet | Affordance segmentation · IIT-AFF | F1-Score: 85.85 -> 89.46 | Helped |
| Use MaskPrompt instead of ADOSMNet | Affordance segmentation · UMD | F1-Score: 91.81 -> 93.83 | Helped |
| Use MaskPrompt instead of FLOAT | Open-vocabulary part segmentation · Pascal-Part-58 | mIoU: 61.0 -> 63.8 | Helped |
| Use MaskPrompt instead of FLOAT | Open-vocabulary part segmentation · Pascal-Part-108 | mIoU: 48.0 -> 49.5 | Helped |
| Use MaskPrompt instead of FLOAT | Open-vocabulary part segmentation · Pascal-Part-201 | mIoU: 37.1 -> 40.6 | Helped |
| Add MPFEM to the basic model (ablation) | MaskPrompt · OVAS-25 | mIoU: 60.51 -> 67.41 | Helped |
| Add MPGM (fine-grained text prompts) to MPFEM (ablation) | MaskPrompt · OVAS-25 | mIoU: 67.41 -> 69.65 | Helped |
| Replace the MaskFormer pixel decoder with the proposed pixel decoder (APM) (ablation) | MaskPrompt with MPFEM and MPGM · OVAS-25 | mIoU: 69.65 -> 71.26 | Helped |
Surprising findings
- The method is slower than some baselines (3.8 FPS vs 10.5 for OV-PARTS in Table 5), despite having fewer parameters and GFLOPs.
Limitations
- Depends on accuracy of the object detector and SAM mask generation pipeline.
- Inference speed (3.8 FPS) is slower than some compared methods despite lower parameter count and FLOPs.
Experimental setup
Datasets: OVAS-25 IIT-AFF Pascal-Part-108 UMD Pascal-Part-58 Pascal-Part-116 Pascal-Part-201 ADE20K-Part-234
Benchmarks: OVAS-25 IIT-AFF UMD Pascal-Part-58 Pascal-Part-108 Pascal-Part-201 Pascal-Part-116 ADE20K-Part-234
Metrics: mIoU mAvg F1-Score
Baselines: FLOAT OPS VLPart OV-PARTS GSE RelaNet BPN ADOSMNet BSANet GMNet CO-Rank ZSseg+
Builds on: MaskFormer FLOAT ADOSMNet Alpha-CLIP
Closest papers by meaning
- One-Shot Open Affordance Learning with Foundation Models (CVPR 2024 · similarity 0.89)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.88)
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.88)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.87)
- UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation (NeurIPS 2026 · similarity 0.86)
- LASO: Language-guided Affordance Segmentation on 3D Object (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.