Research map / Detection and segmentation / Open-set and few-shot detection

Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary semantic segmentation · 18 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mask-Adapter, a plug-in module that turns proposal masks and CLIP features into semantic activation maps for mask embedding pooling, trained with an IoU matcher, mask consistency loss, and a two-stage warmup for open-vocabulary segmentation.

Problem

Open-vocabulary segmentation with mask pooling or mask cropping for CLIP-based mask classification has a limited upper bound: accurate masks often yield poor classification because pooled CLIP features lack contextual and semantic detail and masked images differ from CLIP's pretraining distribution.

Approach

Replace direct mask pooling with a learned adapter that maps proposal masks and CLIP features to K semantic activation maps, which weight CLIP features over a context-including region; add an IoU-based matcher, a cosine mask consistency loss, and ground-truth-then-mixed mask-text alignment training.

Key results

• Mask-Adapter with MAFTP (ConvNeXt-L) reaches 38.2 mIoU on A-150 and 16.2 on A-847, state-of-the-art among listed methods, exceeding CAT-Seg's 37.9 and 16.0.
• Plugging Mask-Adapter into FC-CLIP without ensemble raises ADE20K mIoU from 26.5 to 35.4 (+8.9) and into Mask2Former from 26.0 to 34.0.
• On ground-truth ADE20K masks, Mask-Adapter classification accuracy reaches 66.7% with ConvNeXt-L versus 53.0% for mask pooling, and 74.1% with MAFTP.

Measured effects reported in the paper

Surprising findings

• Mask pooling with accurate ground-truth masks yields only about 53% classification accuracy on ADE20K, showing mask pooling is a bottleneck even with perfect masks.
• Training with mixed predicted and ground-truth masks directly slightly reduced unseen-class performance, requiring the warmup stage.

Limitations

• Gains with stronger baselines are modest (e.g., +1.1 mIoU on A-150 for MAFTP with ConvNeXt-B; PC-459 drops for MAFTP w/ Mask-Adapter).
• Adding a subset of the Grand dataset degraded performance, attributed to mask quality issues.
• Gains on SAM with Mask-Adapter are mixed (PC-59 drops from 53.5 to 49.5 for MAFT vs. Mask-Adapter).

Experimental setup

Datasets: COCO-Stuff COCO-Panoptic LVIS ADE20K ground truth masks

Benchmarks: ADE20K (A-150, A-847) Pascal-Context (PC-59, PC-459) Pascal-VOC (PAS-20)

Metrics: mIoU mIoU seen mIoU unseen classification accuracy

Baselines: FC-CLIP MAFTP Mask2Former ZegFormer OVSeg CAT-Seg SAN EBSeg SED GBA SMART MROVSeg

Builds on: FC-CLIP MAFTP Mask2Former SparseInst OVSeg

Closest papers by meaning

• Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
• CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.90)
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.90)
• Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
• Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.89)
• Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)