Research map / Detection and segmentation / Open-set and few-shot detection

From Pixels to Concepts: Do Segmentation Models Understand What They Segment?

NeurIPS 2026 · Detection and segmentation › Open-set and few-shot detection · approach: Interactive and referring segmentation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CAFE, a 2,146-sample counterfactual benchmark with attribute-level image edits to test whether promptable concept segmentation models reject misleading concept prompts, and proposes an agentic SAM3 verification variant (CAFE-SAM3).

Problem

Existing segmentation benchmarks measure mask accuracy or object presence but do not test whether promptable concept segmentation models faithfully ground a concept when its target remains visible and localizable under misleading attribute cues (appearance, context, material).

Approach

Attribute-level counterfactual image editing that preserves the target region and its mask while changing surface appearance, context, or material, paired with a positive and a misleading negative prompt; evaluated with class-gated F1 plus new target-aligned false positive and concept swap metrics.

Key results

• SAM 3 reaches overall cgF1 38.5 and IL_MCC 0.590 but drops to IL_MCC -0.241 on Ontological Conflict, and has 66.3% FPR there.
• CAFE-SAM3 (GPT-5.5) raises overall cgF1 to 63.3 (+24.8) and IL_MCC to 0.843, and cuts overall FPR from 21.2% to 13.5%.
• Grounded SAM 2 keeps pmF1 around 60 across categories but has IL_MCC of only 0.165 overall and 94.5% FPR, showing accurate masks do not imply semantic rejection.

Surprising findings

• Most non-agentic models obtain negative IL_MCC on Ontological Conflict, indicating inverse correlation with the semantic labels.
• Accurate masks (high pmF1) coexist with near-chance concept discrimination, so mask quality does not indicate semantic grounding.

Limitations

• Only a single counterfactually edited target per image; crowded or mixed-instance scenes are not covered.
• OC false-positive rates remain high even for the best agentic variant (29.2%).
• Dataset relies on generative image editing and filtering retains only 4.4% of 48,423 raw samples.

Experimental setup

Datasets: COCO-Val2017 SA-Co/Gold LVIS-Val

Benchmarks: CAFE

Metrics: cgF1 IL_MCC pmF1 FPR AFPR UFPR ACSR UCSR

Baselines: SAM 3 YOLO-World OpenSeeD Grounded SAM 2 OWLv2 + SAM1

Builds on: SAM3 HalluSegBench SA-Co Grounded SAM 2

Closest papers by meaning

• Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.89)
• Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.87)
• Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.87)
• Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.86)
• Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.86)
• Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.86)