Research map / Vision-language and multimodal / CLIP and zero-shot
DeiSAM: Segment Anything with Deictic Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeiSAM, a modular neuro-symbolic pipeline where an LLM turns deictic text prompts into first-order logic rules, a differentiable forward reasoner runs over scene graphs, and SAM segments the matched regions, plus the DeiVG benchmark.
Problem
Segmenting objects described by complex deictic textual prompts (references defined relative to other objects and context) that purely neural grounding-plus-segmentation pipelines fail to reason over reliably.
Approach
Combine LLM-generated first-order logic rules with a differentiable forward reasoner (NEUMANN) over scene graphs, plus embedding-based semantic unification of terms, and then segment the logically inferred regions with SAM; the whole pipeline is differentiable so rule/SGG weights can be learned end-to-end.
Key results
- On DeiVG_2 DeiSAM reaches 85.40 mAP versus 56.03 for LISA and 32.33 for Grounded DINO-SAM.
- On DeiRefCOCO+ val DeiSAM gets 71.56 mAP while LISA gets 44.92 and GroundedSAM 30.06, showing robustness to abstract prompts.
- End-to-end learning of an SGG mixture raises DeiVG_1 test mAP from 37.61 (naive) to 64.44.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DeiSAM instead of the strongest reported neural baseline | DeiVG deictic segmentation · DeiVG_1, DeiVG_2, DeiVG_3 | mAP (%): 14.90 -> 65.14; 56.03 -> 85.40; 75.79 -> 87.83 | Helped |
| use DeiSAM instead of LISA | RefCOCO+ · val, testA, testB | mAP (%): 67.55 -> 71.72; 74.86 -> 77.29; 63.03 -> 64.98 | Helped |
| use DeiSAM instead of LISA | DeiRefCOCO+ abstract prompts · val, testA, testB | mAP (%): 44.92 -> 71.56; 47.60 -> 79.51; 43.23 -> 66.43 | Helped |
| use DeiSAM instead of the strongest baseline on each operation | DeiCLEVR abstract reasoning segmentation · Delete and Sort | mAP: 12.88 -> 99.29; 15.39 -> 99.57 | Helped |
| train the mixture weights instead of keeping them randomly initialized (ablation) | DeiSAM mixture of VETO and Visual Genome scene graphs · DeiVG_1 and DeiVG_2 test split | mAP (%): 37.61 -> 64.44; 59.81 -> 86.57 | Helped |
| add few-shot examples to instruction-only rule generation (ablation) | Llama-2-13B-Chat rule generation · DeiVG_1, DeiVG_2, DeiVG_3 | Overall Success (%): 0.00 -> 94.04; 0.00 -> 92.52; 0.00 -> 90.17 | Helped |
| add chain-of-thought predicate decomposition to few-shot prompting (ablation) | Llama-2-13B-Chat rule generation · DeiVG_1, DeiVG_2, DeiVG_3 | Overall Success (%): 94.04 -> 91.00; 92.52 -> 95.17; 90.17 -> 93.45 | Mixed |
| replace few-shot prompting with chain-of-thought alone (ablation) | Llama-2-13B-Chat rule generation · DeiVG_1, DeiVG_2, DeiVG_3 | Overall Success (%): 94.04 -> 0.00; 92.52 -> 0.00; 90.17 -> 0.00 | Hurt |
| replace few-shot prompting with instruction-only prompting (ablation) | Llama-2-13B-Chat rule generation · DeiVG_1, DeiVG_2, DeiVG_3 | Overall Success (%): 94.04 -> 0.00; 92.52 -> 0.00; 90.17 -> 0.00 | Hurt |
| use larger instruction-tuned models for rule generation (ablation) | LLM rule generation · DeiVG, especially complex prompts | Combined success (%) | Helped |
| use MPNet-Base-v2 rather than OpenAI-ada-002 for semantic unification (ablation) | DeiSAM semantic unifier · Visual Genome synonym benchmark | Unification Success (%): 66.50 -> 71.62 | Helped |
| add target object labels to the prompts (ablation) | neural baselines on DeiVG · DeiVG_1, DeiVG_2, DeiVG_3 | mAP (%): LISA + 11.91, + 12.75, + 6.41; GroundedSAM +11.39, + 15.58, + 43.44 | Helped |
Surprising findings
- Llama-2-13B-Chat with few-shot prompting reaches roughly 91-95% combined rule-generation success, while zero-shot instruct and CoT alone score 0%.
- Pre-trained VG scene graph generators miss many object types present in the newer VG v1.4 annotations, limiting performance with pretrained SGGs.
Limitations
- Runtime is about 10 seconds per example, dominated by LLM rule generation and semantic unification via the OpenAI API.
- Semantic unification via embeddings is weak (best about 72% success), and VG annotation errors make some DeiVG samples inconsistent.
- Performance depends on scene graph quality; ground-truth VG scene graphs are used in the default configuration.
Experimental setup
Datasets: Visual Genome RefCOCO RefCOCO+ CLEVR ILP visual dataset
Benchmarks: DeiVG DeiRefCOCO+ RefCOCO+ DeiCLEVR
Metrics: mean average precision overall success rate
Baselines: SEEM OFA-SAM GLIP-SAM Grounded SAM LISA
Builds on: Segment Anything Model NEUMANN Grounded SAM Visual Genome
Closest papers by meaning
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.86)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.86)
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.85)
- DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.85)
- Fast Reasoning Segmentation for Images and Videos (CVPR 2026 · similarity 0.85)
- Discriminative Perception via Anchored Description for Reasoning Segmentation (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.