Research map / Detection and segmentation / Open-set and few-shot detection
Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Segment Anyword, a training-free open-set grounded segmentation method that test-time optimizes text embeddings of a frozen diffusion model, extracts cross-attention mask prompts, and refines them with SAM using linguistic dependency and syntax regularization.
Problem
Open-set language-grounded segmentation struggles with diverse text reference expressions and typically requires extensive training or fine-tuning of MLLMs or VLMs, yielding inconsistent masks when wording varies.
Approach
Treat segmentation reference expressions as diffusion conditioning text, optimize only the token embeddings at test time via textual inversion/reconstruction, average token cross-attention maps as mask prompts for SAM, and regularize prompts with adjective positive clustering and negative binding of other entities derived from parsed syntax.
Key results
- Achieves 52.5 mIoU on Pascal Context 59, versus 45.7 for EmerDiff among training-free diffusion methods.
- Reaches 67.73 cIoU on gRefCOCO validation with the LoRA-fine-tuned variant, outperforming SAM4MLLM (66.33).
- Obtains 67.4 mIoU on GranDf validation for the training-free variant, above GLaMM (66.3 fine-tuned on GranDf val mIoU reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAM post-processing to the thresholded cross-attention mask (ablation) | Segment Anyword on GranDf Val · GranDf Val | mIoU: 13.7 to 32.4 | Helped |
| Add test-time textual embedding updates with 550 optimizing steps (ablation) | Segment Anyword with SAM on GranDf Val · GranDf Val | mIoU: 6.4 mIoU increase | Helped |
| Increase test-time textual embedding updates to 1100 steps (ablation) | Segment Anyword with SAM on GranDf Val · GranDf Val | mIoU: further 10.2 mIoU increase | Helped |
| Add positive adjective prompt clustering (R1) (ablation) | Segment Anyword with SAM and 1100-step prompt learning · GranDf Val | mIoU: 42.6 -> 62.2 | Helped |
| Add negative mutual-exclusive prompt binding (R2) (ablation) | Segment Anyword with SAM and 1100-step prompt learning · GranDf Val | mIoU: 42.6 -> 63.1 | Helped |
| Use both R1 and R2 prompt regularization (ablation) | Segment Anyword with SAM and 1100-step prompt learning · GranDf Val | mIoU: 63.1 -> 67.4 | Helped |
| Use LoRA text-encoder adaptation and 50 prompt-learning steps instead of 1100 (ablation) | Segment Anyword with R1 and R2 · GranDf Val | mIoU: 67.4 -> 65.9 | Hurt |
| Use GPT-4o instead of SpaCy (RoBERTa) for POS tagging (ablation) | Segment Anyword on RefCOCO sample · RefCOCO sample | mIoU: 46.9 -> 68.2 | Helped |
| Replace ground-truth text with GLaMM-generated text (ablation) | Segment Anyword_f on GranDf Val · GranDf Val | mIoU: 65.9 -> 62.5 | Hurt |
| Use Segment Anyword instead of EmerDiff | Pascal Context 59 · Pascal Context 59 | mIoU: 45.7 -> 52.5 | Helped |
| Use Segment Anyword_f instead of SAM4MLLM | gRefCOCO Val · gRefCOCO Val | cIoU: 66.33 -> 67.73 | Helped |
Surprising findings
- Plain cross-attention masks with a threshold reach only 13.7 mIoU on GranDf, and SAM post-processing raises this to 32.4, showing raw diffusion attention alone is weak.
- Using GLaMM-generated captions instead of ground-truth text drops GranDf validation mIoU from 65.9 to 62.5.
Limitations
- Struggles with very tiny objects and thin structures due to the 16x16 cross-attention resolution.
- Performance depends on the quality of the text parsing and on the input expression, with noisy captions degrading results.
Experimental setup
Datasets: GranDf gRefCOCO RefCOCO RefCOCO+ RefCOCOg PASCAL Context
Benchmarks: GranDf gRefCOCO RefCOCO RefCOCO+ RefCOCOg PASCAL Context 59
Metrics: AP50 mIoU Recall cIoU gIoU
Baselines: GLaMM LISA OMG-LLaVA GSVA SAM4MLLM PSALM CLIPasRNN EmerDiff OVDiff
Builds on: multi-concept prompt learning latent diffusion model Segment Anything Model direct inversion
Closest papers by meaning
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.90)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.90)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.90)
- Visual In-Context Prompting (CVPR 2024 · similarity 0.89)
- From Pixels to Concepts: Do Segmentation Models Understand What They Segment? (NeurIPS 2026 · similarity 0.89)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.