Research map / Detection and segmentation / Open-set and few-shot detection
RESAnything: Attribute Prompting for Arbitrary Referring Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free, zero-shot arbitrary referring expression segmentation that uses SAM proposals, MLLM-generated attribute descriptions (reference and candidate texts), and MLLM+CLIP multi-metric selection, plus a new 2,989-instance part-level benchmark ABO-Image-ARES.
Problem
Referring expression segmentation is limited to object-level, label-centric expressions; part-level and implicit expressions (function, style, material, logos) are poorly handled, and existing MLLM-based methods require fine-tuning.
Approach
Attribute prompting: an MLLM generates detailed shape/color/location descriptions for the referring expression and for each SAM mask proposal (via bounding-box and mask-cropped visual prompts), then matches them with text-to-text and text-to-image MLLM binary decisions combined with CLIP scores to select a mask, with no training.
Key results
- On ReasonSeg val, RESAnything reaches 74.6 gIoU and 72.5 cIoU, versus 57.7 gIoU for LISA-13B-LLaVA1.5.
- On ABO-Image-ARES test, RESAnything achieves 78.2 gIoU and 72.4 cIoU, versus 46.2 gIoU for GLaMM.
- On RefCOCO val, RESAnything gets 68.5 cIoU among training-free zero-shot methods, versus 33.6 for CaR and 26.2 for GLCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RESAnything instead of the strongest listed baseline | RESAnything · ReasonSeg | gIoU: 74.6 vs 58.4 | Helped |
| Use RESAnything instead of the strongest listed baseline | RESAnything · ReasonSeg | cIoU: 72.5 vs 60.4 | Helped |
| Use RESAnything instead of GLaMM | RESAnything · ABO-Image-ARES | gIoU: 78.2 vs 46.2 | Helped |
| Use RESAnything instead of GLaMM | RESAnything · ABO-Image-ARES | cIoU: 72.4 vs 38.7 | Helped |
| Use mask-cropped and bounding-box visual prompts rather than mask-cropped alone (ablation) | RESAnything candidate-text generation · RefCOCO test A | gIoU: 47.2 -> 72.2 | Helped |
| Add the original image to the mask-cropped visual prompt (ablation) | RESAnything candidate-text generation · RefCOCO test A | gIoU: 47.2 -> 56.2 | Helped |
| Use the mask-cropped and bounding-box prompt combination rather than mask-cropped plus contour (ablation) | RESAnything candidate-text generation · RefCOCO test A | gIoU: 68.5 -> 72.2 | Helped |
| Add CLIP scores to the LLM mask selector (ablation) | RESAnything mask selection · ReasonSeg | gIoU: 70.5 -> 74.6 | Helped |
| Add LLM decisions to the CLIP-only mask selector (ablation) | RESAnything mask selection · ReasonSeg | gIoU: 42.5 -> 74.6 | Helped |
| Replace Pixtral 12B with Claude 3.5 Sonnet (ablation) | RESAnything · ReasonSeg | gIoU: 74.6 -> 76.2 | Helped |
| Replace Pixtral 12B with Qwen 2-VL (ablation) | RESAnything · ReasonSeg | gIoU: 74.6 -> 74.2 | No clear effect |
Surprising findings
- MLLM-generated numeric similarity scores were found to be high-variance and poorly correlated, so binary yes/no decisions were used instead.
- CLIP text-to-image scores consistently stayed below 0.3, giving limited discriminative power.
Limitations
- Performance lags supervised methods on large complete objects because the approach relies on smaller SAM mask proposals.
- Results depend on the chosen MLLM and SAM proposal quality; SAM sometimes fails to produce the best mask.
- Inference is slow due to per-proposal candidate text generation.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReasonSeg ABO g-RefCOCO R-RefCOCO RefZOM RefCOCOm
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReasonSeg ABO-Image-ARES RefCOCOm part-only g-RefCOCO R-RefCOCO RefZOM
Metrics: gIoU cIoU mIoU
Baselines: GLCLIP CaR LISA-7B LISA-13B-LLaVA1.5 GLaMM GSVA SAM4MLLM UniRES CLIP as RNN GRES
Builds on: SAM CLIP LISA CaR
Closest papers by meaning
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.90)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.87)
- SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction (NeurIPS 2026 · similarity 0.86)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.86)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.86)
- SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.