Research map / Detection and segmentation / Open-set and few-shot detection

Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation

NeurIPS 2026 · Detection and segmentation › Open-set and few-shot detection · approach: Few-shot segmentation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Repurposes pretrained instruction-based image editing models (FLUX.2-klein, Step1X-Edit) for training-free zero-shot referring image segmentation using single-step attention priors and feature prototypes.

Problem

Zero-shot referring image segmentation currently relies on SAM proposals plus CLIP selection, which is geometry-driven and needs complex linguistic preprocessing; the question is whether implicit grounding in image editing models can localize referred objects.

Approach

Image editing DiT models show strong foreground-background feature separability at the earliest denoising timestep; the method combines cross-attention maps refined by affinity propagation (where to look) with masked-prototype cosine classification of MMSA features (what to segment), using a single denoising step.

Key results

• Step1X-Edit variant reaches 53.94 oIoU on RefCOCO val, about 5-7 points above the strongest zero-shot baseline RefAM (46.91).
• Ablation on RefCOCO oIoU: CAMs alone 33.02, adding affinity-propagated RAMs 49.56, adding FSL 53.94.
• Single denoising step inference takes about one second per image, reported as over 30x faster than RefAM.

Surprising findings

• Post-AdaLN and block-output features are less effective than MMSA features for referring segmentation, contrary to prior claims about AdaLN semantic correspondence.
• Feature separability is already strong at the first denoising timestep, so iterative denoising is unnecessary for localization.

Limitations

• Fails on fine-grained object boundaries because masks follow the edited region of the underlying editing model.
• Activates on all instances of the same category when the referent is not distinguished.
• Produces partial segmentations when the editing model does not capture the full object extent.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg

Benchmarks: RefCOCO RefCOCO+ RefCOCOg

Metrics: oIoU mIoU

Baselines: Pseudo-RIS VLM-VG Grad-CAM MaskCLIP Global-Local Ref-Diff TAS CaR SAM 3 HybridGL RefAM

Builds on: RefAM HybridGL Fisher discriminant criterion random walk over affinity matrices

Closest papers by meaning

• Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.91)
• Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.90)
• Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.89)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation (NeurIPS 2025 · similarity 0.89)
• Referring Image Editing: Object-level Image Editing via Referring Expressions (CVPR 2024 · similarity 0.88)
• IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis (AAAI 2025 · similarity 0.88)