Research map / Detection and segmentation / Open-set and few-shot detection

TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models

NeurIPS 2026 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary 3D and world perception

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TextRegion, a training-free method that pools frozen image-text model patch values within SAM2 masks (with global-patch filtering and multi-resolution features) to produce text-aligned region tokens for open-vocabulary segmentation, referring expression comprehension, and multi-object grounding.

Problem

Image-text models like CLIP operate at the global image level and their patch embeddings lack spatial coherence and boundary alignment, limiting region-level text matching for segmentation, grounding, and retrieval.

Approach

Use SAM2 masks to restrict the final-layer [CLS]/query attention to patches inside each region, pooling value features into region tokens comparable to text embeddings; add region-consistency filtering of global patches and multi-resolution feature maps. No training is required.

Key results

• Open-world segmentation: with OpenCLIP ViT-H/14 and SAM2, TextRegion averages 49.5 mIoU vs. 48.6 for Trident across six benchmarks.
• Zero-shot ReC with CLIP ViT-L/14@336: 48.7 RefCOCO val and 55.8 RefCOCOg val, above FGVP's 46.1 and 54.5.
• Ablation: with CLIP ViT-B/16 on COCO-Stuff, global patch removal plus multi-resolution raises mIoU from 24.0 to 27.9 for Perception Encoder B/16 (the Perception Encoder row) and from 26.9 to 28.7 for CLIP B/16.

Surprising findings

• Multi-resolution features reduce SigLIP2-SO400M/16 performance, unlike other models, which the authors attribute to more global patches in large models.
• Global patch removal alone has limited effect for CLIP but significantly helps when combined with multi-resolution features.

Limitations

• Region token quality depends on the accuracy of the SAM2 (or other) masks; poor masks yield poor tokens.
• Effectiveness is bounded by the visual-language features of the underlying image-text model, including weak spatial awareness or language understanding.
• Reasoning segmentation performance is bottlenecked by the text encoder for long queries; the paper relies on LLaVA-generated short queries.

Experimental setup

Datasets: PASCAL VOC 2012 PASCAL Context COCO-Stuff COCO-Object Cityscapes ADE20K RefCOCO RefCOCO+ RefCOCOg ReasonSeg

Benchmarks: PASCAL VOC 2012 PASCAL Context COCO-Stuff COCO-Object Cityscapes ADE20K RefCOCO RefCOCO+ RefCOCOg ReasonSeg

Metrics: mIoU accuracy (IoU>=0.5) gIoU cIoU

Baselines: MaskCLIP SCLIP ResCLIP ProxyCLIP LaVG LPOSS+ Trident CLIPtrase FGVP Red Circle CPT CLIP-DINOiser

Builds on: CLIP MaskCLIP SAM2 ProxyCLIP Trident LLaVA AnyResolution

Closest papers by meaning

• GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.88)
• Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
• Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.86)
• Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.86)
• Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation (CVPR 2024 · similarity 0.86)
• Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.86)