Research map / Vision-language and multimodal / CLIP and zero-shot
Text4Seg: Reimagining Image Segmentation as Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text4Seg casts image segmentation as text generation by encoding masks as 16x16 semantic descriptor sequences (with row-wise run-length encoding) that MLLMs produce via LoRA fine-tuning, with SAM as an optional refiner.
Problem
Integrating dense image segmentation into multimodal LLMs without extra decoders, extra losses, or architectural changes, while keeping output sequences short enough for practical inference.
Approach
Represent a segmentation mask as a purely textual sequence of semantic descriptors, one per flattened image patch (16x16 grid), and compress repeated labels row-wise with R-RLE so the MLLM generates masks as ordinary next-token text, with no segmentation decoder.
Key results
- Text4Seg with InternVL2-8B and SAM-H refiner reaches 75.4 average cIoU on refCOCO(+/g), above Groundhog's 74.2 and GLaMM's 75.6 is trained on a much larger dataset; LLaVA-1.5-13B variant reaches 76.2.
- R-RLE cuts semantic descriptor length by 74% (583 to 154 tokens on average) and speeds up inference about 3x with no performance loss relative to full-length descriptors.
- Text4Seg with InternVL2-8B reaches 87.1% average Acc@0.5 on referring expression comprehension, up from 82.9% for the InternVL2-8B baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Text4Seg segmentation training to LLaVA-1.5-13B | LLaVA-1.5-13B · refCOCO (+/g) | average cIoU: 72.8 -> 76.2 | Helped |
| add Text4Seg segmentation training to InternVL2-8B | InternVL2-8B · refCOCO (+/g) | average Acc@0.5: 82.9% to 87.1% | Helped |
| add Text4Seg segmentation training to LLaVA-1.5-7B | LLaVA-1.5-7B · visual question answering benchmarks | VQA score: 78.0 -> 76.6; 61.7 -> 60.2; 50.6 -> 50.9; 68.4 -> 68.1; 55.0 -> 55.0; 85.4 -> 84.2 | No clear effect |
| add SAM-L mask refinement (ablation) | Text4Seg · refCOCO (+/g) RES | average cIoU: 73.5 to 79.1 | Helped |
| replace no refiner with SAM-H (ablation) | Text4Seg InternVL2-8B · refCOCO (+/g) | average cIoU: 67.5 -> 75.4 | Helped |
| replace no refiner with CRF (ablation) | Text4Seg InternVL2-8B · refCOCO (+/g) | average cIoU: 67.5 to 70.1 | Helped |
| increase descriptor resolution from 16×16 to 32×32 without a refiner (ablation) | Text4Seg InternVL2-8B · refCOCO (+/g) | average cIoU: 67.5 -> 71.4 | Helped |
| replace full-length descriptors with I-RLE (ablation) | Text4Seg · refCOCO | cIoU: 74.2 to 70.4 | Hurt |
| replace full-length descriptors with I-RLE (ablation) | Text4Seg · refCOCO+ | cIoU: 68.0 to 64.7 | Hurt |
| use R-RLE instead of full-length descriptors (ablation) | Text4Seg · refCOCO and refCOCO+ | cIoU: 74% reduction; 3× | No clear effect |
| use SAM-L instead of SAM-H (ablation) | Text4Seg · refCOCO (+/g) RES | cIoU: 79.1 vs 79.3; 72.8 vs 72.6; 74.2 vs 74.6 | No clear effect |
| use image-wise RLE instead of row-wise RLE (ablation) | Text4Seg · refCOCO and refCOCO+ | cIoU: 74.2 to 70.4; 68.0 to 64.7 | Hurt |
Surprising findings
- Without any refiner, 16x16 semantic descriptors give only 67.5 average cIoU, below LISA; the gap is closed mainly by the SAM refiner (to 75.4) or 32x32 descriptors (71.4).
Limitations
- Without a mask refiner, 16x16 descriptors trail LISA and GSVA on RES; a SAM or CRF refiner is needed for competitive results.
- Higher-resolution descriptors (32x32) increase token length and computational cost.
- Open-vocabulary segmentation still lags specialised models such as SAN and OVSeg.
Experimental setup
Datasets: refCLEF refCOCO refCOCO+ refCOCOg grefCOCO COCO-Stuff LLaVA-v1.5-mix665k
Benchmarks: refCOCO refCOCO+ refCOCOg grefCOCO ADE20K (A-150) PASCAL Context 59 PASCAL VOC 20 VQAv2 GQA VisWiz ScienceQA TextVQA
Metrics: cIoU gIoU Acc@0.5 mIoU
Baselines: LISA GSVA GLaMM Groundhog PixelLM LaSagnA Shikra Groma VisionLLM NEXT-Chat
Builds on: LLaVA-1.5 LISA ViT Run-Length Encoding
Closest papers by meaning
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
- ARGenSeg: Image Segmentation with Autoregressive Image Generation Model (NeurIPS 2025 · similarity 0.88)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.88)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.87)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.