Research map / Vision-language and multimodal / Multimodal LLMs
ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARGenSeg, which turns an MLLM (InternVL 2.5) into a segmenter by having it autoregressively output multi-scale VAR visual tokens that a frozen VQ tokenizer decodes into masks, with next-scale prediction for parallel inference.
Problem
Integrating dense pixel-level image segmentation into multimodal LLMs is hard: boundary-point representations give incomplete masks, and dedicated decoders conditioned on hidden states weaken the MLLM's pixel-level understanding and add architectural complexity; existing generation-based methods are
Approach
Segmentation is cast as image generation: the MLLM directly predicts discrete VQ tokens (via its own classification head over an extended vocabulary) and these are detokenized by a frozen VAR tokenizer, using next-scale prediction so that all tokens of a scale are generated in parallel, with no segmentation head.
Key results
- ARGenSeg achieves 82.2/77.9/78.4 cIoU on RefCOCO/+/g val, slightly above HiMTok-8B (81.1/77.1/75.8) with 402K vs 2.91M segmentation samples; the finetuned version reaches 86.3/82.3/81.7 val.
- On gRefCOCO, ARGenSeg averages 72.4 gIoU versus 66.9 for HiMTok-8B and 69.1 for SAM4MLLM-8B.
- Inference takes 1.28 s per image versus 1.89 s for HiMTok and 2.64 s for VARGPT, and 59.4 s for Emu3 on an A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune ARGenSeg on the in-domain RefCOCO/+/g training sets | ARGenSeg · RefCOCO/+/g referring segmentation | cIoU: 82.2 -> 86.3 (RefCOCO val); 77.9 -> 82.3 (RefCOCO+ val); 78.4 -> 81.7 (RefCOCOg val) | Helped |
| Add multimodal understanding data to segmentation training (ablation) | ARGenSeg · RefCOCO / RefCOCO+ / RefCOCOg | Average: 75.8 -> 79.5 | Helped |
| Replace the single-layer generation projector with a two-layer variant (ablation) | ARGenSeg · RefCOCO / RefCOCO+ / RefCOCOg | Average: 79.5 -> 75.9 | Hurt |
| Add projector-only pretraining before full fine-tuning (ablation) | ARGenSeg · RefCOCO / RefCOCO+ / RefCOCOg | Average: 79.5 -> 76.6 | Hurt |
| Use multi-scale next-scale generation instead of single-scale next-token generation (ablation) | ARGenSeg · RefCOCO / RefCOCO+ / RefCOCOg val | Average gIoU: 73.23 -> 75.87 | Helped |
| Use multi-scale next-scale generation instead of single-scale next-token generation (ablation) | ARGenSeg · Mask generation | Time: 5.50 s -> 1.28 s | Helped |
| Replace the InternVL-2.5 backbone with LLaVA-1.5 (ablation) | ARGenSeg · RefCOCO / RefCOCO+ / RefCOCOg | cIoU: 82.2 -> 72.7 (RefCOCO); 77.9 -> 68.3 (RefCOCO+); 78.4 -> 69.1 (RefCOCOg) | Hurt |
| Use semantic embeddings with a separate diffusion head instead of direct visual-token generation (ablation) | ARGenSeg · RefCOCO / RefCOCO+ / RefCOCOg | cIoU: 82.2 -> 59.0 (RefCOCO); 77.9 -> 62.7 (RefCOCO+); 78.4 -> 64.1 (RefCOCOg) | Hurt |
| Jointly fine-tune on segmentation and understanding data rather than understanding data alone | InternVL2.5-8B · POPE | score: 86.73 -> 87.57 | Helped |
| Use ARGenSeg instead of HiMTok | RefCOCO-val segmentation · RefCOCO-val | cIoU: 81.1 -> 82.2 | Helped |
| Use ARGenSeg instead of HiMTok | 256 × 256 mask generation · Inference | Time: 1.89 -> 1.28 | Helped |
Surprising findings
- Replacing the VAR tokenizer with a diffusion (DiT) head conditioned on MLLM embeddings degraded RefCOCO cIoU to 59.0, showing direct visual-token prediction matters for pixel accuracy.
- Fine-tuning on understanding data reduced some general benchmarks (e.g., TextVQA 56.98 vs 63.54 for InternVL2.5-ft), though the authors attribute this to the smaller understanding corpus.
Limitations
- Understanding performance degrades on some benchmarks after segmentation finetuning (e.g., TextVQA, MMMU, AI2D).
- Interactive segmentation is finetuned for only one epoch and lags PSALM on point and scribble prompts (65.6 vs 74.0 point).
- Extensions to image editing, depth estimation and text-to-image generation are left largely unexplored; the paper reports no error bars.
Experimental setup
Datasets: ADE20K COCO-Panoptic gRefCOCO RefCOCO/+/g LISA++ Inst.Seg AI2D ChartQA COCO-Text DocVQA LLaVA-150K GQA DVQA
Benchmarks: RefCOCO RefCOCO+ RefCOCOg gRefCOCO POPE TextVQA VQAv2 MMMU AI2D COCO-Interactive
Metrics: cIoU gIoU inference time per image accuracy
Baselines: PolyFormer VistaLLM LISA PixelLM GSVA PSALM GLaMM u-LLaVA HiMTok VARGPT Emu3 SAM4MLLM
Builds on: VAR InternVL 2.5 HiMTok VQ-VAE LISA
Closest papers by meaning
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.89)
- Text4Seg: Reimagining Image Segmentation as Text Generation (ICLR 2025 · similarity 0.88)
- SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories (CVPR 2025 · similarity 0.88)
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.87)
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.87)
- FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.