Research map / Vision-language and multimodal / Multimodal LLMs

Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits GRPO training of VLLMs for perception and proposes Dr. Seg, adding a Look-to-Confirm <look> reasoning format and a quantile-based Distribution-Ranked accuracy reward, achieving SOTA on reasoning segmentation, REC, detection and counting with Qwen2.5-VL-7B.

Problem

Direct transfer of GRPO training recipes from language reasoning to visual perception tasks (segmentation, detection, counting) is suboptimal due to narrow output-space exploration and poorly scaled, binary or unnormalized multi-component rewards.

Approach

Encourage broader output-space exploration by requiring the VLLM to mark visual evidence with <look> tags during reasoning, and replace raw multi-metric accuracy rewards with per-metric empirical-quantile ranks computed over a FIFO history queue so that no metric dominates the gradient.

Key results

• Dr. Seg reaches 67.8 gIoU on ReasonSeg val and 65.6 on test, versus 65.5 and 63.6 for VisionReasoner baseline (ID avg 77.1 vs 75.0).
• On COCO detection val, Dr. Seg achieves 40.1 AP vs 37.7 for VisionReasoner.
• Ablation shows raw unnormalized continuous rewards degrade ReasonSeg from 67.8 to 64.4 gIoU; Look-to-Confirm alone adds 0.6 on ReasonSeg while the combination yields the best results on both ID and OOD.

Surprising findings

• Raw continuous rewards, a natural extension, reduce OOD ReasonSeg performance below the baseline (64.7 vs 65.5).
• Naively prompting for longer chain-of-thought or Look-to-Confirm at test time lowers gIoU, so gains are not from longer responses.

Limitations

• Evaluated only on a subset of perception tasks; transfer to tracking, pose, video or point clouds is untested.
• Visual augmentation is implicit in reasoning text and adds no explicit visual feature enrichment, possibly suboptimal for dense perception.

Experimental setup

Datasets: VisionReasoner multi_object_7k_840 (LVIS, RefCOCOg, gRefCOCO, LISA++) RefCOCO RefCOCO+ RefCOCOg ReasonSeg COCO Pixmo-Count CountBench COCONut

Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReasonSeg COCONut COCO Pixmo-Count CountBench

Metrics: gIoU AP count accuracy cIoU

Baselines: VisionReasoner Seg-Zero Pixel-Think LISA GLaMM PixelLM GSVA OMG-LLaVa Qwen2.5-VL Qwen2-VL LAVT ReLA

Builds on: GRPO VisionReasoner Seg-Zero SAM2 VERL

Closest papers by meaning

• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
• ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models (ICLR 2026 · similarity 0.90)
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.90)