Research map / Vision-language and multimodal / Multimodal LLMs

Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a Qwen3-VL-8B-based agent with SFT then GRPO to orchestrate six heterogeneous vision experts via multi-turn calls, using dynamic visual memory archiving to keep rollouts tractable.

Problem

General-purpose MLLMs are weak at precise low-level visual execution (segmentation, dense small-object detection, counting), while specialist models are isolated task solvers; existing tool-augmented MLLMs use manual prompts or single-turn decisions and cannot self-correct.

Approach

Decouple an MLLM agent's perception, reasoning and decision-making from execution by training it to harness a suite of heterogeneous vision experts over multi-turn interactions, with on-policy GRPO made tractable by archiving past visual feedback into textual placeholders and splitting trajectories into step-wise single-turn samples.

Key results

• On ReasonSeg, VisHarness reaches 68.6 average gIoU/cIoU (test gIoU 73.2), above Seg-Zero (56.2) and SAM3-Agent (63.0 test gIoU).
• On Dense200, VisHarness (Qwen3-VL-8B) achieves 85.5 F1@IoU0.5 and 69.3 mIoU, versus 78.4 for Rex-Omni and 76.9 for SEED1.5-VL.
• On gRefCOCO, VisHarness with 0.7% of training data reaches 63.58 average, below GSVA(FT) at 65.60 but above LISA(FT) at 62.90.

Surprising findings

• Qwen3-VL-8B with the same visual experts but no training scores only 16.5 F1@IoU0.5 on Dense200, showing an untrained MLLM cannot select experts well.
• SFT alone mechanically imitates tool-use patterns; GRPO reduces unnecessary SplitImageIntoPatches and SuperResolution calls on gRefCOCO.

Limitations

• Counting results on REC-8K are slightly worse than the latest specialized counting method (Dense200 and gRefCOCO also trail GSVA on some metrics).
• Dense200 mF1@0.95 remains low (14.7), indicating weak strict localization.
• Details of SFT data, training hyperparameters and compute are deferred to a supplementary material not included in the text.

Experimental setup

Datasets: gRefCOCO REC-8K

Benchmarks: gRefCOCO ReasonSeg Dense200 REC-8K

Metrics: gIoU cIoU F1@IoU0.5 F1@IoU0.95 mIoU MAE RMSE

Baselines: LISA GSVA Seg-Zero SAM3-Agent Rex-Omni Grounding DINO CountGD CounTX Qwen3-VL-8B (w/ tools) Qwen3.5-9B SEED1.5-VL

Builds on: GRPO LISA OpenThinkIMG LLaVA-Plus ToolRL

Closest papers by meaning

• MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning (ICLR 2026 · similarity 0.89)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.88)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.88)
• Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding (ICML 2026 · similarity 0.88)
• Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design (CVPR 2026 · similarity 0.88)