Research map / Vision-language and multimodal / Multimodal LLMs
SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HLMAT, where MLLMs imitate human annotators with interactive segmentation tools via text click points, trains SegAgent on auto-generated click trajectories, and adds StaR+ policy improvement and PRM-guided greedy/tree search.
Problem
MLLMs struggle with pixel-level understanding, and existing segmentation integrations rely on implicit tokens and external pixel decoders, which alter the text output space and fail to measure intrinsic pixel comprehension.
Approach
Model segmentation as a multi-step MDP in which an MLLM outputs text-based positive/negative click coordinates that drive an external interactive segmenter (SAM or SimpleClick), with annotation trajectories simulated from existing masks by a rule-based click simulator.
Key results
- SegAgent-LLaVA+SAM reaches 79.20 on RefCOCO val and 74.80 on RefCOCOg val(U), comparable to LISA and GSVA-level LLM-based methods.
- On ThinObject5K, StaR+ raises IoU from a baseline of 71.45 to 86.57.
- With PRM tree search (K=3), performance gains of 7.43 on DIS and 10.59 on ThinObject over PRM alone are reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the MLLM on human-like annotation trajectories to create SegAgent | LLaVA-v1.5-7B or Qwen-VL-7B with interactive segmentation · RES (refCOCO, refCOCO+, refCOCOg) | cIoU | Helped |
| apply one round of StaR+ policy improvement after trajectory SFT | SegAgent · ThinObject5K | IoU: 71.45 compared to 86.57 | Helped |
| use PRM alone (K=1) rather than fixed-step greedy decoding (ablation) | SegAgent · DIS5K | IoU: +2.36 | Helped |
| use PRM alone (K=1) rather than fixed-step greedy decoding (ablation) | SegAgent · ThinObject5K | IoU: +4.09 | Helped |
| add tree search with K=3 to PRM (ablation) | SegAgent · DIS5K | IoU: +7.43 | Helped |
| add tree search with K=3 to PRM (ablation) | SegAgent · ThinObject5K | IoU: +10.59 | Helped |
| add PRM-guided stopping instead of fixed-step decoding (ablation) | SegAgent · DIS5K, 7 steps | IoU: 68.22 -> 75.43 | Helped |
| use a blue rather than green mask overlay (ablation) | SegAgent · HRES | mIoU: 0.749 -> 0.750 | No clear effect |
| use a red rather than green mask overlay (ablation) | SegAgent · HRES | mIoU: 0.749 -> 0.749 | No clear effect |
| represent coordinates in [0, 1000) rather than [0, 1) (ablation) | SegAgent-LLaVA · HRES | mIoU: 0.749 -> 0.747 | No clear effect |
| use Qwen Box rather than no initial box (ablation) | SegAgent with SAM · refCOCO val | cIoU: 77.81 -> 78.01 | No clear effect |
| use SegAgent-Qwen's predicted Self Box rather than no initial box (ablation) | SegAgent with SAM · refCOCO val | cIoU: 77.81 -> 77.85 | No clear effect |
Surprising findings
- LLaVA outperforms Qwen-VL on mask refinement despite Qwen's stronger general vision-language performance, which the authors attribute to the Q-former losing spatial locality.
- Mask color and coordinate format (decimal vs. integer) have negligible effect on mIoU.
Limitations
- Failures from incorrect initial localization and inaccurate boundary point coordinates in refinement steps.
- Fine-grained pixel-level localization of MLLMs is identified as the main bottleneck.
- Only a single round of StaR+ was explored; more complex search strategies (BFS, MCTS) were left for future work.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg DIS5K ThinObject5K HQSeg-44K
Benchmarks: RefCOCO RefCOCO+ RefCOCOg HRES (DIS5K, ThinObject5K)
Metrics: cIoU mIoU MSE MAE Pearson Spearman
Baselines: LISA PixelLM PerceptionGPT GSVA SAM4MLLM PolyFormer Qwen box + SAM GLaMM
Builds on: LISA StaR SAM SimpleClick LLaVA-v1.5 Qwen-VL
Closest papers by meaning
- PixelLM: Pixel Reasoning with Large Multimodal Model (CVPR 2024 · similarity 0.90)
- IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (CVPR 2026 · similarity 0.88)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.88)
- MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness (NeurIPS 2025 · similarity 0.88)
- ARGenSeg: Image Segmentation with Autoregressive Image Generation Model (NeurIPS 2025 · similarity 0.88)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.