Research map / Vision-language and multimodal / Multimodal LLMs

Artemis: Structured Visual Reasoning for Perception Policy Learning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Artemis, a GRPO-based RL method that trains Qwen2.5-VL-3B to reason via structured (label, bbox) visual evidence instead of linguistic chains, jointly on grounding and detection.

Problem

Language-based chain-of-thought in RL for perception policies often hurts perception performance and generalizes poorly across perception tasks and visual domains.

Approach

Replace natural-language thinking with structured visual reasoning: the think block outputs label–bounding-box pairs, rewarded via Hungarian-matched IoU and ROUGE-based label similarity with key-object weighting, plus outcome rewards for format and answer, all optimized by GRPO in one unified model.

Key results

• On RefCOCO testB, Artemis (3B) reaches 65.6 average accuracy versus 62.4 for VLM-R1 and 61.0 for Qwen2.5-VL-3B.
• On COCO2017 val, Artemis achieves mAP 31.0 vs 15.4 for Qwen2.5-VL-3B and AP50 48.0 vs Perception-R1's 46.7.
• Zero-shot Pixmo-Count val accuracy 81.4 and LISA test 69.8, outperforming VisionReasoner (70.1 on Pixmo val is higher, so Artemis leads only on test/LISA relative to other baselines) and UniVG-R1 (59.7).

Surprising findings

• Linguistic reasoning reduced zero-shot Pixmo-Count val accuracy to 13.4, far below no-reasoning (65.2), while structured visual reasoning reached 81.4.
• Detection-only training sharply reduced RefCOCOg val grounding accuracy to 51.6.

Limitations

• Structured visual reasoning depends on high-quality label-box annotations, with reasoning objects derived from existing RefCOCO and COCO annotations.
• Fixed thresholds (eta=0.8, tau_IoU=0.8, tau_sim=0.9) were chosen empirically with limited tuning; uniform reward weighting was not exhaustively tuned.

Experimental setup

Datasets: Artemis-RFT (built from MS-COCO, RefCOCO/+/g via LLaVA-665k, Vision-R1 region annotations) COCO detection annotations (80k cold-start) RefCOCO/+/g COCO2017 val LISA Pixmo-Count MATHGLANCE MMBench MMVet MMStar ScienceQA SEEDBench

Benchmarks: RefCOCO RefCOCO+ RefCOCOg COCO2017 val Pixmo-Count LISA test MATHGLANCE MMBench MMVet MMStar ScienceQA SeedBench

Metrics: top-1 accuracy at IoU 0.5/0.75/0.95 mAP AP50 AP75 AR100 counting accuracy accuracy

Baselines: Qwen2.5-VL-3B VLM-R1 Perception-R1 Vision-R1 VisionReasoner UniVG-R1 No-Thinking-RL Qwen2-VL LLaVA-OV DeepSeek-VL2-Tiny Griffon MDETR

Builds on: GRPO VLM-R1 Perception-R1 Qwen2.5-VL Vision-R1

Closest papers by meaning

• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
• SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.90)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding (ICML 2026 · similarity 0.90)
• Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning (ICML 2026 · similarity 0.90)
• VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning (ICLR 2026 · similarity 0.90)