Research map / Vision-language and multimodal / Multimodal LLMs

ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ACTIVE-o3, a GRPO-based RL framework that trains Qwen2.5-VL-7B to propose parallel zoom-in regions (sensing) for a task model, improving small/dense object grounding and interactive segmentation without region-level supervision.

Problem

MLLMs act as passive consumers of fixed views; existing zoom-in search (e.g., GPT-o3) is sequential and inaccurate, and there is no systematic framework or benchmark for MLLM-based active perception on small/dense objects.

Approach

Decouple one MLLM into a sensing module that outputs K parallel region proposals in a single forward pass and a task module applied to each crop, and train the sensing policy with GRPO using a dual-form reward (heuristic format/non-overlap/area/coverage terms plus task-aware AP/AR or mIoU from the task model).

Key results

• On LVIS dense, ACTIVE-o3 reaches AP_s 4.3 and AR_s 5.5 versus 1.6/2.0 for Qwen2.5-VL, and AR_l improves by +14.6.
• On SODA-A AP_s, ACTIVE-o3 reaches 9.2 versus 0.7 for Qwen2.5-VL; on SODA-D AP_s 15.1 versus 2.1.
• On general benchmarks, ACTIVE-o3 scores 69.7 on RealWorldQA versus 67.9 for its Qwen2.5-VL-7B-Instruct initialization, with MME 2316 versus 2308.

Surprising findings

• Direct RFT (GRPO without the sensing module) performs well on large objects (AP_l 23.4 vs 20.9) but poorly on small objects, indicating that resolution limits, not RL alone, drive small-object gains.
• Qwen2.5-VL-CoT degrades as zoom-in budget increases on ThinObjects (mIoU 0.561 at budget 3), while ACTIVE-o3 improves to 0.863.

Limitations

• Domain gap: task model struggles with specialized categories in remote sensing, limiting task-aware reward quality.
• Constrained action space: at most three zoom-in regions per step, axis-aligned boxes only, no memory across steps.
• Interactive segmentation uses an oracle task model rather than a real one, so results are not end-to-end.

Experimental setup

Datasets: LVIS SODA-A SODA-D ThinObjects

Benchmarks: LVIS small/dense grounding SODA-A SODA-D ThinObjects MMBench MME RealWorldQA

Metrics: AP AR mIoU

Baselines: Grounding DINO Qwen2.5-VL-7B Qwen2.5-VL-CoT V* + GDINO Direct RFT PPO GMPO GPG Reinforce++

Builds on: GRPO DeepSeek-R1 Qwen2.5-VL V* Grounding DINO

Closest papers by meaning

• I-Perceive: A Foundation Model for Vision-Language Active Perception (NeurIPS 2026 · similarity 0.90)
• PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.89)
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.88)
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Real-World Reinforcement Learning of Active Perception Behaviors (NeurIPS 2025 · similarity 0.88)
• $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.88)