Research map / Vision-language and multimodal / Multimodal LLMs

EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EARL, a two-stage analysis-guided RL framework that first generates an egocentric interaction analysis, then uses an Analysis-guided Feature Synthesizer and GRPO-trained MLLM with SAM2 to answer queries and ground masks.

Problem

Egocentric human-environment interaction reasoning and pixel grounding with MLLMs remains inaccurate and poorly generalizing; existing approaches treat analysis, answering and grounding as independent subtasks.

Approach

A coarse-to-fine two-stage design where a global interaction descriptor from the analysis stage is fused into the answering/grounding stage via an Analysis-guided Feature Synthesizer (self-attention over analysis features added residually to multimodal embeddings), with the response stage trained by GRPO using format, answer-relevance and IoU rewards.

Key results

• EARL reaches 65.48% cIoU on Ego-IRGBench test grounding, +8.37 over the best prior RL-based baseline (Seg-Zero 57.11%).
• Answering CIDEr 6.682 on the test set vs 5.000 for the best general baseline (+1.682).
• On OOD EgoHOS grounding, EARL achieves 38.21% cIoU, +0.58 over Sa2VA-8B (37.63%).

Surprising findings

• Seg-Zero has very low analysis CIDEr (0.049) despite strong grounding, suggesting grounding gains do not transfer to description quality.
• In the hyperparameter study, down-weighting the format reward (lambda_f=0.5) raised grounding cIoU to 69.87 while lowering answer and analysis scores.

Limitations

• Small or heavily occluded objects lead to inaccurate boxes and masks.
• OOD evaluation limited to grounding only, on EgoHOS.
• Designed for egocentric scenes; may not generalize to exocentric benchmarks.

Experimental setup

Datasets: Ego-IRGBench HOI4D EgoHOS

Benchmarks: Ego-IRGBench EgoHOS

Metrics: METEOR CIDEr cIoU IoU

Baselines: Qwen3-VL-4B Qwen3-VL-8B Qwen2.5-VL-3B Qwen2.5-VL-7B InternVL2.5-7B EgoThinker ANNEXE Sa2VA-8B Sa2VA-InternVL3-8B UniPixel-7B Seg-R1-3B Seg-R1-7B

Builds on: GRPO DAPO Qwen2.5-VL SAM2 Seg-Zero Seg-R1

Closest papers by meaning

• ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · similarity 0.91)
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.89)
• EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.87)
• DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.87)
• EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.86)
• Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models (NeurIPS 2026 · similarity 0.86)