Research map / Vision-language and multimodal / Multimodal LLMs

OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OmniEVA, an InternVL3-8B-based embodied planner with a task-adaptive gated router that selectively injects 3D positional encodings and a TE-GRPO post-training stage that rewards embodiment-feasible plans.

Problem

MLLM-based embodied systems either lack 3D geometric information or inject it statically regardless of task relevance, and their plans often ignore real robot constraints (kinematics, reachability, workspace), so they are not executable.

Approach

A Gumbel-Softmax hard gate, conditioned on sentence embeddings of the instruction and pooled visual features, decides per input whether to add patch-level 3D positional encodings to visual tokens; post-training uses GRPO with a task reward and an embodiment-feasibility reward scheduled by a curriculum coefficient.

Key results

• OmniEVA-Base (8B) reports the best scores on 2D embodied benchmarks, e.g. Where2Place 74.95 vs 73.59 for RoboBrain-2.0-32B, and VSI-bench 57.17 vs 48.83 for Gemini-2.5-Pro.
• On 3D benchmarks, OmniEVA-Base scores 62.9 on SQA3D, 30.6 on ScanQA and 94.6 CIDEr on Scan2Cap, leading three of four; on HM3D object navigation it reaches 74.2 SR and 42.5 SPL versus UniNavid's 73.7 and 37.1.
• In arm-length generalization, OmniEVA-ER reaches 81.89 average success rate versus 43.30 for OmniEVA-Base, and 80.52 on unseen arm lengths.

Surprising findings

• Soft gating of 3D features underperforms hard gating, and cross-attention fusion drops sharply (about 50 points on Scan2Cap), indicating naive attention fusion is harmful here.
• The gate activation is low for counting and state-condition prompts (6.3% to 9.0%), suggesting 3D use is task-selective.

Limitations

• Scene-level gating may give suboptimal 3D integration in heterogeneous environments, with low activation for spatial-relation tasks.
• Embodiment-aware planning does not model unmodeled physical parameters such as arm degrees of freedom or installation height.

Experimental setup

Datasets: ScanNet Matterport3D 3RScan ArkitScenes SQA3D ScanQA Scan2Cap ScanRefer

Benchmarks: Where2Place VSI-Bench PACO-LVIS RoboRefit Where2Go Where2Fit Where2Approach Where2Grasp SQA3D ScanQA Scan2Cap ScanRefer

Metrics: accuracy EM CIDEr IoU@0.25 success rate SPL

Baselines: RoboBrain2.0-32B RoboBrain2.0-7B Gemini-2.5-Pro GPT-4o Qwen2.5-VL-72B InternVL3-78B RoboPoint UniNavid 3DRS Video-3D-LLM

Builds on: GRPO InternVL3 Gumbel-Softmax 3DRS

Closest papers by meaning

• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.89)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
• EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.88)
• AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)
• NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoning (NeurIPS 2025 · similarity 0.88)
• R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action Planner (CVPR 2025 · similarity 0.88)