Research map / Vision-language and multimodal / Robot action and navigation

I-Perceive: A Foundation Model for Vision-Language Active Perception

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Pretraining and data augmentation for VLN · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes I-Perceive, a VGGT-style geometric transformer that takes a start image, context images, and a language instruction and directly predicts a 6D target camera pose, trained on ~232K automatically generated language-viewpoint samples.

Problem

Language-conditioned active perception in large indoor scenes: predicting a feasible camera viewpoint that fulfills an open-ended natural-language perception intent from sparse observations, which existing task-specific NBV, VLN, and VLM approaches handle poorly.

Approach

A dual-pathway model that couples a pretrained VLM (Qwen3-VL-4B) with a VGGT-style multi-view geometric transformer: a latent target-frame token attends to observed frames, and VLM key-value features from multiple layers are injected into semantic tokens via gated cross-attention, so language conditions geometric reasoning end to end.

Key results

• I-Perceive achieves the best constraint violation, pose error, and voxel-matching F1 on HSSD and Replica, with about 1.1 s (HSSD) and 1.3 s (Replica) mean inference time.
• In judge ranking, I-Perceive achieves pairwise winning rates of 87.5%-94.1% over all baselines, with GPT and human rankings consistent.
• Closed-loop execution from a single start image progressively improves viewpoint quality and nearly matches the full-context oracle at the final step, although the model was not trained for closed-loop operation.

Surprising findings

• Closed-loop active perception emerges from variable-length context acceptance without explicit closed-loop training.
• Pose error can be misleading under multi-modal valid viewpoints, so the authors rely on voxel-matching F1 and judge ranking.

Limitations

• Constraint checking only detects collisions and does not correct generated viewpoints.
• Training pipeline depends on object annotations from scanning datasets and simulation, limiting open-ended targets.
• Relies heavily on pretrained VLM and geometric foundation models.

Experimental setup

Datasets: ScanNet CA-1M HSSD Replica

Benchmarks: HSSD (52 tasks, in-distribution) Replica (89 tasks, out-of-distribution)

Metrics: constraint violation translation error rotation error voxel-matching F1 mean judge rank pairwise winning rate runtime

Baselines: NBV+L VLN+P (NaVILA) VLM prompting (GPT-5.4) SigLIP2 variant

Builds on: VGGT Qwen3-VL NaVILA Wan2.2

Closest papers by meaning

• ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning (ICML 2026 · similarity 0.90)
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.89)
• Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning (ICLR 2025 · similarity 0.89)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.88)
• CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models (CVPR 2025 · similarity 0.88)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.87)