Research map / Vision-language and multimodal / Multimodal LLMs

PRISM: Perception Reasoning Interleaved for Sequential Decision Making.

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRISM, which couples a VLM perception module and an LLM decision module through an automatic dynamic question-answering loop that produces a compact task-driven scene description, then fine-tunes the LLM policy with LoRA via behavioral cloning and PPO.

Problem

Embodied LLM agents struggle to act from raw visual observations because standalone VLM descriptions are generic and omit task-critical details, while prior VLM-LLM systems are loosely coupled or rely on predefined questions and oracle answers.

Approach

The LLM critiques the VLM's initial scene description, generates goal-oriented questions, queries the VLM for answers, and synthesizes the interaction into a compact description (instead of concatenating raw QA pairs), with the same LLM then acting as policy via LoRA adapters trained by BC and PPO.

Key results

• On ALFWorld, PRISM BC+RL reaches 80% average success rate, versus 69% for Idefics2 8B and 75% for VIPER BC+RL, and comparable to EMMA (82%) and EMAC+ (86%) which use textual supervision.
• On R2R, PRISM reaches 46.08% SR, 65.38% OSR and 43.14 SPL, exceeding the authors' DiscussNAV* reimplementation (35.52% SR) by over 10 points.
• Interactive goal-oriented perception improves average ALFWorld success from 69% (raw perception) to 75% and LLM-as-judge preference from 25.9% to 46.6%.

Surprising findings

• Conditioning the VLM directly on the goal (goal-aware perception) degrades ALFWorld success by about 4 points and produces hallucinated objects, relative to raw perception.
• Automatically generated questions underperform human-authored oracle questions by only about 2 points on average (75% vs 77%).

Limitations

• Per-step inference latency is roughly 3x higher than raw perception due to three sequential forward passes.
• Generated questions rely on general common sense rather than environment-specific priors, leaving a small gap to oracle questions.
• Evaluation is limited to ALFWorld and R2R, with ablations run only on ALFWorld and the RL phase relying on bot-generated demonstrations.

Experimental setup

Datasets: ALFWorld Room-to-Room (R2R) DialFRED

Benchmarks: ALFWorld Room-to-Room (R2R)

Metrics: success rate navigation error oracle success rate SPL RougeL METEOR BERTScore LLM-as-judge preference

Baselines: ReAct Reflexion DEPS EMMA EMAC+ MiniGPT-4 RL4VLM Idefics2 8B VIPER LangNav NavGPT DiscussNAV

Builds on: VIPER DiscussNav RL4VLM PPO LoRA

Closest papers by meaning

• Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.94)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.90)
• SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.88)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.87)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.87)