Research map / Vision-language and multimodal / Multimodal LLMs

EVLP: Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EVLP, a unified LLM-based vision-language planner that jointly generates language actions and one-step discrete image subgoals, trained with dynamics-based pretraining and a reinforced SFT objective using a dynamic-aware reward.

Problem

Long-horizon embodied manipulation needs coordinated textual reasoning and visual subgoal imagination, but existing multimodal planners lack a unified generation framework, causing inconsistencies between language plans and visual outcomes.

Approach

Combines a SigLIP semantic encoder with a MAGVIT2-style tokenizer's spatial features in an LLM (Qwen2.5-1.5B) that predicts all discrete image tokens in one forward pass instead of AR or diffusion denoising; adds inverse/forward dynamics pretraining and a policy-gradient loss with a dynamic-region matching reward added to the SFT likelihood objective.

Key results

• On LoHoRavens block tasks, EVLP reaches 79.4 success on Stacking vs 63.9 for PERIA (best baseline).
• Ablation: removing the forward dynamics pretraining drops success rate on Meeting Preparation from 67.6 to 26.8.
• One-step generation gives LPIPS 0.046 vs 0.197 for the autoregressive variant (EVLP-AR).

Surprising findings

• RL-only training without the likelihood term collapses completely (0.0 success rate), so the SFT term is essential for stability.
• Removing the generation task (text-only planning) lowers success rate from 67.6 to 53.7, showing multimodal training helps language planning too.

Limitations

• Meeting Preparation is an in-house benchmark and dataset that is not released, limiting reproducibility.
• Pretraining and RSFT results are reported with a 1.5B model; the 7B scaling claims are mostly about sampling speed rather than planning accuracy.
• Reward function depends on frame-differencing region detection, which may be brittle for scenes with little or noisy motion.

Experimental setup

Datasets: LoHoRavens Meeting Preparation (in-house) BridgeData v2 (real-world subset)

Benchmarks: LoHoRavens Meeting Preparation BridgeData v2 real-world subset

Metrics: success rate language accuracy LPIPS SSIM

Baselines: CLIPort PAR EmbodiedGPT SuSIE CoTDiffusion PERIA

Builds on: SigLIP Open-MAGVIT2 Qwen2.5 DDPO GRPO PERIA

Closest papers by meaning

• ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning (NeurIPS 2025 · similarity 0.89)
• Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.89)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
• GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.89)
• Video Language Planning (ICLR 2024 · similarity 0.89)
• TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models (ICLR 2026 · similarity 0.88)