Research map / Vision-language and multimodal / Robot action and navigation
PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PERIA, which aligns an MLLM's vision encoder for perception, instruction-tunes it to reason stepwise sub-instructions while emitting latent [IMG] tokens that condition an image-editing diffusion model to imagine subgoal images, and adds a consistency loss between the two plans for long-horizon manipulation.
Problem
Long-horizon manipulation with general, complex instructions requires decomposition into sub-tasks, but text-only language planning can be ambiguous and vision-only subgoal planning lacks logical reasoning and semantic consistency.
Approach
Jointly trains an MLLM (Vicuna-7B with LoRA) and a latent diffusion image editor so the MLLM outputs both language sub-instructions and latent image tokens that guide subgoal image generation, with a CLIP-style consistency loss tying generated images to instructions.
Key results
- PERIA reaches 63.9% stacking and 89.3% VIMA-BENCH rearrange success vs 48.6% and 76.1% for the best prior baselines EmbodiedGPT and CoTDiffusion, respectively (per Table 1 rows as reported).
- Language planning token accuracy of 97.6 vs 65.9 for EmbodiedGPT and 58.2 for PAR.
- FID on Blocks 10.2 vs 13.1 for CoTDiffusion; removing the consistency loss raises FID to 12.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine language and vision planning rather than use vision planning alone | PERIA · Blocks&Bowls (Stacking, Sort, Matching) | success rate (%): 47.9±6.0 -> 63.9±5.8; 44.3±7.6 -> 65.0±6.4; 56.6±5.2 -> 72.3±7.1 | Helped |
| combine language and vision planning rather than use vision planning alone | PERIA · Letters (Shape, Orders, Spell) | success rate (%): 46.1±6.5 -> 60.6±5.2; 53.9±4.8 -> 65.2±6.7; 52.7±6.2 -> 71.1±7.5 | Helped |
| combine language and vision planning rather than use vision planning alone | PERIA · VIMA-BENCH (Rearrange, Follow, Constraint) | success rate (%): 51.2±6.3 -> 74.8±6.0; 54.5±7.3 -> 67.2±7.8; 76.1±5.6 -> 89.3±4.9 | Helped |
| remove perception pretraining (ablation) | PERIA · Reasoning accuracy | Token accuracy: 97.6 -> 80.2 | Hurt |
| remove perception pretraining (ablation) | PERIA · Reasoning accuracy | Semantic similarity: 0.98 -> 0.83 | Hurt |
| remove vision planning from joint training (ablation) | PERIA · Reasoning accuracy | Token accuracy: 97.6 -> 83.7 | Hurt |
| remove vision planning from joint training (ablation) | PERIA · Reasoning accuracy | Semantic similarity: 0.98 -> 0.79 | Hurt |
| remove consistency alignment (ablation) | PERIA · Blocks | FID: 10.2 -> 12.3 | Hurt |
| remove consistency alignment (ablation) | PERIA · Letters | FID: 13.5 -> 14.2 | Hurt |
| remove consistency alignment (ablation) | PERIA · VIMA-BENCH | FID: 11.4 -> 15.9 | Hurt |
| use holistic language-and-vision planning instead of either modality alone (ablation) | PERIA low-level policy · Tasks with varying horizon lengths | success rate | Helped |
| increase the number of [IMG] tokens (ablation) | PERIA · Instruction-image consistency | CLIP consistency score | Mixed |
Surprising findings
- Removing the perception pretraining or vision planning both substantially reduce language-planning accuracy, indicating the joint image-generation loss itself improves reasoning.
Limitations
- High training and inference time cost of joint MLLM and diffusion training and sampling.
- Baselines SuSIE were given oracle stepwise instructions, and the Spell task relies on an oracle engine with a fixed list of answers.
Experimental setup
Datasets: LoHoRavens VIMA-BENCH Letters (new scenario)
Benchmarks: LoHoRavens (Blocks&Bowls) Letters VIMA-BENCH
Metrics: success rate token accuracy semantic similarity (CLIP) FID CLIP consistency score
Baselines: CLIPort LISA PAR EmbodiedGPT SuSIE CoTDiffusion
Builds on: LLaVA InstructPix2Pix CLIPort SuSIE CoTDiffusion
Closest papers by meaning
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · similarity 0.91)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.88)
- EVLP: Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning (ICLR 2026 · similarity 0.88)
- VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models (NeurIPS 2025 · similarity 0.88)
- Using Left and Right Brains Together: Towards Vision and Language Planning (ICML 2024 · similarity 0.88)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.