Research map / Vision-language and multimodal / Multimodal LLMs
Using Left and Right Brains Together: Towards Vision and Language Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vision-Language Planning (VLP), which runs parallel language planning (zero-shot CoT with ChatGPT) and vision planning (video diffusion future-frame generation with selectors) and fuses them in an LMM or BERT decision maker, evaluated on video QA, captioning, and action prediction.
Problem
LLMs and LMMs plan only in language space and lack visual/spatial imagination, so their planning misses environmental details that vision-based reasoning could capture.
Approach
Combine a language plan (zero-shot CoT decomposition by ChatGPT) with a vision plan (future frames generated by a video diffusion model, filtered by coarse and fine selectors), then let an LMM or BERT decision maker answer via multi-round conversation and a voting mechanism.
Key results
- On STAR, VLP (LLaVA-based, 4 input frames + 1 generated) reaches 50.5% average accuracy vs. 47.4% for LLaVA and 44.6% for SEVILA.
- On BDD-X captioning, VLP achieves BLEU-4 35.7, CIDEr 256.7, METEOR 31.1, surpassing ADAPT (34.6, 247.5, 30.6).
- On BAIR action planning with 1 initial + 2 end frames, Baseline+VP reduces summed RMSE from 14.39 to 13.85 cm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add language planning to LLAVA (ablation) | LLAVA on STAR · STAR | accuracy (Avg.): 47.4 -> 50.4 | Helped |
| Add vision planning to LLAVA (ablation) | LLAVA on STAR · STAR | accuracy (Avg.): 47.4 -> 49.6 | Helped |
| Add vision planning and language planning to LLAVA (ablation) | LLAVA on STAR · STAR | accuracy (Avg.): 47.4 -> 50.5 | Helped |
| Add vision planning to ADAPT (ablation) | ADAPT on BDD-X · BDD-X | BLEU-4: 34.6 -> 35.7 | Helped |
| Add language planning to ADAPT (ablation) | ADAPT on BDD-X · BDD-X | BLEU-4: 34.6 -> 35.2 | Helped |
| Add both vision planning and language planning to ADAPT (ablation) | ADAPT on BDD-X · BDD-X | BLEU-4: 34.6 -> 36.2 | Helped |
| Add the Coarse Selector (CS) to vision planning (ablation) | VLP on STAR · STAR | accuracy (Avg.): 48.0 -> 49.6 | Helped |
| Add the Fine Selector (FS) to vision planning (ablation) | VLP on STAR · STAR | accuracy (Avg.): 45.6 -> 49.6 | Helped |
| Add voting to the vision-planning decision maker (ablation) | VP on STAR · STAR | accuracy (Avg.): 48.3 -> 49.6 | Helped |
| Add voting to the language-planning decision maker (ablation) | LP on STAR · STAR | accuracy (Avg.): 46.3 -> 50.4 | Helped |
| Increase the number of generated frames from 1 to 2 or 3 (ablation) | VLP on STAR · STAR | accuracy (Avg.): 49.6; 49.7; 49.7 | No clear effect |
| Use ground-truth future frames instead of generated frames (ablation) | VLP on STAR · STAR | accuracy (Prediction): 50.0 -> 57.5 | Helped |
Surprising findings
- Using ground-truth future frames on STAR yields much larger gains (e.g., 52.7 avg) than Stable Video Diffusion-generated frames, showing generation quality is the bottleneck.
- Generated frames from general-domain Stable Video Diffusion underperform domain-specific DMVFN (trained on Cityscapes/KITTI) on BDD-X captioning.
Limitations
- Performance is bottlenecked by the quality of video generation models, which produce artifacts and unreliable long sequences.
- GPT-4V case studies were qualitative only due to usage limits; no quantitative GPT-4V results.
- Language plans from ChatGPT can introduce noise, and the multi-round voting pipeline adds cost.
Experimental setup
Datasets: STAR NExT-QA BDD-X BAIR Cityscapes KITTI
Benchmarks: STAR NExT-QA BDD-X BAIR
Metrics: accuracy BLEU-4 CIDEr METEOR RMSE tolerant accuracy
Baselines: ViperGPT Flamingo-80B VFC InternVideo BLIP-2 SEVILA LLaVA S2VT S2VT++ SAA WAA ADAPT
Builds on: chain-of-thought zero-shot CoT LLaVA ADAPT BLIP-2
Closest papers by meaning
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.88)
- PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation (NeurIPS 2024 · similarity 0.88)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.87)
- CoFFT: Chain of Foresight-Focus Thought for Visual Language Models (NeurIPS 2025 · similarity 0.87)
- Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.86)
- DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.