Research map / Vision-language and multimodal / Robot action and navigation
APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APIVOT, a Qwen3-VL-8B planner that interleaves language reasoning with latent visual thoughts (imagined future-state embeddings) trained via a three-stage SFT curriculum, evaluated on KitchenWorlds long-horizon kitchen planning tasks.
Problem
Long-horizon robot planning requires jointly reasoning about semantic task structure and geometric feasibility (free space, collisions), which text-only VLM planners handle poorly.
Approach
A VLM planner generates latent visual thoughts (K hidden-state tokens aligned via cosine loss to encoded ground-truth future scene images) interleaved with text reasoning, and learns via a three-stage SFT curriculum to decide adaptively when to use visual versus language reasoning.
Key results
- APIVOT achieves average task success 0.419 vs 0.338 for the best VLM baseline (Gemini-ER-1.5) and 0.329 for VLM-TAMP.
- APIVOT retains 91% of the always-visual-thought variant's performance while reducing token usage by 39%.
- Text-only SFT on the same backbone improves average success only from 0.188 to 0.24, versus 0.419 for APIVOT.
Surprising findings
- Text-only SFT yields only modest gains compared with visual thoughts, suggesting the visual thoughts drive most of the improvement.
- APIVOT shows one of the largest ID-to-OOD performance drops, attributed to limited supervision for visual thoughts in cluttered scenes.
Limitations
- Evaluated only in the KitchenWorlds simulator, not real-world or visually realistic settings.
- Latent visual representations trained within the task distribution may not generalize to real-world geometry and appearance.
- Modality selection is learned via SFT rather than optimized directly for task success.
Experimental setup
Datasets: KitchenWorlds-generated demonstrations via PDDLStream
Benchmarks: KitchenWorlds Containment KitchenWorlds Sorting KitchenWorlds Storing Leftovers (held-out) KitchenWorlds out-of-distribution long horizon and more complex
Metrics: task success rate token usage relative ID-OOD gap
Baselines: Gemini-3.1-Pro Gemini-ER-1.5 Qwen3-VL-8B-Instruct Qwen3-VL-8B-Thinking FastDownward Reflect-VLM VLM-TAMP
Builds on: Qwen3-VL PDDLStream VLM-TAMP Reflect-VLM BLADE
Closest papers by meaning
- World-aware Planning Narratives Enhance Large Vision-Language Model Planner (NeurIPS 2025 · similarity 0.90)
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.90)
- ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning (NeurIPS 2025 · similarity 0.89)
- Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.88)
- Video Language Planning (ICLR 2024 · similarity 0.88)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.