Research map / Large language models / Agents, code and math
PlanLLM: Video Procedure Planning with Refinable Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PlanLLM, a cross-modal joint learning framework that fine-tunes a Vicuna-7B LLM (via LoRA) with a Q-Former-based mutual information maximization module to produce free-form and closed-set video procedure plans.
Problem
Weakly supervised video procedure planning from start and goal frames: existing methods decode steps as closed-set one-hot vectors, which fails to generalize to new steps or tasks, and frozen-LLM step descriptions ignore sample-specific visual states.
Approach
Uses a trainable LLM (Vicuna-7B with LoRA) that generates free-form step captions from visual-text fused step embeddings, aligned via a Q-Former mutual information maximization loss in a progressive two-stage training scheme, and fuses LLM hidden states into a conventional closed-set step decoder with a step refiner.
Key results
- On CrossTask at t=3, PlanLLM reaches 39.74 SR, 65.78 mAcc, 81.50 mIoU, surpassing the previous best SCHEMA (38.93 SR).
- On COIN at t=3, PlanLLM achieves 33.22 SR vs. 32.09 for SCHEMA.
- Cross-dataset (train COIN, test CrossTask) at t=3 gives 12.13 SR vs. 5.37 for SCHEMA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PlanLLM instead of the strongest reported baseline | CrossTask · CrossTask t=3 | SR: +0.81% | Helped |
| Use PlanLLM instead of the strongest reported baseline | CrossTask · CrossTask t=4 | SR: +3.04% | Helped |
| Use PlanLLM instead of the previous best method | NIV · NIV t=4 | SR: +0.41% | Helped |
| Use PlanLLM instead of the previous best method | COIN · COIN t=3 | SR: +1.13% | Helped |
| Use PlanLLM instead of the previous best method | COIN · COIN t=4 | SR: +1.66% | Helped |
| Add MIM to the baseline without MIM or LLM-enhanced planning (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR: +0.96/0.56% | Helped |
| Add LLM-enhanced planning to the baseline without MIM or LLM-enhanced planning (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR: +1.00/1.38% | Helped |
| Add both MIM and LLM-enhanced planning (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR: +1.71/3.68% | Helped |
| Fine-tune the LLM with progressive training instead of keeping it frozen (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR | Helped |
| Use progressive two-stage training instead of one-stage training (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR | Helped |
| Combine LLM planning and step prediction with knowledge fusion (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR | Helped |
| Use step prediction only instead of LLM-only sequence generation (ablation) | PlanLLM · CrossTask t=3 and t=4 | SR | Helped |
Surprising findings
- LLM-only free-form outputs underperform the closed-set step-only branch, attributed to caption/label inconsistency in the test set.
Limitations
- Gains over SCHEMA are small on several metrics (e.g., +0.81 SR on CrossTask at t=3), and PlanLLM does not win on NIV mIoU at t=4 or CrossTask mAcc at t=4.
- Open-vocabulary generalization is evaluated mainly via cross-dataset retrieval, where absolute scores remain low.
Experimental setup
Datasets: CrossTask COIN NIV
Benchmarks: CrossTask COIN NIV
Metrics: SR mAcc mIoU
Baselines: UPN DDN Ext-GAIL PlaTe P3IV EGPP PDPP SkipPlan LangFirst KEPP SCHEMA
Builds on: Q-Former BLIP-2 SCHEMA LoRA Vicuna
Closest papers by meaning
- Video Language Planning (ICLR 2024 · similarity 0.89)
- AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond (CVPR 2024 · similarity 0.89)
- ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning (NeurIPS 2025 · similarity 0.88)
- EVLP: Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning (ICLR 2026 · similarity 0.87)
- VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting (AAAI 2025 · similarity 0.87)
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.