Research map / Video / Video-language understanding

VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting

AAAI 2025 · Video › Video-language understanding · approach: Video-language pretraining · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VG-TVP, a zero-shot LLM-driven multimodal procedural planning framework that fuses video captions from instructional videos into visually grounded text plans and generates matching video plans with ModelScope, evaluated on a new Daily-PP dataset.

Problem

Generating cohesive text and video procedural plans for high-level goals, with temporal coherence, textual/visual informativeness and plan accuracy, especially for tasks lacking instructional videos.

Approach

Uses LLM zero-shot reasoning plus video captioning (VLog) and text-to-video diffusion (ModelScope) bridged by a Fusion of Captioning step (FoC) and V2T/T2V bridges, so text plans are aligned with IV captions and video plans are conditioned on the revised text plans; unseen tasks are composed from captions of related seen tasks.

Key results

• On 50 seen tasks, VG-TVP is preferred over Mistral-7B-q4 on textual informativeness at 58.00% win rate and over the TIP model on textual informativeness at 71.43% win rate.
• On 15 unseen tasks, VG-TVP achieves visual informativeness win rates from 46.67% to 86.67% against all LLM baselines.
• Automatic BLEU/METEOR scores are mixed: baselines often match or beat VG-TVP text plans, which the authors attribute to a single reference being inadequate for daily tasks.

Measured effects reported in the paper

Surprising findings

• BLEU and METEOR favor baseline text plans for several LLMs, contradicting the human preference results.
• GPT-3.5 beats VG-TVP on temporal coherence and plan accuracy for unseen tasks in human evaluation.

Limitations

• Relies on human preference surveys with 28 participants on small task sets (50 seen, 15 unseen), limiting statistical strength.
• Automatic text metrics (BLEU, METEOR) do not favor VG-TVP, and the LLM judge uses the same GPT family as a baseline.
• VLog captioning errors (mislabeling objects) can propagate into misordered steps.

Experimental setup

Datasets: Daily-PP CrossTask COIN ProceL WikiPlan&RecipePlan

Benchmarks: Daily-PP

Metrics: human win-tie-lose preference (textual informativeness, visu BLEU METEOR CLIP mean similarity score

Baselines: Llama2-7B-q4 Llama2-7B-q8 Llama2-13B-q4 Llama2-13B-q8 Mistral-7B-q4 Mistral-7B-q8 GPT-3.5 TIP model

Builds on: Text-to-Image Prompting (TIP) VLog ModelScope zero-shot chain-of-thought

Closest papers by meaning

• PlanLLM: Video Procedure Planning with Refinable Large Language Models (AAAI 2025 · similarity 0.87)
• EVLP: Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning (ICLR 2026 · similarity 0.87)
• Video Language Planning (ICLR 2024 · similarity 0.87)
• Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.87)
• AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond (CVPR 2024 · similarity 0.87)
• VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.86)