Research map / Vision-language and multimodal / Human and embodied modeling
MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MultiScript, a WikiHow-based multimodal benchmark for script generation from video and subsequent step prediction, and a knowledge-guided UniVL framework that injects LLM-prompted (Vicuna) instructional knowledge with an NLI-based adaptive selector.
Problem
Generative script learning typically assumes well-structured text/image preceding steps or is limited to narrow domains, which does not match real-world assistant scenarios where only raw video demonstrations are available.
Approach
Generate scripts directly from unsegmented video demonstrations and predict subsequent steps from partial videos, augmenting a multimodal encoder-decoder (UniVL) with LLM-prompted task knowledge that is weighted per decoding step by an NLI-based confidence score.
Key results
- On multimodal script generation, UniVL+Step+Knowledge achieves an average score of 43.47 vs 38.8 for UniVL and 23.92 for keyframe captioning.
- On subsequent step prediction, UniVL+Step+Knowledge reaches an average of 37.14 vs 36.19 for UniVL and 19.38 for Vicuna fine-tuned.
- Human annotators score higher than machines on semantic similarity for script generation but lower on subsequent step prediction across all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add video keyframe captions (ablation) | UniVL for Multimodal Script Generation · Multimodal Script Generation | Average: 38.8 -> 42.24 | Helped |
| Add LLM-prompted instructional knowledge (ablation) | UniVL for Multimodal Script Generation · Multimodal Script Generation | Average: 38.8 -> 39.63 | Helped |
| Add LLM-prompted instructional knowledge alongside video keyframe captions (ablation) | UniVL+Step for Multimodal Script Generation · Multimodal Script Generation | Average: 42.24 -> 43.47 | Helped |
| Add preceding-step descriptions generated by the script-generation model (ablation) | UniVL for Subsequent Step Prediction · Subsequent Step Prediction | Average: 36.19 -> 36.35 | Helped |
| Add selectively incorporated LLM-prompted instructional knowledge (ablation) | UniVL for Subsequent Step Prediction · Subsequent Step Prediction | Average: 36.19 -> 36.60 | Helped |
| Add selectively incorporated LLM-prompted instructional knowledge alongside preceding-step descriptions (ablation) | UniVL+Step for Subsequent Step Prediction · Subsequent Step Prediction | Average: 36.35 -> 37.14 | Helped |
Surprising findings
- Adding extracted keyframe captions (UniVL+Step) outperforms adding LLM knowledge (UniVL+Knowledge) on script generation average score (42.24 vs 39.63).
- Human performance on subsequent step prediction is below machine performance across all metrics, attributed to lack of domain knowledge.
Limitations
- Generated preceding steps contain errors that accumulate into subsequent step prediction.
- LLM-prompted instructional knowledge is generic and may not align with the method shown in the video.
Experimental setup
Datasets: MultiScript (built from WikiHow) YouCookII (UniVL pretraining checkpoint)
Benchmarks: MultiScript
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L BERTScore SBERT similarity
Baselines: Keyframe Caption Vicuna-13B-1.1 UniVL UniVL+Step UniVL+Knowledge T5-Large
Builds on: UniVL Katna OFA DeBERTa
Closest papers by meaning
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.88)
- Generative Multimodal Models are In-Context Learners (CVPR 2024 · similarity 0.86)
- VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought (NeurIPS 2024 · similarity 0.86)
- VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting (AAAI 2025 · similarity 0.86)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.85)
- BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.