Research map / Large language models / Agents, code and math
Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Qualcomm Interactive Cooking benchmark (built on CaptainCook4D with timed instructions and mistake feedback) and LiveMamba, a streaming Mamba-130M-based multimodal LLM for live step-by-step guidance.
Problem
Multi-modal LLMs are turn-based and cannot proactively give live step-by-step task guidance, detect instruction completion, and alert users to mistakes in real time from video streams.
Approach
A streaming VLM with "when-to-say" special tokens that responds asynchronously to video frames, trained with counterfactual mistake and instruction-completion augmentations on narration data, plus an external LLM re-planner for out-of-order steps.
Key results
- LiveMamba achieves 31.5% IC-Acc on the main set streaming evaluation versus 18.9% for the best zero-shot model (Qwen2.5-VL-7B) and 23.1% for Gemini-2.5-Flash in the zero-shot table (Gemini-2.5-Flash is reported as best at 23.1).
- LiveMamba reaches mistake F1 of 0.13 on the main set and 0.19 on the advanced planning set, versus at most 0.05 for zero-shot baselines in the streaming setting.
- In turn-based evaluation, LiveMamba attains 51.0% IC-Acc versus 38.9% for Qwen2.5-VL-7B and 0.19 mistake F1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LiveMamba on the cooking guidance task | LiveMamba · Qualcomm Interactive Cooking Main Set | IC-Acc: 7.6 -> 31.5 | Helped |
| add instruction completion augmentation (ablation) | LiveMamba · Qualcomm Interactive Cooking Main Set | IC-Acc: 7.8 -> 14.3 | Helped |
| add instruction completion augmentation (ablation) | LiveMamba · Qualcomm Interactive Cooking Main Set | Mistake F1: 0.01 -> 0.13 | Helped |
| add counterfactual mistake augmentation (ablation) | LiveMamba · Qualcomm Interactive Cooking Main Set | Mistake F1: 0.05 -> 0.10 | Helped |
| add counterfactual mistake augmentation (ablation) | LiveMamba · Qualcomm Interactive Cooking Main Set | BERT: 0.558 -> 0.651 | Helped |
| add external re-planning module (ablation) | LiveMamba · Qualcomm Interactive Cooking Advanced Planning Set | Mistake F1: 0.16 -> 0.19 | Helped |
| add external re-planning module (ablation) | LiveMamba · Qualcomm Interactive Cooking Advanced Planning Set | IC-Acc: 10.9 -> 12.6 | Helped |
| use LiveMamba instead of Qwen2.5-VL-7B | Qualcomm Interactive Cooking guidance · Qualcomm Interactive Cooking Main Set, turn-based | IC-Acc: 38.9 -> 51.0 | Helped |
| fine-tune LiveMamba on the benchmark with augmentation | LiveMamba · Qualcomm Interactive Cooking Main Set, turn-based | Mistake F1: 0.06 -> 0.19 | Helped |
Surprising findings
- Zero-shot models tend to answer 'yes' to instruction completion too early, indicating poor streaming scene understanding.
- Mistake detection remains weak for all approaches (best F1 around 0.19 even for the fine-tuned model).
Limitations
- Focused only on the cooking domain.
- Subtle mistakes and order errors / missing steps in the advanced planning set remain hard for all models.
- Benchmark uses non-reactive simulation since the subject cannot respond to feedback in offline videos.
Experimental setup
Datasets: CaptainCook4D LVIS EPIC-KITCHENS SSv2 Ego4D Ego4D Goal-Step FHO
Benchmarks: Qualcomm Interactive Cooking
Metrics: IC-Acc mistake precision mistake recall mistake F1 BERT score ROUGE-L
Baselines: Gemini-2.5-Flash Qwen2.5-VL-7B Qwen2-VL-7B VideoLLaMA3-7B VideoLLM-online LLaVA-NeXT Video-ChatGPT VideoChat2 Video-LLaVA
Builds on: CaptainCook4D VideoLLM-online Mamba Q-Former
Closest papers by meaning
- LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning (NeurIPS 2024 · similarity 0.87)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.86)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.86)
- Do LLMs estimate uncertainty well in instruction-following? (ICLR 2025 · similarity 0.86)
- Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (NeurIPS 2026 · similarity 0.85)
- Can Large Language Models Understand Real-World Complex Instructions? (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.