Research map / Video / Video-language understanding
Step Differences in Instructional Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates paired-video instruction tuning data from HowTo100M/HT-Step using LLM-written difference QA, then trains a projection-only video-conditioned LLM (StepDiff) to compare a reference and candidate video, evaluated on a new step-difference benchmark.
Problem
Comparing a user's how-to video against a reference video to describe and rank fine-grained step differences; existing VCLMs and captioning work only handle single videos or synthetic/static image pairs.
Approach
Automatically build paired-video QA data by pairing clips of the same keystep from distinct HowTo100M videos, synthesizing step descriptions from captions, detections and ASR, and having Llama 2 write difference QA; then train only a projection module of a frozen video-LLM to jointly reason over two videos.
Key results
- On DiffMCQ, StepDiff reaches 0.104 top-1 accuracy-style score (Acc) versus 0.115 for VCLM (AnyMAL) on the same column, so it is not the top scorer there; on DiffCap it reaches CIDEr 0.223 and BLEU 0.541 as the best of listed baselines.
- On DiffRank, StepDiff achieves Kendall's tau 0.215 (vs. 0.199 for VCLM LLaVA... best baseline 0.172 Socratic step desc. and 0.170 InternVideo embedding), outperforming all baselines.
- Ablation: removing interleaved training data reduces DiffCap CIDEr from 0.541 to 0.499 and DiffMCQ accuracy from 0.104 to 0.094.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train StepDiff on paired-video difference QA data | StepDiff · DiffCap | BLEU: 0.217 -> 0.223 | Helped |
| train StepDiff on paired-video difference QA data | StepDiff · DiffCap | CIDEr: 0.115 -> 0.104 | No clear effect |
| train StepDiff on paired-video difference QA data | StepDiff · DiffCap | ROUGE-L: 0.210 -> 0.215 | Helped |
| train StepDiff on paired-video difference QA data | StepDiff · DiffMCQ | accuracy: 0.497 -> 0.541 | Helped |
| train StepDiff on paired-video difference QA data | StepDiff · DiffRank | Kendall's τ: 0.170 -> 0.181 | Helped |
| add interleaved ASR training data (ablation) | StepDiff · DiffMCQ | accuracy: 0.499 -> 0.541 | Helped |
| add interleaved ASR training data (ablation) | StepDiff · DiffRank | Kendall's τ: 0.185 -> 0.181 | No clear effect |
| filter generated QA training data (ablation) | StepDiff · DiffRank | Kendall's τ: 0.120 -> 0.181 | Helped |
| use a 70B LLM instead of a 13B LLM (ablation) | StepDiff · DiffCap | CIDEr: 0.124 -> 0.104 | Hurt |
Surprising findings
- Joint video-embedding models outperform LLM-based baselines on difference ranking, attributed to direct embedding similarity rather than YES-token likelihood.
- Models trained on in-domain interleaved HowTo100M data generally beat VCLMs not trained for multi-video input, and naively adding reference video tokens to existing VCLMs degrades performance.
Limitations
- LLM-generated training QA is noisy and can hallucinate details not present in the video.
- The model can hallucinate content in comparative reasoning, a failure case shown in the paper.
- Training data is largely third-person HowTo100M cooking videos, with evaluation limited to cooking-themed instructional videos.
Experimental setup
Datasets: HowTo100M HT-Step StepDiff benchmark (evaluation)
Benchmarks: StepDiff DiffCap StepDiff DiffMCQ StepDiff DiffRank
Metrics: BLEU CIDEr ROUGE-L top-1 accuracy Kendall's tau
Baselines: CLIP InternVideo BLIP-2 LLaVA-1.5 AnyMAL IDEFICS Socratic models
Builds on: LLaVA AnyMAL InternVideo Llama 2 HowTo100M
Closest papers by meaning
- Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision (ICLR 2025 · similarity 0.87)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.86)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.86)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.86)
- Detours for Navigating Instructional Videos (CVPR 2024 · similarity 0.86)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.