Research map / Video / Action recognition
Video Action Differencing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Video Action Differencing (VidDiff), a benchmark (VidDiffBench, 549 video pairs) for finding fine-grained skill differences between two videos of the same action, plus a zero-shot agentic pipeline (LLM proposer, CLIP-based Viterbi frame localizer, VLM frame differencer) that beats GPT-4o and other LMMs in the closed setting.
Problem
Comparing two videos of the same action to identify subtle, skill-relevant differences (for coaching and skill learning) is poorly supported: existing video difference work focuses on feature visualization or coarse comparisons between different actions.
Approach
Decompose video action differencing into three zero-shot stages: LLM-proposed difference candidates, CLIP-based temporal sub-action localization with Viterbi decoding to align keyframes across both videos, and VLM frame-pair differencing on the localized frames.
Key results
- In the closed setting, VidDiff reaches 56.3% average accuracy versus 53.5% for GPT-4o and 57.7% for Gemini-1.5-Pro, with the hard split near chance for all models.
- In the open setting, VidDiff achieves 42.1 average recall@N_diff, versus 41.7 for GPT-4o, 35.6 for Claude-3.5-Sonnet, and 7.2 for Qwen2-VL-7B.
- With ground-truth frame alignment, zero-shot VLM frame differencing reaches 78.6% on easy but drops to 51.0% on hard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the VidDiff workflow instead of raw-frame GPT-4o | Closed-set video action differencing · VidDiffBench, Avg | accuracy: 53.5 -> 56.3 | Helped |
| Use the VidDiff workflow instead of GPT-4o | Open-set video action differencing · VidDiffBench, Average | recall@N_diff: 41.7 -> 42.1 | Helped |
| Use VidDiff rather than GPT-4o | Open-set video action differencing · VidDiffBench, Medium | recall@N_diff: 41.5 -> 37.9 | Hurt |
| Replace full frame localization with random frame retrieval (ablation) | VidDiff closed-set frame localization · VidDiffBench, Easy | accuracy: 62.7 -> 50.1 | Hurt |
| Add Viterbi-based decoding to frame localization (ablation) | VidDiff closed-set frame localization · VidDiffBench, Easy | accuracy: 57.4 -> 62.7 | Helped |
| Use ground-truth timestamps for frame localization instead of the method's retrieval (ablation) | VidDiff closed-set frame localization · VidDiffBench, Easy | accuracy: 62.7 -> 78.6 | Helped |
| Use VQA on ground-truth-aligned frames (ablation) | Zero-shot VLM frame differencing · VidDiffBench, Easy | accuracy: 78.6 | Helped |
Surprising findings
- Even with ground-truth timestamps, zero-shot VLMs perform near chance on the hard split (51.0%), indicating image-level fine-grained comparison is a bottleneck.
- Gemini-1.5-Pro has the best closed-set average accuracy yet much lower open-set recall than GPT-4o, because its main weakness is generating the correct differences.
Limitations
- Closed-set results on medium and hard splits are not significantly better than random for most models.
- Relies on general foundation models that may struggle with domain-specific or fine-grained comparisons.
- Benchmark uses trimmed, atomic action videos and largely fixed camera angles in some categories.
Experimental setup
Datasets: HuMMan Ego-Exo4D JIGSAWS FineDiving
Benchmarks: VidDiffBench
Metrics: accuracy (closed setting) recall@N_diff (open setting)
Baselines: GPT-4o Gemini-1.5-Pro Claude-3.5-Sonnet LLaVA-Video Qwen2-VL-7B
Builds on: CLIP Viterbi algorithm Video-MME
Closest papers by meaning
- FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · similarity 0.86)
- JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts (AAAI 2025 · similarity 0.86)
- ViDiC: Video Difference Captioning (NeurIPS 2026 · similarity 0.86)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.85)
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.85)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.