Research map / Video / Video-language understanding
Progress-Aware Video Frame Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces progress-aware frame-level video captioning, with ProgressCaptioner (LLaVA-OV-7B fine-tuned via SFT and DPO on a pseudo-labeled FrameCap dataset) and the FrameCapEval benchmark.
Problem
Existing image and video captioners produce either isolated per-image descriptions or one event-level caption per clip, missing fine-grained, frame-by-frame action progression and suffering from temporal hallucination.
Approach
Generate a caption per frame that is both accurate and progress-aware by pseudo-labeling frame pairs and sequences with an ensemble of VLMs, filtering them with LLM/VLM-based progression detection and caption matching, and training a 7B model with SFT then DPO in two stages.
Key results
- ProgressCaptioner (7B) reaches 37.3 caption matching and 73.6 progression detection on HowToChange, outperforming GPT-4o (32.4/64.2) and Gemini-1.5-Pro (31.4/63.8) on FrameCapEval.
- In the user study, ProgressCaptioner has the highest top-1 selection rate at 31.6%, 2 to 3.6 times the best comparably sized open-source models.
- On NExT-QA ATP-Hard, ProgressCaptioner plus GPT-4o reaches 61.8% accuracy versus 58.8% for LLaVA-OV plus GPT-4o and 58.4% for VideoAgent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProgressCaptioner instead of GPT-4o for frame-wise captioning | video frame captioning on HowToChange · HowToChange (HTC) | caption matching accuracy: 32.4 -> 37.3 | Helped |
| Use ProgressCaptioner instead of Qwen2-VL for frame-wise captioning | video frame captioning on HowToChange · HowToChange (HTC) | progression detection accuracy: 69.6 -> 73.6 | Helped |
| Use ProgressCaptioner instead of Gemini-1.5-Pro for frame-wise captioning | video frame captioning on COIN · COIN | caption matching accuracy: 25.0 -> 32.3 | Helped |
| Use ProgressCaptioner instead of Qwen2-VL for frame-wise captioning | video frame captioning on COIN · COIN | progression detection accuracy: 70.8 -> 66.1 | Hurt |
| Use ProgressCaptioner instead of GPT-4o for frame-wise captioning | video frame captioning on Penn Action and Kinetics · Penn Action and Kinetics (Penn&K) | caption matching accuracy: 18.2 -> 31.3 | Helped |
| Use ProgressCaptioner instead of GPT-4o for frame-wise captioning | video frame captioning on Penn Action and Kinetics · Penn Action and Kinetics (Penn&K) | progression detection accuracy: 63.2 -> 63.7 | Helped |
| Use ProgressCaptioner-generated frame-wise captions instead of LLAVA-OV captions | video QA with GPT-4o on NExT-QA (ATP-Hard) · NExT-QA (ATP-Hard) temporal subset | Acc@T: 53.4 -> 58.1 | Helped |
| Use ProgressCaptioner-generated frame-wise captions instead of LLAVA-OV captions | video QA with GPT-4o on NExT-QA (ATP-Hard) · NExT-QA (ATP-Hard) | Acc@All: 58.8 -> 61.8 | Helped |
| Replace uniformly sampled frames with keyframes selected using ProgressCaptioner captions | SlowFast on Kinetics Temporal · Kinetics Temporal | top-1 accuracy: +1.7% | Helped |
| Replace uniformly sampled frames with keyframes selected using ProgressCaptioner captions | X3D-XS on Kinetics Temporal · Kinetics Temporal | top-1 accuracy: +1.7% | Helped |
Surprising findings
- Full-sequence input to leading VLMs produces brief descriptions with temporal misalignment, while single-frame captioning loses temporal context.
- Qwen2-VL excels at progression detection (reducing hallucination) but yields less detailed captions, lowering caption matching accuracy.
Limitations
- Evaluation depends on LLM and VLM evaluators (Llama-3.1-70B, Gemini-1.5-Pro) whose reliability is only partly validated in supplementary material.
- Frames sampled at a fixed 1 FPS; the task is limited to action videos from specific datasets.
Experimental setup
Datasets: HowToChange COIN Penn Action Kinetics NExT-QA
Benchmarks: FrameCapEval NExT-QA (ATP-Hard) Kinetics Temporal
Metrics: caption matching accuracy progression detection accuracy user study selection rate top-1 accuracy video QA accuracy
Baselines: GPT-4o Gemini-1.5-Pro Idefics2 VILA Qwen2-VL LLaVA-Video LLaVA-OV VFC VideoAgent
Builds on: LLaVA-OneVision direct preference optimization SigLIP Qwen2
Closest papers by meaning
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.90)
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.89)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.88)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.