Research map / Video / Video-language understanding
Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Video-STaR proposes a self-training loop in which a video LVLM generates answers (with rationalization when needed) that are filtered by matching any available video label, enabling instruction tuning from any labeled video dataset.
Problem
Video instruction tuning datasets are small and mostly descriptive caption-style QA, and existing labeled video datasets are hard to integrate into LVLMs for instruction tuning and adaptation to novel tasks.
Approach
Cycle between prompted answer generation, label rationalization (giving the gold label and asking the model to rationalize when direct answers fail), label-based verification filtering, and retraining from the pre-trained checkpoint, using existing labels as weak supervision for any video dataset.
Key results
- On TempCompass, Video-STaR raises Video-LLaVA average accuracy from 45.7 to 50.3 (+10%), versus 47.2 for the template baseline.
- On Kinetics700-QA, accuracy rises from 50.0 to 59.9 (+20%), and on FineDiving-QA from 17.6 to 20.2 (+15%).
- On MSVD-QA, accuracy improves from 69.7 to 71.3 while the template baseline on 650K videos does not improve its gains as much (67.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Video-STaR self-training with generated answers and label verification | Video-LLaVA · TempCompass | Avg. accuracy: 45.7 -> 50.3 | Helped |
| add label rationalization (ablation) | Video-STaR · Kinetics700-QA | Accuracy: 59.8 -> 59.9 | No clear effect |
| add label rationalization (ablation) | Video-STaR · STAR-bench-QA | Accuracy: 26.6 -> 33.0 | Helped |
| add label rationalization (ablation) | Video-STaR · FineDiving-QA | Accuracy: 12.8 -> 20.2 | Helped |
| add answer generation (ablation) | Video-STaR · Kinetics700-QA | Accuracy: 50.0 -> 59.9 | Helped |
| add answer generation (ablation) | Video-STaR · STAR-bench-QA | Accuracy: 24.9 -> 33.0 | Helped |
| add answer generation (ablation) | Video-STaR · FineDiving-QA | Accuracy: 17.6 -> 20.2 | Helped |
| add label rationalization (ablation) | Video-STaR · MSVD-QA | Accuracy: 70.6 -> 71.3 | Helped |
| add label rationalization (ablation) | Video-STaR · MSRVTT-QA | Accuracy: 57.5 -> 58.2 | Helped |
| add label rationalization (ablation) | Video-STaR · TGIF-QA | Accuracy: 47.7 -> 46.8 | Hurt |
| add label rationalization (ablation) | Video-STaR · ActivityNet-QA | Accuracy: 42.2 -> 42.2 | No clear effect |
| add answer generation (ablation) | Video-STaR · MSVD-QA | Accuracy: 69.7 -> 71.3 | Helped |
Surprising findings
- Naively training on source-dataset labels converted to templates (the template baseline) did not improve, and sometimes hurt, adapted-task performance (e.g., Kinetics700-QA 50.0 to 49.5), showing labels cannot be used directly.
- Gemini-1.5 pro-vision distillation on small subsets performed worse than the base Video-LLaVA on several adapted tasks, e.g., FineDiving-QA 16.3 and Kinetics700-QA 41.9.
Limitations
- Method is computationally intensive due to repeated generation, rationalization and retraining cycles.
- Label rationalization is prone to hallucinations, with fabricated details in STAR-benchmark and FineDiving examples.
- Zero-shot benchmark gains are small and the authors argue the benchmarks are near noise limits; GPT-3.5 judge variance of about 10 points is noted.
Experimental setup
Datasets: Kinetics700 STAR-benchmark FineDiving VideoInstruct-100K LLaVA v1.5 instruction tuning data
Benchmarks: TempCompass MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA Kinetics700-QA STAR-bench-QA FineDiving-QA
Metrics: accuracy average score (1-5 GPT rating)
Baselines: Video-LLaVA mPLUG-Owl Video-ChatGPT VideoChat Video-LLaMA Gemini-1.5 distillation baseline template-based baseline
Builds on: Self-Taught Reasoner (STaR) Video-LLaVA VideoInstruct-100K LLaVA v1.5 instruction tuning
Closest papers by meaning
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.88)
- SLVMBench: Skill Learning from Video Memory (NeurIPS 2026 · similarity 0.88)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.88)
- VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.88)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.