Research map / Video / Video-language understanding
SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SF2T, a self-supervised fragment-level fine-tuning method for Video-LLMs (counting, consistency verification, localization, disorder detection, rearrangement) and the FineVidBench benchmark for fine-grained video understanding.
Problem
Video-LLMs struggle with fine-grained understanding such as visual dynamics and video details, and existing SFT/RLHF approaches depend on costly manual annotation.
Approach
Generate training labels automatically from frame-level fragments of videos via proxy tasks (counting, consistency verification, localization, disorder detection and rearrangement), so no human annotation is needed, and use them as a complement to SFT. A new benchmark, FineVidBench, evaluates scene- and fragment-level understanding.
Key results
- On FineVidBench scene-level Action, Qwen2-VL improved from 68.18 to 73.86 with SF2T.
- SF2T improved MVBench for LLaVA-NeXT-Video from 36.84 to 42.92 and MiniCPM-V 2.6 from 40.23 to 56.02.
- Combining SF2T with SFT raised Speed accuracy for LLaVA-NeXT-Video from 22.35 to 32.01 but SFT alone still leads on most scene-level tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SF2T fine-tuning | LLaVA-NeXT-Video · FineVidBench | Action: 37.31 -> 48.67 | Helped |
| Add SF2T fine-tuning | MiniCPM-V 2.6 · FineVidBench | Action: 43.37 -> 65.91 | Helped |
| Add SF2T fine-tuning | VideoLLaMA 2.1 · FineVidBench | Action: 63.26 -> 67.42 | Helped |
| Add SF2T fine-tuning | Qwen2-VL · FineVidBench | Action: 68.18 -> 73.86 | Helped |
| Add SF2T fine-tuning | Video-LLMs · FineVidBench Speed | Speed | Helped |
| Add SF2T fine-tuning | LLaVA-NeXT-Video · MVBench | performance: 36.84 -> 42.92 | Helped |
| Add SF2T fine-tuning | MiniCPM-V 2.6 · Video-MME (no subtitle) | performance: 43.17 -> 53.19 | Helped |
| Use motion-salient area sampling instead of random sampling (ablation) | SF2T on Qwen2-VL · FineVidBench Action | accuracy: 70.31 -> 73.86 | Helped |
| Use motion-salient area sampling instead of uniform interval sampling (ablation) | SF2T on Qwen2-VL · FineVidBench Action | accuracy: 71.67 -> 73.86 | Helped |
| Use motion-salient area sampling instead of keyframe sampling (ablation) | SF2T on Qwen2-VL · FineVidBench Action | accuracy: 72.11 -> 73.86 | Helped |
| Use long-range temporal modeling instead of multi-scale modeling (ablation) | SF2T on Qwen2-VL · FineVidBench Action | performance: 73.86 -> 69.38 | Hurt |
| Use short-range temporal modeling instead of multi-scale modeling (ablation) | SF2T on Qwen2-VL · FineVidBench Action | performance: 73.86 -> 71.40 | Hurt |
Surprising findings
- Models generally detect sequence errors (Adjust-or-Not) but perform poorly at rearranging frames.
- Frame-count accuracy often declines as the number of frames increases, except for Video-CCAM.
Limitations
- SF2T currently lags behind supervised fine-tuning on scene-level tasks because its objective is not fully aligned with them.
- Only one epoch of training and four models evaluated; performance is not optimized for maximum results.
- FineVidBench QA pairs are auto-generated with partial manual review and use limited video sources.
Experimental setup
Datasets: Something-Something V2 Moments in Time Ego4D MVBench Video-MME MLVU
Benchmarks: FineVidBench MVBench Video-MME MLVU
Metrics: accuracy
Baselines: LLaVA-NeXT-Video MiniCPM-V 2.6 VideoLLaMA 2.1 Qwen2-VL ShareGPT4Video Video-CCAM Base (SFT)
Builds on: LoRA MGSampler
Closest papers by meaning
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.92)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.