Research map / Video / Video-language understanding
Improving LLM Video Understanding with 16 Frames Per Second
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces F-16, a 7B video LLM that processes video at 16 FPS using a high-frame-rate MLP aligner that compresses each 1-second clip into visual tokens, plus a training-free variable-frame-rate decoding method.
Problem
Existing video LLMs sample at low frame rates (FPS<=2) or fixed frame counts, losing rapidly changing visual information needed for fine-grained motion and high-speed sports understanding.
Approach
A high-frame-rate aligner concatenates the features of all frames in a 1-second window and maps them with a 3-layer MLP to the LLM token space, initialized from a single-image aligner via block-matrix decomposition so the output initially averages per-frame representations; a variable-frame-rate decoding trims or repeats frames at test time.
Key results
- F-16-7B reaches 65.0% on Video-MME, the best among 7B models in the paper's table, and 84.1% on NExT-QA and 37.2 on TemporalBench short.
- On high-speed sports, F-16 at 16 FPS gets 64.1% gymnastics accuracy and 86.5% diving accuracy versus 48.5% and 76.0% at 1 FPS, and beats GPT-4o (76.7 F1) and Gemini-1.5-Pro (80.6 F1) on NBA with 92.9 F1.
- Variable-frame-rate decoding with frame repetition at 8 FPS gives 63.9% on Video-MME versus 65.0% at 16 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 16 FPS instead of 1 FPS for sports-task training and fine-tuning | F-16 · Gymnastics | accuracy: 48.5 -> 64.1 | Helped |
| Use 16 FPS instead of 1 FPS for sports-task training and fine-tuning | F-16 · Diving48 | mean phase accuracy: 76.0 -> 86.5 | Helped |
| Use 16 FPS instead of 1 FPS for sports-task training and fine-tuning | F-16 · NBA | F1: 87.1 -> 92.9 | Helped |
| Use 16 FPS instead of 1 FPS for sports-task training and fine-tuning | F-16 · SoccerNet | accuracy: 55.4 -> 57.7 | Helped |
| Apply max pooling after rather than before the modality aligner at FPS=16 (ablation) | F-16 · Video-MME | accuracy (Avg.): 60.8 -> 65.0 | Helped |
| Apply max pooling after rather than before the modality aligner at FPS=1 (ablation) | F-16 · Video-MME | accuracy (Avg.): 62.3 -> 62.9 | Helped |
| Use frame repetition rather than trimming the aligner for 8 FPS inference (ablation) | F-16 · Video-MME | accuracy (Avg.): 62.3 -> 63.9 | Helped |
| Reduce decoding rate from 16 FPS to 8 FPS using frame repetition (ablation) | F-16 · Video-MME | accuracy (Avg.): 65.0 -> 63.9 | Hurt |
| Replace the linear aligner with a 2D-CNN (ablation) | F-16 · Video-MME | accuracy (Avg.): 61.5 -> 57.7 | Hurt |
| Replace the linear aligner with a 3D-CNN (ablation) | F-16 · Video-MME | accuracy (Avg.): 61.5 -> 57.2 | Hurt |
| Replace max pooling with a learnable linear spatial-compression layer (ablation) | F-16 · Video-MME | accuracy (Avg.): 61.5 -> 58.8 | Hurt |
| Replace the first linear layer with self-attention when training with LoRA (ablation) | F-16 · Video-MME | accuracy (Avg.): 61.5 -> 61.8 | Helped |
Surprising findings
- Max-pooling before the aligner hurts the 16 FPS model (60.8 vs 65.0 with post-pooling), while it helps slightly at 1 FPS.
- Trimming aligner weights for low-frame-rate inference degrades performance (62.3 at 8 FPS), whereas repeating frames nearly preserves the 16 FPS result.
Limitations
- Slight performance drop on long-video benchmarks (Video-MME Long, LongVideoBench, MLVU) versus LLaVA-Video-7B due to sparser sampling within windows.
- High-frame-rate encoding significantly slows inference because every frame must pass through the image encoder.
- Sports data (NBA) is manually annotated and not released, limiting reproducibility.
Experimental setup
Datasets: LLaVA-Video-178K LLaVA-Hound NExT-QA ActivityNet-QA PerceptionTest FineGym Diving48 SoccerNet NBA game videos
Benchmarks: Video-MME NeXT-QA TemporalBench MotionBench VideoVista MLVU LongVideoBench FineGym Diving48 SoccerNet NSVA
Metrics: accuracy multiple binary accuracy F1
Baselines: GPT-4o Gemini-1.5-Pro Qwen2-VL-7B LLaVA-OV-7B LLaVA-Video-7B NVILA-7B MiniCPM-V2.6-8B VideoLLaMA2-7B VideoChat2-HD-7B
Builds on: LLaVA-OneVision LLaVA-Video SigLIP Qwen2
Closest papers by meaning
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.91)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.90)
- Understanding Long Videos with Multimodal Language Models (ICLR 2025 · similarity 0.90)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.