Research map / Video / Video-language understanding
FastVID: Dynamic Density Pruning for Fast Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FastVID is a training-free, inference-time pruning framework for Video LLMs that combines dynamic temporal segmentation (transition-similarity-based) with density-peak token merging and attention-based token selection, reaching 90.3% token pruning on LLaVA-OneVision while keeping 98.0% accuracy.
Problem
Video LLMs incur high inference costs because of the large number of redundant video tokens; existing image and video token pruning methods fail to exploit spatiotemporal redundancy while preserving temporal order and visual context.
Approach
Partition the frame sequence into temporally ordered segments via adjacent-frame transition similarity, then within each segment merge tokens around density-peak anchors (keeping their positions for RoPE) and add [CLS]-attention-selected salient tokens, all before the LLM prefill.
Key results
- On LLaVA-OneVision-7B, FastVID prunes 90.3% of video tokens (9.7% retained), reduces TFLOPs to 4.04 (8.3%), and keeps 98.0% of vanilla average accuracy (58.1 vs 59.3).
- FastVID achieves 7.1x LLM prefill speedup on LLaVA-OneVision (67.2 ms vs 476.3 ms total) on an A100 GPU.
- On LLaVA-Video at 9.5% retention, FastVID reaches 94.7% average accuracy versus 90.7% for VisionZip*.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FastVID instead of PruneVID* at 9.7% retention | LLaVA-OneVision-7B · LLaVA-OneVision, Avg. Acc. | Avg. Acc.: 56.6 -> 58.1 | Helped |
| Use FastVID instead of PruneVID* at comparable retention | Qwen2-VL · VideoMME | Overall: 56.3 -> 60.6 | Helped |
| Use FastVID instead of PruneVID* at comparable retention | Qwen2.5-VL · VideoMME | Overall: 59.3 -> 62.8 | Helped |
| Combine DTM and ATS with d=0.4 instead of using ATS alone (ablation) | FastVID on LLaVA-OneVision · VideoMME | accuracy: 55.0 -> 57.3 | Helped |
| Combine DTM and ATS with d=0.4 instead of using DTM alone (ablation) | FastVID on LLaVA-OneVision · VideoMME | accuracy: 55.9 -> 57.3 | Helped |
| Add the SigLIP head to ATS (ablation) | FastVID on LLaVA-OneVision · VideoMME | accuracy: 56.3 -> 57.3 | Helped |
| Use DySeg instead of fixed-interval segmentation (ablation) | FastVID on LLaVA-OneVision · MVBench | accuracy: 55.1 -> 55.9 | Helped |
| Use DySeg instead of cluster-based segmentation (ablation) | FastVID on LLaVA-OneVision · MVBench | accuracy: 53.2 -> 55.9 | Helped |
| Use density-based DTM instead of uniform token merging (ablation) | FastVID on LLaVA-Video · Avg. Acc. | Avg. Acc.: 91.5% -> 94.7% | Helped |
| Use density-based DTM instead of cluster-based token merging (ablation) | FastVID on LLaVA-Video · Avg. Acc. | Avg. Acc.: 91.8% -> 94.7% | Helped |
| Increase sampled frames from 32 to 320 using FastVID at r=10% (ablation) | LLaVA-OneVision · VideoMME | accuracy: 58.6 -> 61.4 | Helped |
Surprising findings
- Query-agnostic pruning causes notable accuracy drops on long-frame models (LLaVA-Video, Qwen2-VL, Qwen2.5-VL) despite outperforming baselines.
- Increasing sampled frames to 320 with FastVID at comparable token budget improves VideoMME accuracy above the vanilla 32-frame model (61.4 vs 58.6).
Limitations
- Query-agnostic pruning loses query-relevant tokens for long videos, causing accuracy drops on LLaVA-Video and Qwen-VL models.
- Requires reattaching the SigLIP head to compute [CLS] attention for ATS in SigLIP-based models; uses pseudo [CLS] tokens for Qwen-VL.
- Notable accuracy gap to vanilla under extreme compression on long-frame models.
Experimental setup
Datasets: MVBench LongVideoBench MLVU VideoMME ANet-QA
Benchmarks: MVBench LongVideoBench MLVU VideoMME ANet-QA
Metrics: accuracy average accuracy retention TFLOPs prefill latency
Baselines: FastV VisionZip DyCoke PruneVID PyramidDrop SparseVLM LLaVA-PruMerge
Builds on: VisionZip FastV density peaks clustering PruneVID DyCoke
Closest papers by meaning
- HoliTom: Holistic Token Merging for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.92)
- V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.92)
- F³-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis (AAAI 2024 · similarity 0.90)
- Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning (AAAI 2026 · similarity 0.90)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.