Research map / Video / Video-language understanding
FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models
Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, Yu Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compute similarity between spatially corresponding tokens of adjacent frames instead of random pairs (ablation) | FrameFusion (Llava-Video-7B) · VideoNIAH/VideoMME/NExT-QA avg | average score: 58.5 -> 64.9 | Improved |
| compute similarity between corresponding tokens of adjacent frames instead of adjacent patches (ablation) | FrameFusion (Llava-Video-7B) · VideoNIAH/VideoMME/NExT-QA avg | average score: 60.2 -> 64.9 | Improved |
| use cosine similarity instead of inner product (ablation) | FrameFusion · VideoNIAH/VideoMME/NExT-QA avg | average score: 61.7 -> 64.5 | Improved |
| use cosine similarity instead of Minkowski-1/2 distance (ablation) | FrameFusion · VideoNIAH/VideoMME/NExT-QA avg | average score: 63.1 -> 64.5 | Improved |
| merge at layer 2 instead of layer 14 (ablation) | FrameFusion (Llava-Video-7B) · 50% budget avg | average score: 50.8 -> 65.2 | Improved |
| merge at layer 2 instead of layer 13 (ablation) | FrameFusion (Llava-Video-7B) · 50% budget avg | average score: 59.2 -> 65.2 | Improved |
| merge before pruning rather than prune before merge (ablation) | FrameFusion (Llava-Video-7B) · VideoNIAH/VideoMME/NExT-QA avg | average score: 63.0 -> 63.6 | Improved |
| cascaded instead of non-cascaded merging at equal FLOPs (ablation) | FrameFusion (Llava-Video-7B) · VideoNIAH/VideoMME/NExT-QA avg | average score: 63.4 -> 65.4 | Improved |
| raise S_threshold from 0.6 to 0.9 (ablation) | FrameFusion (NVILA-8B) · VideoNIAH | accuracy: 73.6 -> 72.2 | Mixed |
| raise N_threshold from 0.1 to 0.3 (ablation) | FrameFusion (NVILA-8B) · VideoNIAH | accuracy: 73.3 -> 74.2 | No clear effect |
| replace FastV with FrameFusion at 30% budget | FastV (Llava-Video-7B) · 9-benchmark average | average score: 57.9 -> 64.8 | Improved |
| replace PruMerge with FrameFusion at 30% budget | PruMerge (NVILA-8B) · 9-benchmark average | average score: 52.0 -> 65.9 | Improved |
Later papers that compared with this method
6 later accepted papers report FrameFusion as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.
- ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs (ICLR 2026 · better on 9 of 9 shared results)
- AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding (CVPR 2026 · better on 3 of 3 shared results)
- Contribution-aware Token Compression for Efficient Video Understanding via Reinforcement Learning (AAAI 2026 · better on 3 of 3 shared results)
- Enhancing Visual Token Representations for Video Large Language Models via Training-free Spatial-Temporal Pooling and Gridding (ICLR 2026 · better on 1 of 2 shared results)
- MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs (AAAI 2026 · better on 1 of 1 shared result)
- Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs (ICCV 2025 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- HoliTom: Holistic Token Merging for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.91)
- Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.91)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.91)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
