Research map / Vision-language and multimodal / Multimodal LLMs
Libra-Merging: Importance-redundancy and Pruning-merging Trade-off for Acceleration Plug-in in Large Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Libra-Merging, a training-free visual token compression plug-in for LVLMs that selects target tokens via position-driven intervals and merges non-target tokens with importance-informed grouped merging plus an information compensation token.
Problem
Visual token compression in large vision-language models reduces inference cost, but existing methods face an importance-redundancy trade-off in choosing target tokens and a pruning-merging trade-off in handling non-target tokens.
Approach
Select one highest-attention token per spatial interval to avoid redundant targets, then merge non-target tokens into targets only when similar (importance-weighted softmax), and condense dissimilar ones into a single importance-weighted compensation token; apply hierarchically across layers.
Key results
- On LLaVA-1.5-7B, Libra-Merging at 47% Flops (1.78T vs 3.82T vanilla) reaches 62.1 average vs 62.5 vanilla and 61.8 for FastV/Turbo at 56% Flops.
- On LLaVA-NeXT-8B, 47% tokens yields 71.7 average, slightly above the 71.6 vanilla score.
- On VideoLLaMA-2 Video-MME at 34% Flops, Libra-Merging gets 48.8 overall vs 46.5 FastV and 47.9 Turbo (w/o subs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Libra-Merging at 47% Flops instead of vanilla inference | LLaVA-1.5-7B · image-text benchmarks | Avg: 62.5 -> 62.1 | Hurt |
| Apply Libra-Merging at 37% Flops instead of vanilla inference | LLaVA-1.5-7B · image-text benchmarks | Avg: 62.5 -> 61.9 | Hurt |
| Apply Libra-Merging at 47% Flops instead of vanilla inference | LLaVA-1.5-13B · image-text benchmarks | Avg: 65.9 -> 65.9 | No clear effect |
| Apply Libra-Merging at 37% Flops instead of vanilla inference | LLaVA-1.5-13B · image-text benchmarks | Avg: 65.9 -> 65.4 | Hurt |
| Apply Libra-Merging at 47% Flops instead of vanilla inference | LLaVA-NeXT-8B · image-text benchmarks | Avg: 71.6 -> 71.7 | Helped |
| Apply Libra-Merging at 37% Flops instead of vanilla inference | LLaVA-NeXT-8B · image-text benchmarks | Avg: 71.6 -> 71.3 | Hurt |
| Apply Libra-Merging rather than FastV | VideoLLaMA-2 (7B) · Video-MME Overall w/o subs | score: 46.5 -> 48.8 | Helped |
| Apply Libra-Merging rather than Turbo | VideoLLaMA-2 (7B) · Video-MME Overall w subs | score: 52.1 -> 52.9 | Helped |
| Use hierarchical merging across layers {7, 15, 23} instead of single-layer merging (ablation) | LLaVA-1.5-7B · ablation, R=50% | Avg: 62.9 -> 63.1 | Helped |
| Add position-driven token identification (ablation) | LLaVA-1.5-7B · ablation, R=67% | Avg: 62.5 -> 62.7 | Helped |
| Add importance-informed grouped merging (ablation) | LLaVA-1.5-7B · ablation, R=67% | Avg: 62.7 -> 63.0 | Helped |
| Add position-driven identification and importance-informed grouped merging (ablation) | LLaVA-1.5-7B · ablation, R=80% | Avg: 61.5 -> 62.2 | Helped |
Surprising findings
- The stronger the LVLM, the more effective the token compression becomes; LLaVA-NeXT-8B with 47% tokens matches or slightly exceeds the uncompressed model.
- Token pruning is preferred when target and non-target tokens have low similarity, while merging is preferred when similarity is high.
Limitations
- Gains over FastV and Turbo are small on several image benchmarks (often within about 0.5 average points).
- Hyperparameters such as the similarity threshold tau=0.7 and temperature eta are set heuristically.
Experimental setup
Datasets: GQA ScienceQA TextVQA MME MMBench Video-MME
Benchmarks: GQA ScienceQA MME MMBench MMBench-CN TextVQA Video-MME
Metrics: accuracy average score Flops (T) latency per example GPU memory
Baselines: FastV Turbo vanilla LVLM
Builds on: FastV Turbo EViT
Closest papers by meaning
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs (CVPR 2026 · similarity 0.89)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.89)
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.89)
- Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.