Research map / Vision-language and multimodal / Multimodal LLMs
Semantic Impact–Driven Visual Scheduling in Vision-Language Models
Xuan Wang, Yilin Liu, Fangxiang Feng, Caixia Yuan, Huixing Jiang, Xiaojie Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free framework that schedules visual tokens in VLMs by measuring their semantic impact on the output distribution, rather than via attention weights, to reduce inference latency.
Problem
VLMs have high inference latency because of long visual token sequences, and existing methods infer visual necessity from intermediate signals such as attention weights.
Approach
Assess visual necessity by its semantic impact on the output distribution, using a token embedding subspace, rather than attention-based interaction signals; applied on demand in a training-free manner.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| score visual tokens by semantic impact instead of attention weight (ablation) | SIVS · TextVQA (no OCR) | accuracy (%): 37.16 -> 38.40 | Improved |
| score visual tokens by semantic impact instead of attention-weighted value norm (ablation) | SIVS · TextVQA (no OCR) | accuracy (%): 36.90 -> 38.40 | Improved |
| SIVS versus FastV, SparseVLM, VisPruner at higher compression | visual KV compression · LLaVA-1.5-7B seven benchmarks | relative accuracy: 99.95% at 87.66% compression | Improved |
| SIVS versus vanilla on POPE and MileBench | SIVS · LLaVA-1.5-7B | score: 85.9 -> 86.2 (POPE); 38.08 -> 38.94 (MileBench) | Improved |
| SIVS versus vanilla on GQA and TextVQA | SIVS · LLaVA-1.5-7B | score: 61.9 -> 61.7 (GQA); 58.3 -> 56.85 (TextVQA) | Hurt |
| apply SIVS to Qwen2.5-VL 7B | SIVS · TextVQA | score: 84.50 -> 81.92 | Hurt |
| apply SIVS to Qwen2.5-VL 3B-72B | SIVS · GQA | score: 58.89 -> 58.86 (3B); 62.16 -> 62.26 (72B) | No clear effect |
| increase Top-K size K (ablation) | SIVS · TextVQA | accuracy | Improved |
| increase threshold coefficient alpha beyond 0.5 (ablation) | SIVS · TextVQA | accuracy | Hurt |
| refresh visual selection every step (smaller T) (ablation) | SIVS · LLaVA-Bench | accuracy and latency | Mixed |
| use mid-to-late layer for semantic impact estimation (ablation) | SIVS · TextVQA | accuracy | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SparseVLM — SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025) · better on 9 of 9 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.89)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.89)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.88)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
