Research map / Vision-language and multimodal / Multimodal LLMs
Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Simignore, a training-free image token reduction method that masks image tokens with low cosine similarity to text embeddings, improving ScienceQA (IMG) accuracy for LLaVA-1.5 and Mipha-3B.
Problem
LVLMs spend computation on hundreds of image tokens, many irrelevant to the text prompt, and these irrelevant tokens can hurt complex chain-of-thought reasoning; prior token pruning methods ignore the text interaction.
Approach
Using information-flow and attention visualizations, it observes that image tokens semantically related to the text converge information flow and get high attention; it then ranks image tokens by cosine similarity to projected text embeddings and masks out the less similar ones in attention.
Key results
- LLaVA1.5-7B accuracy on ScienceQA (IMG) rises from 65.15% to 68.02% after ignoring 124 of 576 image tokens with Simignore.
- LLaVA1.5-13B improves from 72.09% to 73.23%; Mipha-3B from 70.40% to 70.85%.
- Ignoring the most important (similar) image tokens drops accuracy to 61.73%, versus 68.02% for ignoring unimportant ones; cosine similarity beats Euclidean (66.73%) and Manhattan (66.88%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Simignore | Mipha-3B · ScienceQA (Image) | accuracy: 70.40 -> 70.85 | Helped |
| Apply Simignore | LLaVA1.5 (Vicuna-7B) · ScienceQA (Image) | accuracy: 65.15 -> 68.02 | Helped |
| Apply Simignore | LLaVA1.5 (Vicuna-13B) · ScienceQA (Image) | accuracy: 72.09 -> 73.23 | Helped |
| Ignore 124/576 image tokens (ablation) | LLaVA1.5-7B · ScienceQA (Image) | accuracy: 65.15 -> 68.02 | Helped |
| Ignore image tokens, with counts varied from 72/576 to 576/576 (ablation) | LLaVA1.5-7B · ScienceQA (Image) | accuracy: 67.23; 68.02; 67.67; 67.28; 66.48; 66.04; 65.54; 63.96; 53.35; baseline 65.15 | Mixed |
| Ignore unimportant image tokens rather than important tokens (ablation) | LLaVA1.5-7B · ScienceQA (Image) | accuracy: 65.15 -> 68.02 | Helped |
| Ignore intermediate-importance image tokens (ablation) | LLaVA1.5-7B · ScienceQA (Image) | accuracy: 65.15 -> 64.55 | Hurt |
| Ignore important image tokens (ablation) | LLaVA1.5-7B · ScienceQA (Image) | accuracy: 65.15 -> 61.73 | Hurt |
| Randomly ignore image tokens (ablation) | LLaVA1.5-7B · ScienceQA (Image) | accuracy: 65.15 -> 65.12 | No clear effect |
| Use cosine similarity rather than Euclidean or Manhattan distance (ablation) | Simignore on LLaVA1.5-7B · ScienceQA (Image) | accuracy: Euclidean Distance 66.73; Manhattan Distance 66.88; Cosine Similarity 68.02 | Helped |
Surprising findings
- Ignoring all 576 image tokens still yields 53.35% accuracy on ScienceQA (IMG).
- Pruning a subset of unimportant image tokens improves accuracy over the full-token baseline, suggesting some image tokens harm reasoning.
Limitations
- Number of tokens to ignore is fixed by hand rather than adaptively chosen.
- Evaluated only on ScienceQA (IMG), the only complex reasoning dataset considered.
- Analysis of 'spy' tokens is based on limited examples and conjecture.
Experimental setup
Datasets: ScienceQA
Benchmarks: ScienceQA (IMG)
Metrics: accuracy running time
Baselines: Instruct-BLIP BLIP-2 Shikra DDCoT Otter MiniGPT-4 Qwen-VL-Chat MobileVLM Qwen-VL Mipha-3B
Builds on: LLaVA-1.5 FastV LLaVA-PruMerge EAH
Closest papers by meaning
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- Enhancing Multi-Image Understanding through Delimiter Token Scaling (ICLR 2026 · similarity 0.89)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.89)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.89)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.