Research map / Vision-language and multimodal / Multimodal LLMs
Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AcFormer, a vision-language connector that selects "visual anchors" from a ViT's [CLS] attention map and uses them as input queries for cross-attention, cutting visual tokens while improving LLaVA-1.5 accuracy.
Problem
Vision-language connectors in MLLMs either keep many visual tokens (costly LLM computation) or use randomly initialized learnable queries that need massive data and lose information for different images.
Approach
Observe that ViT tokens with high [CLS] attention ("visual anchors") aggregate information; select them per image with a head-wise progressive top-k search over the [CLS] attention map and use them, rather than random learnable queries, as the information aggregator for a Perceiver-Resampler-style cross-attention module.
Key results
- AcFormer with 145 visual tokens on Vicuna-7B reaches MMBench 68.4 and MME 1846.1 versus LLaVA-1.5's 64.3 and 1794.6 with 577 tokens, at 2.23x pre-training speed.
- On Vicuna-13B with 145 tokens, AcFormer scores 69.2 on MMBench and 1870.0 on MME, with 2.30x pre-training speed relative to LLaVA-1.5.
- In the ablation with 145 tokens on Vicuna-7B, AcFormer improves TextVQA to 58.0 over Perceiver Resampler's 52.1 and Random-PR's 54.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LLaVA-1.5's linear connector with AcFormer | Vicuna-7B MLLM · MMB | accuracy: 64.3 -> 68.4 | Helped |
| replace LLaVA-1.5's linear connector with AcFormer | Vicuna-7B MLLM · MM-Vet | GPT-4 score: 30.5 -> 30.3 | Hurt |
| replace LLaVA-1.5's linear connector with AcFormer | Vicuna-7B MLLM · GQA | EM: 62.0 -> 61.2 | Hurt |
| replace LLaVA-1.5's linear connector with AcFormer | Vicuna-7B MLLM · TextVQA | VQA score: 58.2 -> 58.2 | No clear effect |
| replace LLaVA-1.5's linear connector with AcFormer | Vicuna-7B MLLM · Pre-training | relative speed: $1.00\times$ -> $\mathbf{1.65\times}$ | Helped |
| use visual anchors with cross-attention instead of direct pooling (ablation) | LLaVA-1.5, Vicuna-7B, 65 visual tokens · TextVQA | VQA score: 53.4 -> 56.1 | Helped |
| use visual anchors instead of randomly selected visual-token queries (ablation) | LLaVA-1.5, Vicuna-7B, 65 visual tokens · TextVQA | VQA score: 53.9 -> 56.1 | Helped |
| use visual anchors instead of learnable Perceiver Resampler queries (ablation) | LLaVA-1.5, Vicuna-7B, 65 visual tokens · TextVQA | VQA score: 51.0 -> 56.1 | Helped |
| increase visual-token count from 65 to 145 with AcFormer (ablation) | LLaVA-1.5, Vicuna-7B · TextVQA | VQA score: 56.1 -> 58.0 | Helped |
| increase visual-token count from 145 to 257 with AcFormer (ablation) | LLaVA-1.5, Vicuna-7B · TextVQA | VQA score: 58.0 -> 58.2 | No clear effect |
| use AcFormer instead of C-Abstractor (ablation) | LLaVA-1.5, Vicuna-7B, 145 visual tokens · TextVQA | VQA score: 53.4 -> 58.0 | Helped |
| retain unselected tokens through cross-attention rather than directly use selected tokens (ablation) | LLaVA-1.5, Vicuna-7B, 145 visual tokens · TextVQA | VQA score: 56.3 -> 58.0 | Helped |
Surprising findings
- Randomly selected ViT tokens used as Perceiver Resampler queries (Random-PR) perform comparably to or better than learnable queries in several benchmarks.
- Directly pooling visual tokens without a learned aggregator performs strongly, though it drops about 3 points on TextVQA relative to AcFormer.
Limitations
- Larger training datasets and larger models were not explored due to resource constraints.
- No theoretical analysis explaining why visual anchors emerge.
- Slightly underperforms LLaVA-1.5 on GQA and VQAv2.
Experimental setup
Datasets: LLaVA-1.5 pre-training data (558k) LLaVA-665k instruction tuning LAION-115M COYO LAION-COCO Cauldron
Benchmarks: POPE MME MMBench MM-Vet TextVQA GQA VQAv2 VizWiz ScienceQA-IMG OK-VQA
Metrics: accuracy VQA score F1 EM GPT-4 score relative pre-training speed
Baselines: Perceiver Resampler C-Abstractor Pooling Q-Former LLaVA-1.5 E-ViT Top-P
Builds on: LLaVA-1.5 Perceiver Resampler Q-Former C-Abstractor EViT
Closest papers by meaning
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.91)
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.90)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.89)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.89)
- ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.