Research map / Vision-language and multimodal / Multimodal LLMs
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how the MLLM projector compresses and aligns vision patches with word embeddings, then adds a cosine-based patch-alignment loss during pretraining (patch-aligned training) using an automatically annotated dataset (PAD), improving REC, VQA, and instruction-following over LLaVA-1.5.
Problem
Projectors connecting pretrained vision encoders to LLMs are trained mainly with caption loss, which yields weak, coarse patch-level alignment between vision tokens and word embeddings, hurting region-level understanding.
Approach
Measure projector compression via Von Neumann entropy and patch-level alignment via IoU of embedding-similarity masks, then add a cosine patch-alignment loss on mask-pooled vision embeddings versus LLM word embeddings during stage-1 pretraining, using masks from an RAM+Grounding DINO+SAM annotation pipeline.
Key results
- Patch-aligned projector reaches Align 0.279 vs 0.142 for LLaVA projector and 0.065 for random, with entropy reduction 3.84 vs 2.80.
- Patch-aligned LLaVA-1.5-7B improves RefCOCO val Acc@0.5 from 56.22 to 65.97 and RefCOCOg test from 48.4 to 56.24 (about 16% average gain claimed).
- Improves GQA 62.0→63.0, OKVQA 53.4→58.3, and MMVet 30.70→31.61 under the same SFT setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add patch-alignment loss and PAD annotations during pretraining | LLaVA 1.5-7B · RefCOCO, RefCOCO+, and RefCOCOg test splits | Acc@0.5: approximately a 16% improvement on average | Helped |
| Add patch-alignment loss and PAD annotations during pretraining | LLaVA-1.5 with Vicuna-7B · GQA, SciQA, VizWiz, and OKVQA | accuracy: 4% | Helped |
| Add patch-alignment loss and PAD annotations during pretraining | LLaVA 1.5 · MMMU, MMVet, CMMMU, MMB EN, MME C, and MME P | instruction-following benchmark score: 3% | Mixed |
| Use the Stage 1 pretrained projector instead of a random MLP projector (ablation) | LLaVA projector · GranDf | patch alignment (mIoU): 0.065 -> 0.142 | Helped |
| Use the Stage 2 SFT projector instead of the Stage 1 projector (ablation) | LLaVA projector · GranDf | patch alignment (mIoU): 0.142 -> 0.152 | Helped |
| Replace LLaVA pretraining with patch-aligned training | LLaVA projector · 100 COCO 2017 images | entropy reduction: 2.7991 -> 3.8352 | Helped |
| Replace LLaVA pretraining with patch-aligned training | LLaVA projector · 100 COCO 2017 images | patch alignment (mIoU): 0.142 -> 0.279 | Helped |
| Replace LLaVA pretraining with patch-aligned training | LLaVA projector · 100 COCO 2017 images | cosine similarity: 0.07 -> 0.56 | Helped |
| Use a trained Linear projector instead of a randomly initialized Linear projector (ablation) | Linear projector · COCO2017 images | Von Neumann entropy after projection: 4.8197 -> 2.4829 | Hurt |
| Use a trained MLP projector instead of a randomly initialized MLP projector (ablation) | MLP projector · COCO2017 images | Von Neumann entropy after projection: 4.8245 -> 2.0362 | Hurt |
| Use a trained C-Abstractor instead of a randomly initialized C-Abstractor (ablation) | C-Abstractor · COCO2017 images | Von Neumann entropy after projection: 7.4913 -> 3.5850 | Hurt |
| Replace a trained Linear projector with a trained MLP projector (ablation) | LLaVA projector · COCO2017 images | Von Neumann entropy after projection: 2.4829 -> 2.0362 | Hurt |
Surprising findings
- The LLaVA projector's patch alignment is only slightly above random (0.142 vs 0.065), and it decreases compression to a limited extent, suggesting caption loss implicitly gives weak alignment.
- Patch-aligned training slightly lowers MMBench (63.14 vs 64.00) despite gains elsewhere.
Limitations
- Finding an optimal per-token representation remains open; aligning to averaged word embeddings may limit expressiveness and interpretability.
- Manually guiding the projector toward semantic alignment may cause information loss in visual tokens.
- Multi-semantic alignment is evaluated only qualitatively due to lack of ground-truth multi-semantic labels.
Experimental setup
Datasets: LLaVA pretraining 558K (LAION-CC-SBU subset) LLaVA-Instruct TextVQA GQA OCR-VQA Visual Genome COCO2017 GranDf
Benchmarks: RefCOCO RefCOCO+ RefCOCOg GQA SciQA VizWiz OKVQA MMMU MMVet CMMMU MMBench MME
Metrics: Acc@0.5 mIoU METEOR ROUGE-L SPICE Von Neumann entropy reduction cosine similarity
Baselines: LLaVA-1.5-7B LLaVA (LLaMA3.1 8B) SEA BLIP-2 InstructBLIP Qwen-VL
Builds on: LLaVA-1.5 C-Abstractor RAM Grounding DINO SAM
Closest papers by meaning
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.91)
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.91)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.91)
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.90)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.90)
- Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.