Research map / Vision-language and multimodal / Cross-modal learning

Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

NeurIPS 2024 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Contrastive Alignment (CAL), a training-time token re-weighting that weights each text token's MLE loss by its logit difference with vs. without the image, improving VLMs such as LLaVA-1.5, LLaVA-NeXT and MGM at negligible extra cost.

Problem

Standard autoregressive image-text alignment in VLMs weights all text tokens equally, so tokens weakly correlated with or contradicting the image receive as much training signal as visually grounded ones.

Approach

Use the logit difference between predictions with and without the image as a per-token visual-correlation signal; clamp and average-pool these differences and use them as weights in the cross-entropy loss, so visually correlated tokens are prioritized. This is a training-time analogue of visual contrastive decoding.

Key results

• On LLaVA-NeXT-13B, CAL improves VQA-Doc by 1.7 ANLS (78.4 to 80.1), VQA-Chart by 3.4 (63.8 to 67.2), and OCR-Bench by 21 points (553 to 574).
• CAL improves COCO Caption CIDEr by 2.1 and TextCaps by 6.2 on LLaVA-NeXT-13B, with RefCOCOg val/test IoU gains of 0.6/0.7.
• CAL adds about 20% training time on LLaVA-NeXT-7B and 13B, with no inference overhead, and degrades more slowly than the baseline under 10-30% injected label noise.

Measured effects reported in the paper

Surprising findings

• Human annotation found roughly half of sampled ShareGPT4V and LLaVA detail-caption samples contain text tokens contradicting the image.
• Gaussian blur and random patch masking as contrast conditions performed worse than full-image masking in the ablations.

Limitations

• Lacks a clear quantitative distinction between the three token types, so weight guidance is heuristic.
• Clamping bounds and pooling window are chosen empirically from logit frequency statistics.
• LLaVA-NeXT was reproduced on a different training data mixture because the instruction-tuning data is not public, and the image is still forwarded twice in the contrastive pass.

Experimental setup

Datasets: LLaVA-665k VQA-Doc VQA-Chart ShareGPT4V LLaVA-filtered CC3M ALLaVA LAION-GPT-4V LIMA OpenAssistant2 DVQA AI2D

Benchmarks: VQA-Text VQA-Doc VQA-Chart OCR-Bench MMT-Bench MMStar SQA-I COCO Caption TextCaps RefCOCOg MME POPE

Metrics: accuracy ANLS relaxed accuracy CIDEr IoU

Baselines: LLaVA-1.5 LLaVA-NeXT MGM MGM-HD VCD IBD CRG

Builds on: LLaVA-1.5 LLaVA-NeXT Mini-Gemini visual contrastive decoding image-based decoding

Closest papers by meaning

• BiMAC: Bidirectional Multimodal Alignment in Contrastive Learning (AAAI 2025 · similarity 0.90)
• Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
• Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.90)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.90)
• Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.90)
• Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)