Research map / Vision-language and multimodal / CLIP and zero-shot

Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models

ICLR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes the modality gap and object bias in 98 off-the-shelf contrastive VLMs and in controlled synthetic and CC12M experiments, attributing both phenomena to information imbalance between images and captions.

Problem

Contrastive VLMs like CLIP show a modality gap in the shared embedding space and a bias towards objects over attributes, but their impact on downstream performance and their causes remain unclear.

Approach

Defines the Relative Modality Gap (RMG) and a Matching Object Attribute Distance (MOAD) bias measure, then shows via controlled information-imbalance experiments (synthetic MAD dataset and CC12M caption dropping) that information imbalance drives both the modality gap and object bias, with the gap serving to regulate logit entropy.

Key results

• Across 98 off-the-shelf VLMs, modality gap (RMG) shows no to weak positive correlation with performance; controlling for dataset, most cases show the expected negative correlation (7 of 10 for RMG) but mostly not significant.
• Removing the top one or two embedding dimensions separates the modalities and closes the gap, but performance drops sharply before partially recovering.
• On MAD and CC12M, reducing information imbalance (more shared caption information) decreases both modality gap and object bias while improving accuracy.

Measured effects reported in the paper

Surprising findings

• A larger modality gap correlates with better performance in the raw off-the-shelf correlations, due to confounders such as model size.
• Post-hoc gap closing via shifting ideal words or the gap vector does not improve performance despite closing the gap.

Limitations

• Most correlations with downstream performance are not statistically significant due to small model counts per dataset.
• Causal claims about the gap regulating entropy are explicitly not established; relationship is correlational.
• Evaluation datasets (MIT-States, UT-Zappos) assume mutually exclusive attributes per image.

Experimental setup

Datasets: MS COCO ImageNet MIT-States UT-Zappos CIFAR-10 CIFAR-100 ImageNet-100 CC12M MAD LAION-2B CommonPool

Benchmarks: MS COCO ImageNet MIT-States UT-Zappos

Metrics: R@1 image-to-text retrieval top-1 zero-shot accuracy Kendall-tau correlation RMG MOAD

Baselines: CLIP ViT-B/16 SigLIP ViT-B/16 ideal words modality gap vector shifting

Builds on: CLIP Liang et al. modality gap Morpho-MNIST ideal words

Closest papers by meaning

• Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.90)
• Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.89)
• Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.89)