Research map / Label-efficient and robust learning / Recognition and long-tail

Aligning Multimodal Representations through an Information Bottleneck

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that contrastive losses leave modality-specific nuisance information in multimodal representations, causing misalignment, and adds a variational KL-derived regularizer (squared distance between paired embeddings) to InfoNCE as an information-bottleneck term that improves alignment and captioning.

Problem

Contrastive losses like InfoNCE produce multimodal representations that are poorly aligned (modality gap), because they maximize shared information but do not remove modality-specific nuisance information.

Approach

Frames multimodal alignment as an information bottleneck: sufficiency via InfoNCE (lower bound on MI between modalities) plus minimality via an upper bound on I(Z_alpha; X_alpha) derived from KL divergence between paired representation distributions, which reduces to a squared L2 distance between paired embeddings added to the contrastive loss.

Key results

• Adding 0.1·L_M to ITC+LM raises CIDEr from 91.7 to 93.0 and BLEU@4 from 28.6 to 29.4 on COCO, while I2T R@1 drops from 64.2 to 63.0.
• In toy experiments, nuisance information (URR) decreases with encoder depth and higher temperature, and it is negatively correlated with CKA alignment.
• Increasing beta makes the learned temperature decrease (Information Homeostasis), so nuisances are not removed as much as with fixed temperature.

Measured effects reported in the paper

Surprising findings

• The learned temperature drops as beta increases, partially counteracting the nuisance-removal effect (called Information Homeostasis).
• Larger regularization weight (0.3·L_M) degrades captioning and retrieval, showing a trade-off between alignment and task performance.

Limitations

• Theoretical guarantees rely on Gaussian assumptions with shared variance, so the KL bound is approximate in practice.
• Higher regularization weights hurt retrieval performance and can over-compress representations.
• Experiments are limited to toy disentanglement datasets and a single COCO Q-Former setup.

Experimental setup

Datasets: DSprites MPI3D Shapes3D COCO

Benchmarks: COCO captioning (CIDEr, BLEU@4) COCO image-text retrieval (I2T R@1, T2I R@1) DSprites MPI3D Shapes3D

Metrics: CKA uncertainty reduction ratio CIDEr BLEU@4 R@1

Baselines: ITC+LM ITC+LM+ITM

Builds on: InfoNCE information bottleneck CLIP BLIP-2

Closest papers by meaning

• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
• Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.90)
• Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.89)
• Accept the Modality Gap: An Exploration in the Hyperbolic Space (CVPR 2024 · similarity 0.89)
• Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
• Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.89)