Research map / Vision-language and multimodal / CLIP and zero-shot

Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability

ICLR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: CLIP training recipes and data curation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Narrowing Information Bottleneck Theory (NIBT/NIB), a randomness-free, hyperparameter-free attribution method for CLIP image and text representations that scales a layer's features by a single lambda and integrates the change in mutual information.

Problem

Interpreting multimodal image-text representation models such as CLIP is hard: existing attribution methods rely on extra samples, randomness, or hyperparameters (e.g., the beta trade-off in M2IB), and the information bottleneck approach has randomness and sign limitations.

Approach

Replace the noisy information bottleneck layer with a deterministic scalar scaling lambda applied to an intermediate feature z, and define each feature's importance as the integral over lambda from 0 to 1 of the gradient of the CLIP cosine similarity times dz/dlambda (an integrated-gradients-style attribution), removing sampling noise and the beta hyperparameter and allowing negative attributions.

Key results

• On Conceptual Captions, NIB reaches Image Confidence Drop 0.9439 (vs M2IB 1.1171) and Text Confidence Drop 0.2705 (vs M2IB 1.706), with FPS 1.5817 vs 0.6621 for M2IB.
• On ImageNet, NIB achieves the lowest Image Confidence Drop (0.9012) and Text Confidence Drop (0.4193), and the highest Text Confidence Increase among compared methods (56.1).
• The paper claims average gains of 9% image interpretability, 58.83% text interpretability and 63.95% speed over state-of-the-art methods.

Measured effects reported in the paper

Surprising findings

• Reported M2IB performance is highly sensitive to beta: Image Confidence Drop rises from 0.8738 to 2.9715 as beta goes from 0.01 to 0.9 on Conceptual Captions.
• Ablations show layer 9 is best among tested layers (3, 6, 9 and additional layers 2-8) and num_steps of 10 gives a good trade-off, though results vary non-monotonically across settings.

Limitations

• Evaluation relies on confidence drop/increase metrics rather than insertion/deletion or ROAR, which the authors justify by lacking task labels.
• Theoretical claims are loosely argued; the sigma-to-zero and mutual-information arguments are heuristic and the proofs are sketchy.
• Only CLIP ViT-B/32 and three datasets are evaluated in the main text; generalization is shown only qualitatively on RSICD.

Experimental setup

Datasets: Conceptual Captions ImageNet Flickr8k RSICD

Benchmarks: Conceptual Captions ImageNet Flickr8k

Metrics: Image Confidence Drop Image Confidence Increase Text Confidence Drop Text Confidence Increase FPS

Baselines: M2IB RISE Grad-CAM Chefer et al. Saliency Maps MFABA FastIG

Builds on: Information Bottleneck Principle M2IB Integrated Gradients Grad-CAM CLIP

Closest papers by meaning

• Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation (AAAI 2025 · similarity 0.87)
• Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.86)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.86)
• CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.86)
• Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) (NeurIPS 2024 · similarity 0.86)
• V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.86)