Research map / Vision-language and multimodal / Cross-modal learning

Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

ICLR 2025 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that CLIP-style inter-modal training causes intra-modal misalignment, and that routing image-to-image and text-to-text retrieval through optimization-based modality inversion (OTI/OVI) into the complementary modality improves retrieval without training.

Problem

Pre-trained CLIP-style VLMs are routinely used for intra-modal similarity (image-image, text-text), but the inter-modal contrastive loss does not constrain intra-modal distances, so these similarities are misaligned and suboptimal.

Approach

Approach intra-modal tasks inter-modally by mapping the query into the complementary modality with single-feature optimization-based inversion (OTI for image-to-text, a new OVI for text-to-image) with frozen encoders, and show the gain comes from inter-modal alignment rather than extra information; also show that adding an intra-modal loss (SLIP) or closing the modality gap reduces the misalignmen

Key results

• OTI-inverted image queries improve image-to-image mAP on average by about 2-3 points over the intra-modal baseline (e.g., CLIP L/14 average 52.0 to 55.3 across 15 datasets).
• Using OVI-inverted text queries improves text-to-text retrieval by 1-5 mAP across datasets and models (e.g., CLIP B/32 average 37.7 to 40.6 on Flickr30k, COCO, nocaps).
• Making zero-shot classification intra-modal via OTI drops accuracy substantially (e.g., CLIP L/14 average 72.2 inter-modal vs 50.1 with OVI-style intra-modal inversion; OTI variant 68.4), and SLIP-trained models show OTI gains vanish.

Measured effects reported in the paper

Surprising findings

• Zero-shot classification becomes worse when made intra-modal, even though the same OTI-inverted features improve image-to-image retrieval, showing that modality inversion alone does not help.
• Closing the modality gap (tau=1 fine-tuning) removes the benefit of inter-modal inversion for image retrieval.

Limitations

• Modality inversion is computationally expensive, requiring iterative optimization per sample (150 steps OTI, 1000 steps OVI), limiting practicality.
• The paper analyzes misalignment but offers no efficient practical alternative.
• OVI only applies to ViT-based image encoders.

Experimental setup

Datasets: CUB-200-2011 Stanford Online Products ROxford RParis RParis1M distractors Cars Pets Flowers Aircraft DTD EuroSAT Food101

Benchmarks: CUB-200-2011 Stanford Online Products ROxford RParis Cars Pets Flowers Aircraft DTD EuroSAT Food101 SUN397

Metrics: mAP accuracy R@1 R@5 R@10

Baselines: CLIP intra-modal image-image retrieval CLIP intra-modal text-text retrieval SLIP intra-modal SigLIP

Builds on: Optimization-based Textual Inversion (OTI) CLIP SLIP SigLIP

Closest papers by meaning

• Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.91)
• Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
• SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger (AAAI 2024 · similarity 0.90)
• WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.89)
• Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.89)