Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes READ, a fine-tuning method adding token-level reconstruction (frozen T5 decoder reconstructs alternative caption from text embedding) and sentence-level paraphrase alignment to contrastive loss, yielding READ-CLIP with state-of-the-art compositional reasoning on five benchmarks.
Problem
Contrastively trained vision-language models, such as CLIP, struggle with compositional reasoning because the text encoder focuses on individual words rather than word relations.
Approach
Adds two auxiliary text-encoder objectives to contrastive learning: a frozen pre-trained T5 decoder reconstructs an alternative caption from the original caption's embedding (token-level), and paraphrased captions are aligned in embedding space (sentence-level), with NegCLIP-style hard negatives in the contrastive loss.
Key results
- READ-CLIP achieves 64.1% average accuracy across five compositional benchmarks, versus 59.6% for NegCLIP and 62.4% for FSC-CLIP (strongest baseline).
- Applying READ to CLIP, NegCLIP and FSC-CLIP improves average accuracy from 57.0 to 58.2, 61.5 to 62.5, and 62.8 to 63.6 respectively.
- Reconstruction loss and alignment loss are complementary: contrastive-only 62.2 avg, reconstruction-only 63.9, alignment-only 63.8, both 64.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply READ fine-tuning to pretrained CLIP | CLIP ViT-B/32 · Average across five compositional reasoning benchmarks | accuracy: 52.0 -> 64.1 | Helped |
| Apply READ fine-tuning to CLIP instead of FSC-CLIP fine-tuning | CLIP ViT-B/32 · Average across five compositional reasoning benchmarks | accuracy: 62.4 -> 64.1 | Helped |
| Add token-level reconstruction loss (ablation) | READ-CLIP contrastive-only variant · Average across five benchmarks | accuracy: 62.2% to 63.9% | Helped |
| Add token-level reconstruction loss (ablation) | READ-CLIP contrastive-only variant · WhatsUp | accuracy: 40.5 -> 43.6 | Helped |
| Add token-level reconstruction loss (ablation) | READ-CLIP contrastive-only variant · CREPE | accuracy: 38.4 -> 41.6 | Helped |
| Add token-level reconstruction loss (ablation) | READ-CLIP contrastive-only variant · VALSE | accuracy: 74.7 -> 76.6 | Helped |
| Add sentence-level alignment loss (ablation) | READ-CLIP contrastive-only variant · SugarCrepe++ ITT | accuracy: 69.3 -> 70.2 | Helped |
| Add sentence-level alignment loss (ablation) | READ-CLIP contrastive-only variant · SugarCrepe++ TOT | accuracy: 66.0 -> 67.0 | Helped |
| Combine token-level reconstruction and sentence-level alignment (ablation) | READ-CLIP · Average across five benchmarks | accuracy: 62.2 -> 64.1 | Helped |
| Reconstruct an alternative caption instead of the original caption (ablation) | READ-CLIP · SugarCrepe++ TOT across training epochs | accuracy | Helped |
| Increase paraphrase noise to 10% or 20% (ablation) | READ-CLIP sentence alignment · Average across five benchmarks | accuracy: 64.1 -> 62.9; 64.1 -> 62.1 | Hurt |
| Increase available paraphrases per caption from 1 to 3 or 5 (ablation) | READ-CLIP sentence alignment · Average across five benchmarks | accuracy: 64.1 -> 64.0; 64.1 -> 64.1 | No clear effect |
Surprising findings
- Fine-tuned models, including READ-CLIP, achieve lower average zero-shot classification than the original CLIP across 23 datasets.
- Reconstructing the original caption causes overfitting to exact wording, whereas reconstructing an alternative caption stabilizes performance.
Limitations
- Requires multiple captions per image or LLM-generated paraphrases, adding complexity for single-caption datasets.
- Only T5 decoder explored for reconstruction; alternative generative decoders not studied.
- Fine-tuned models underperform original CLIP on zero-shot ImageNet-style classification across 23 datasets.
Experimental setup
Datasets: MS-COCO SugarCrepe++ WhatsUp CREPE VALSE SugarCrepe
Benchmarks: WhatsUp VALSE CREPE SugarCrepe SugarCrepe++ ImageNet zero-shot classification (23 datasets)
Metrics: accuracy
Baselines: CLIP NegCLIP GNM-CLIP Triplet-CLIP CE-CLIP FSC-CLIP
Builds on: CLIP NegCLIP FSC-CLIP T5
Closest papers by meaning
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.92)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.91)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.90)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.