Research map / Vision-language and multimodal / CLIP and zero-shot
Extract Free Dense Misalignment from CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP4DM, which detects misaligned words between image and text by removing the ReLU in gradient-based attribution over CLIP attention maps, and F-CLIPScore, which aggregates negative attributions with the global CLIPScore.
Problem
Detecting dense (word-level) misalignments between images and text cheaply, since existing approaches rely on large foundation models in zero-shot mode or on human-annotated fine-tuned models, which are computationally costly.
Approach
Modifies generic attention-model explainability (GAE) by removing the ReLU so negative gradient-weighted attributions of text tokens indicate misalignment; words with attribution below a threshold are flagged, and F-CLIPScore multiplies (1 - CLIPScore) by the summed negative attributions of flagged words.
Key results
- On nocaps-FOIL overall, Ours (ViT-H/14) reaches 0.716 localization accuracy and 0.794 AP, versus 0.452 LA and 0.695 AP for ALOHa, with about 44x faster inference than ALOHa.
- On SeeTRUE-Feedback textual misalignment, Ours (ViT-H/14) reaches an NLI score of 0.660, above zero-shot LLaVA-1.5, mPLUG-Owl, InstructBLIP and MiniGPT-v2 (best 0.560), but below fine-tuned PaLI 17B (0.785).
- On Rich-HF, Ours (ViT-H/14) reaches F1 0.427 with recall 0.516, close to fine-tuned Rich-HF models (F1 0.433-0.439).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP ViT-H/14 instead of ViT-B/32 | CLIP4DM · nocaps-FOIL overall | localization accuracy: 0.603 -> 0.716 | Helped |
| Use CLIP ViT-H/14 instead of ViT-B/32 | CLIP4DM · nocaps-FOIL overall | average precision: 0.690 -> 0.794 | Helped |
| Use CLIP4DM instead of zero-shot CLIPScore | CLIP ViT-H/14 · nocaps-FOIL overall | average precision: 0.722 -> 0.794 | Helped |
| Use CLIP4DM instead of CLIPScore | CLIP ViT-L/14 · MMVP | average: 17.8 -> 24.4 | Helped |
| Use F-CLIPScore instead of CLIPScore | CLIP ViT-L/14 · SugarCrepe add-object hard negatives | score: 78.32 -> 88.94 | Helped |
| Use F-CLIPScore instead of CLIPScore | CLIP ViT-L/14 · SugarCrepe replace-object hard negatives | score: 94.07 -> 92.07 | Hurt |
| Use F-CLIPScore instead of CLIPScore | CLIP ViT-L/14 · SugarCrepe swap-attribute hard negatives | score: 62.31 -> 58.26 | Hurt |
| Multiply gradients by attention maps when calculating attributions (ablation) | CLIP4DM · nocaps-FOIL test set | localization accuracy: 0.423 -> 0.716 | Helped |
| Use occlusion-based attribution instead of gradient × attention attribution (ablation) | CLIP4DM · nocaps-FOIL test set | localization accuracy: 0.716 -> 0.566 | Hurt |
| Include positive gradients instead of retaining only negative gradients (ablation) | CLIP4DM · nocaps-FOIL test set | localization accuracy: 0.698 -> 0.716 | Helped |
| Include positive gradients instead of retaining only negative gradients (ablation) | CLIP4DM · nocaps-FOIL test set | average precision: 0.779 -> 0.794 | Helped |
| Combine summed misaligned-token attributions with global similarity (ablation) | F-CLIPScore · nocaps-FOIL | average precision: 0.776 -> 0.794 | Helped |
Surprising findings
- Removing the ReLU so that negative gradients are kept (rather than positive ones) gives the best performance, which is counter to the usual relevance-propagation practice of discarding negative contributions.
- F-CLIPScore performs worse than CLIPScore on SugarCrepe replace and swap hard negatives, and it degrades when CLIPScore is extremely low.
Limitations
- Inherits known CLIP weaknesses such as numbers, small objects, backgrounds, and compositionality; performance on adjectives, adverbs, and adpositions is limited.
- Higher recall but lower precision on Rich-HF word-level detection.
- F-CLIPScore is unreliable when CLIPScore is extremely low, and text is limited to CLIP's 77-token context.
Experimental setup
Datasets: FOIL nocaps-FOIL HAT SeeTRUE-Feedback Rich-HF MMVP SugarCrepe
Benchmarks: FOIL nocaps-FOIL HAT SeeTRUE-Feedback Rich-HF MMVP SugarCrepe
Metrics: localization accuracy average precision F1 precision recall NLI score Pearson correlation Spearman correlation FPS
Baselines: CHAIR CLIPScore RefCLIPScore ALOHa LLaVA-1.5 PaLI mPLUG-Owl InstructBLIP MiniGPT-v2 Rich-HF PickScore occlusion-based attribution
Builds on: Generic Attention-model Explainability (GAE) CLIPScore RefCLIPScore Layer-wise relevance propagation
Closest papers by meaning
- Symbal: Detecting Systematic Misalignments in Model-Generated Captions (ICML 2026 · similarity 0.87)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.87)
- Enhancing CLIP Robustness via Cross-Modality Alignment (NeurIPS 2025 · similarity 0.87)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.87)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.