Research map / Vision-language and multimodal / Multimodal LLMs
Vision-Language Models Do Not Understand Negation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NegBench, a multi-task benchmark showing CLIP-style VLMs suffer from affirmation bias on negation, and fine-tunes CLIP/NegCLIP on millions of synthetic negated captions to improve negation understanding.
Problem
Vision-language models that embed images and text jointly struggle to understand negation (e.g., "a beach with no people"), and existing negation benchmarks rely on rigid templates that do not reflect real queries.
Approach
A two-level benchmark (Retrieval-Neg coarse retrieval and MCQ-Neg fine-grained selection with hard-negative options) built from COCO, VOC2007, MSR-VTT, CheXpert, and synthetic Stable Diffusion images, with LLM-paraphrased negated captions; plus a data-centric fix that fine-tunes CLIP models on LLM-generated, detector-verified negation data (CC12M-NegCap/NegMCQ).
Key results
- On COCO, fine-tuning CLIP on CC12M-NegFull raises MCQ accuracy from 39.2% to 54.4% (+15.2), and NegCLIP from 28.6% to 56.2% (+27.6).
- CLIP-based models perform near chance on MCQ-Neg; CLIP scores 82% on affirmation MCQs but 3% on negation MCQs on VOC2007.
- Retrieval recall@5 drops by 7.7% (CLIP-laion400m) on negated COCO queries, and NegCLIP drops 23.0% on HardNeg-Syn.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP on CC12M-NegCap | OpenAI CLIP · COCO | R-Neg@5: 48.0 to 57.8 | Helped |
| Fine-tune CLIP on CC12M-NegCap | OpenAI CLIP · COCO | MCQ accuracy: 39.2% to 47.3% (+8.1) | Helped |
| Fine-tune CLIP on CC12M-NegCap and CC12M-NegMCQ | OpenAI CLIP · COCO | MCQ accuracy: 39.2 to 54.4 (+15.2) | Helped |
| Fine-tune NegCLIP on CC12M-NegCap | NegCLIP · COCO | MCQ accuracy: 28.6% to 40.4% (+11.8) | Helped |
| Fine-tune NegCLIP on CC12M-NegCap and CC12M-NegMCQ | NegCLIP · COCO | MCQ accuracy: 28.6 to 56.2 (+27.6) | Helped |
| Fine-tune CLIP on CC12M-NegCap | OpenAI CLIP · MSR-VTT | R-Neg@5: 45.8 to 53.5 | Helped |
| Fine-tune CLIP on CC12M-NegCap and CC12M-NegMCQ | OpenAI CLIP · MSR-VTT | MCQ accuracy: 32.1 to 44.9 | Helped |
| Set α to 0 instead of 1 in the combined training objective (ablation) | CLIP fine-tuned on CC12M-NegFull · COCO | R@5: 58.5 to 33.9 | Hurt |
| Set α to 0 instead of 1 in the combined training objective (ablation) | CLIP fine-tuned on CC12M-NegFull · COCO | MCQ accuracy: 47.3 to 59.4 | Helped |
Surprising findings
- Scaling CLIP from ViT-B to ViT-H, and newer models like SigLIP and AIMv2, does not improve negation understanding.
- NegCLIP and ConCLIP do not outperform the original OpenAI CLIP on the MCQ-Neg task.
Limitations
- Fine-grained negation differences remain challenging after synthetic-data fine-tuning.
- LLaVA-style generative VLMs are impractical as retrieval systems since they cannot embed images and texts for efficient search.
- Tradeoff between retrieval performance (CLIP objective weight alpha) and MCQ negation accuracy.
Experimental setup
Datasets: COCO VOC2007 MSR-VTT CheXpert CC12M ImageNet
Benchmarks: NegBench COCO MSR-VTT VOC2007 CheXpert HardNeg-Syn
Metrics: recall@5 R-Neg@5 MCQ accuracy
Baselines: CLIP-OpenAI CLIP-LAION400M CLIP-DataComp NegCLIP ConCLIP SigLIP AIMv2 BioMedCLIP CONCH LLaVA
Builds on: CLIP NegCLIP ConCLIP CREPE
Closest papers by meaning
- What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging (ICLR 2026 · similarity 0.91)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.91)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.