Research map / Vision-language and multimodal / Human and embodied modeling

Half-Truths Break Similarity-Based Retrieval

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Composed and text-based image retrieval · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows CLIP-style dual encoders often rank a correct short description below the same description extended with one plausible but wrong detail (a "half-truth"), and proposes CS-CLIP, which fine-tunes CLIP with entity/relation unit-level contrastive supervision against minimally edited foils to fix this.

Problem

CLIP-style image-text similarity can increase when a correct description is extended with one incorrect entity or relation, so it fails to penalize incorrect additions; existing compositional benchmarks mostly test swaps and reorderings rather than incorrect additions.

Approach

Introduce the half-truth diagnostic (anchor vs. anchor plus one wrong unit) and train with unit-level contrastive loss that scores each correct entity or relation unit above an LLM-generated minimally edited foil, alongside NegCLIP-style global hard negatives, while keeping standard dual-encoder inference.

Key results

• CS-CLIP reaches 69.3% overall half-truth accuracy on COCO versus 40.6% for zero-shot CLIP and 56.5% for NegCLIP.
• CS-CLIP achieves the best average compositional I2T accuracy of 57.8% across benchmarks, versus 52.8% for CLIP-style fine-tuning without negatives in ablation (variant 1) and FSC-CLIP at 57.4%.
• Zero-shot classification average Acc@1 drops from 63.6 (CLIP) to 59.9 (CS-CLIP) when fine-tuned on COCO.

Surprising findings

• Zero-shot CLIP prefers the half-truth over the anchor in most relation-addition cases (32.9% accuracy, below chance).
• Matched unit foils without global sentence negatives give the largest half-truth gain (68.3% overall), outperforming NegCLIP-style negatives alone (56.5%).

Limitations

• Relies on text-only LLM parsing and foil generation, which may introduce artifacts or miss visual details not expressed in captions.
• Fine-tuning on COCO trades some zero-shot accuracy for compositional sensitivity.
• Does not guarantee factual correctness or fairness.

Experimental setup

Datasets: MS-COCO

Benchmarks: Half-Truth diagnostic (COCO) ARO SugarCrepe Winoground VL-CheckList CLIPBench zero-shot classification COCO retrieval Flickr8k retrieval

Metrics: half-truth accuracy image-to-text accuracy text-to-image accuracy group accuracy zero-shot accuracy recall@1

Baselines: CLIP NegCLIP SigLIP SigLIP2 FSC-CLIP ReadCLIP DeGLA CE-CLIP DAC LabCLIP

Builds on: CLIP NegCLIP OpenCLIP

Closest papers by meaning

• Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.88)
• CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.85)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.85)
• Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.85)
• SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.85)
• CLIP in Mirror: Disentangling text from visual images through reflection (NeurIPS 2024 · similarity 0.85)