Research map / Vision-language and multimodal / CLIP and zero-shot
Describing Differences in Image Sets with Natural Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces set difference captioning and VisDiff, a GPT-4 caption-based proposer plus CLIP feature ranker that describes differences between two image sets, evaluated on the new VisDiffBench.
Problem
Describing in natural language how two sets of images differ, which is hard for humans to do manually over thousands of images and not handled by existing vision-language models that cannot reason over large image sets.
Approach
A two-stage proposer-ranker framework adapted from language-domain D3/D5 to images: BLIP-2 captions of sampled subsets are fed to GPT-4 to propose candidate differences, and CLIP feature-based AUROC scoring over all images ranks and filters them by statistical significance.
Key results
- VisDiff (BLIP-2 caption proposer + CLIP ranker) reaches Acc@1 of 88%, 75%, 61% and Acc@5 of 99%, 86%, 80% on PairedImageSets Easy/Medium/Hard.
- Caption-based proposer consistently outperforms image-based and feature-based proposers across VisDiffBench subsets.
- Under noise, Acc@1 on PairedImageSets-Hard remains at 49% at 40% purity versus 63% at 100% purity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the caption-based proposer (BLIP-2 captions + GPT-4) with the CLIP feature ranker | VisDiff · VisDiffBench PIS-Hard | Acc@1: 0.38 -> 0.61 | Helped |
| Use the caption-based proposer (BLIP-2 captions + GPT-4) with the CLIP feature ranker | VisDiff · VisDiffBench PIS-Hard | Acc@5: 0.62 -> 0.80 | Helped |
| Replace the feature-based proposer with the caption-based proposer (ablation) | VisDiff proposer with CLIP ranker · VisDiffBench PIS-Hard | Acc@1: 0.12 -> 0.61 | Helped |
| Replace the image-based proposer (LLaVA-1.5) with the caption-based proposer (ablation) | VisDiff proposer with CLIP ranker · VisDiffBench PIS-Hard | Acc@1: 0.28 -> 0.61 | Helped |
| Replace the caption-based ranker (Vicuna-1.5) with the feature-based ranker (CLIP) (ablation) | VisDiff with BLIP-2 + GPT-4 proposer · VisDiffBench PIS-Hard | Acc@1: 0.31 -> 0.61 | Helped |
| Replace the image-based ranker (LLaVA-1.5) with the feature-based ranker (CLIP) (ablation) | VisDiff with BLIP-2 + GPT-4 proposer · VisDiffBench PIS-Hard | Acc@1: 0.38 -> 0.61 | Helped |
Surprising findings
- Feature-based (CLIP embedding arithmetic) proposer performs well on ImageNet-R/ImageNet* but poorly on PairedImageSets because vector arithmetic often lacks meaningful semantics (e.g., cat minus dog).
Limitations
- Substantial room for improvement; hard subset accuracy is only 61% Acc@1.
- Relies on CLIP, GPT, and BLIP, inheriting their biases and errors.
- Intended as an assistive tool requiring a human in the loop.
Experimental setup
Datasets: ImageNet ImageNetV2 ImageNet-R ImageNet* MetaShift LaMem ImageNet classes evaluated with ResNet-50/101 PartiPrompts DiffusionDB
Benchmarks: VisDiffBench
Metrics: Acc@1 Acc@5 AUROC Pearson correlation with human annotators
Baselines: GS-CLIP LLaVA-1.5 image-based proposer GPT-4V image-based proposer BLIP-2 feature-based proposer Vicuna-1.5 caption-based ranker LLaVA-1.5 image-based ranker
Builds on: D3 D5 GS-CLIP
Closest papers by meaning
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.90)
- Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.88)
- VIXEN: Visual Text Comparison Network for Image Difference Captioning (AAAI 2024 · similarity 0.88)
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning (AAAI 2025 · similarity 0.86)
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.