Research map / Vision-language and multimodal / Multimodal LLMs
VisMin: Visual Minimal-Change Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VisMin, a benchmark of minimal-change image pairs (object, attribute, count, spatial relation) built on COCO with LLM+diffusion synthesis and human verification, plus a 64K minimal-change training set used to fine-tune CLIP and Idefics2.
Problem
VLMs are evaluated mostly on distinguishing very similar captions for one image; fine-grained discrimination between minimally different images in complex natural scenes is under-evaluated, especially counting and spatial relations.
Approach
Generate minimal-change image-caption pairs in complex COCO-like scenes via LLM-generated edit instructions and diffusion-based image editing (SDXL inpainting, LMD/GLIGEN layout diffusion, LaMa removal), filter automatically with VQA, and verify with four rounds of human checks; then use the resulting pairs as hard negatives for fine-tuning.
Key results
- VisMin-Idefics2 raises average VisMin score from 55.99 to 86.43 (group score on spatial relation rises to 69.3 from 4.8 for Idefics2 per the table).
- VisMin-CLIP reaches average 63.74 on VisMin versus 46.02 for base CLIP ViT-L/14 and outperforms NegCLIP, CounterCurate-CLIP and SPEC-CLIP on most categories.
- Zero-shot, VisMin-CLIP is best on 11 of 18 fine-grained tasks, and gives 9% gains on CountBench over NegCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · VisMin | AVG: 46.02 -> 63.74 | Helped |
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · VisMin object split | Group score: 78.07 -> 86.36 | Helped |
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · VisMin attribute split | Group score: 57.82 -> 75.85 | Helped |
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · VisMin spatial relation split | Group score: 1.45 -> 1.29 | Hurt |
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · VisMin counting split | Group score: 18.17 -> 72.33 | Helped |
| fine-tune Idefics2 on minimal-change data | Idefics2 · VisMin | AVG: 55.99 -> 86.43 | Helped |
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · CountBench | accuracy: 33.65 -> 49.84 | Helped |
| fine-tune CLIP on minimal-change data | CLIP (ViT-L/14) · ImageNet | top1: 75.53 -> 68.20 | Hurt |
| fine-tune Idefics2 on minimal-change data | Idefics2 · VQAv2 | accuracy: 78.82 -> 79.75 | Helped |
| fine-tune Idefics2 on minimal-change data | Idefics2 · MathVista | accuracy: 53.00 -> 47.80 | Hurt |
Surprising findings
- Current MLLMs underperform foundational VLMs on object and attribute image-score tasks, and most VLMs score below random chance on spatial relations.
- Fine-tuning CLIP on minimal-change data did not improve spatial relation understanding, while Idefics2 improved substantially.
Limitations
- Dataset noise from diffusion model limitations, including image deformations and text-image mismatches.
- Possible inherited social biases from CLIP and Idefics2 with no mitigation applied.
- Uniform evaluation prompts may affect model performance variably.
Experimental setup
Datasets: COCO 2017 VSR VisMin training set CountBench VALSE SugarCrepe SPEC Winoground MMVP ImageCoDe EQBEN What'sUp
Benchmarks: VisMin VSR CountBench VALSE SugarCrepe SPEC What'sUp ImageCoDe MMVP Winoground EQBEN MathVista
Metrics: text score image score group score recall zero-shot accuracy
Baselines: NegCLIP CounterCurate-CLIP SPEC-CLIP Idefics2 LLaVA 1.6 GPT-4o Gemini 1.0 Pro InternVL1.5 CogVLM SigLIP
Builds on: NegCLIP SPEC CounterCurate LLM-grounded Diffusion GLIGEN
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Describing Differences in Image Sets with Natural Language (CVPR 2024 · similarity 0.90)
- Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.90)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.90)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.