Research map / Vision-language and multimodal / Multimodal LLMs
Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Img-Diff, an automated contrastive data synthesis pipeline that generates pairs of similar images with object replacements, localizes differing regions, and writes region-focused difference captions, then fine-tunes MLLMs on the resulting dataset.
Problem
MLLMs struggle with fine-grained image recognition and distinguishing visually similar images with different content, and existing image difference datasets are either small, manual, or describe whole-image differences rather than specific regions.
Approach
Generate highly similar image pairs differing by one replaced object (SDXL with Prompt-to-Prompt), locate differing regions via segmentation (FastSAM) and CLIP/BLIP filters, and produce region-focused difference captions with an MLLM using red-box highlighted images, then use these as instruction-tuning data.
Key results
- Fine-tuning LLaVA-1.5-7B on IMG-DIFF raises MMVP from 24.0 to 27.3 (and 31.3 with 35K extra samples), surpassing GPT-4V and Gemini (38.7 and 40.7 are their scores; MGM-7B+RP reaches 50.7).
- Average gain across eight MLLM benchmarks: +3.06% for LLaVA-1.5-7B, +1.28% for MGM-7B, +1.01% for InternVL2-8B.
- Human quality assessment on 1,000 samples: nearly 80% have valid object differences in highlighted regions and over 70% of difference captions are fully accurate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaVA-1.5-7B with IMG-DIFF object-replacement data | LLaVA-1.5-7B · MMVP | score: 24 -> 27.3 | Helped |
| fine-tune MGM-7B with IMG-DIFF object-replacement data | MGM-7B · MMVP | score: 40 -> 50.7 | Helped |
| fine-tune InternVL2-8B with IMG-DIFF object-replacement data | InternVL2-8B · MMVP | score: 38.7 -> 43.3 | Helped |
| fine-tune LLaVA-1.5-7B with IMG-DIFF object-replacement data | LLaVA-1.5-7B · Spot-the-Diff | BLEU: 8.5 -> 9.7 | Helped |
| fine-tune MGM-7B with IMG-DIFF object-replacement data | MGM-7B · Spot-the-Diff | CIDEr-D: 46.3 -> 53.5 | Helped |
| fine-tune LLaVA-1.5-7B with IMG-DIFF object-replacement data | LLaVA-1.5-7B · Image-Edit-Request | CIDEr-D: 60.6 -> 60.9 | Helped |
| fine-tune MGM-7B with IMG-DIFF object-replacement data | MGM-7B · Image-Edit-Request | BLEU: 16.5 -> 16.6 | Helped |
| fine-tune LLaVA-1.5-7B with IMG-DIFF object-replacement data | LLaVA-1.5-7B · MM-Vet | score: 30.5 -> 33.2 | Helped |
| fine-tune InternVL2-8B with IMG-DIFF object-replacement data | InternVL2-8B · VQAv2 | score: 81.8 -> 81.8 | No clear effect |
| fine-tune MGM-7B with IMG-DIFF object-replacement data | MGM-7B · MMBCN | score: 58.9 -> 59.6 | Hurt |
| fine-tune LLaVA-1.5-7B with IMG-DIFF data generated using MSCOCO captions and additional LLaVA pre-training captions (ablation) | LLaVA-1.5-7B + RP(13K) · MMVP | score: 27.3 -> 31.3 | Helped |
| replace LLM object-name substitutions with lexicon-based noun replacements (ablation) | LLaVA-1.5-7B · MMVP | score: 24.0 -> 30.0 | Helped |
Surprising findings
- Fine-tuned LLaVA-1.5-7B on the synthetic data outperforms the larger LLaVA-1.5-13B on MMVP.
- Quadrupling the dataset yields a 3.40% average gain versus 3.06% for the original, indicating strongly sublinear returns.
Limitations
- Gains on general MLLM benchmarks are modest (about 1-3% average) and smaller for stronger baselines like InternVL2-8B.
- Pipeline relies on several MLLM and filter thresholds (CLIP, BLIP, LLaVA-NEXT) whose errors propagate; roughly 20% of bounding boxes judged low quality or inaccurate in labels.
- Data is restricted to object replacement within MSCOCO/LLaVA caption domains, with object-category coverage limited to 1,203 categories.
Experimental setup
Datasets: MSCOCO captions LLaVA pre-training captions Spot-the-Diff Image-Edit-Request CLEVR-Change LLaVA visual instruction tuning data Object365 (for coverage analysis)
Benchmarks: MMVP Spot-the-Diff Image-Edit-Request VQAv2 GQA POPE MMBench MMBench-CN MM-Vet ScienceQA-IMG SEED-Bench
Metrics: accuracy BLEU METEOR CIDEr-D ROUGE-L average relative improvement
Baselines: GPT-4V Gemini VARD CLIP4IDC NCT BiDiff VIXEN VAM IFDC DUDA+Aux VACC InternVL2-8B-FT
Builds on: InstructPix2Pix Prompt-to-Prompt Data-Juicer LLaVA-1.5 CoCoT
Closest papers by meaning
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.89)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.88)
- Describing Differences in Image Sets with Natural Language (CVPR 2024 · similarity 0.88)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.