Research map / Vision-language and multimodal / Cross-modal learning
DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DECIDER, a non-autoregressive contrastive text diffusion model for image change captioning that uses information-bottleneck difference-aware learning and adversarially perturbed hard positive/negative samples.
Problem
Image change captioning requires describing subtle differences between two similar images; existing methods suffer from irrelevant uni-image features obscuring visual differences and imprecise correspondence between visual differences and generated captions.
Approach
Combines an information-bottleneck-derived difference-aware cross-modal objective that suppresses redundant features with a contrastive text diffusion model whose positive and negative samples are dynamically generated via adversarial perturbations to give a tighter variational bound.
Key results
- On CLEVR-Change, DECIDER (CLIP) reaches CIDEr 134.6 and ROUGE-L 76.5, versus 128.9 and 74.5 for the previous best.
- On Spot-the-Change, CIDEr improves from 38.9 (SCORER) to 40.3 with DECIDER (CLIP).
- Ablation on CLEVR-Change shows the full model (DCL + positive + negative samples) raises CIDEr from 131.8 to 134.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DECIDER with CLIP instead of the strongest listed baseline | Image change captioning · CLEVR-Change | CIDEr-D: 127.0 -> 134.6 | Helped |
| use DECIDER with CLIP instead of the strongest listed baseline | Image change captioning · Image-Editing-Request | CIDEr-D: 35.7 -> 39.5 | Helped |
| use DECIDER with CLIP instead of the strongest listed baseline | Image change captioning · Spot-the-Change | CIDEr-D: 40.9 -> 40.3 | Hurt |
| use DECIDER with CLIP instead of the strongest listed baseline | Image change captioning · Birds-to-Words | CIDEr-D: 19.2 -> 28.4 | Helped |
| add difference-aware cross-modal learning (ablation) | DECIDER without DCL or hard contrastive samples · CLEVR-Change | CIDEr-D: 131.8 -> 133.7 | Helped |
| add difference-aware cross-modal learning (ablation) | DECIDER without DCL or hard contrastive samples · Image-Editing-Request | CIDEr-D: 38.3 -> 39.0 | Helped |
| add hard positive samples (ablation) | DECIDER without DCL or hard contrastive samples · CLEVR-Change | CIDEr-D: 131.8 -> 132.5 | Helped |
| add hard negative samples (ablation) | DECIDER without DCL or hard contrastive samples · CLEVR-Change | CIDEr-D: 131.8 -> 132.8 | Helped |
| add both hard positive and negative samples (ablation) | DECIDER without DCL · CLEVR-Change | CIDEr-D: 131.8 -> 133.6 | Helped |
| add both hard positive and negative samples (ablation) | DECIDER without DCL · Image-Editing-Request | CIDEr-D: 38.3 -> 39.1 | Helped |
| vary contrastive variational loss weight λ (ablation) | DECIDER · CLEVR-Change | CIDEr-D: 134.1; 134.4; 134.6; 134.2; 133.7 | Mixed |
| vary contrastive variational loss weight λ (ablation) | DECIDER · CLEVR-Change | METEOR: 76.3; 76.3; 76.5; 76.4; 76.1 | Mixed |
Surprising findings
- Large vision-language models Qwen-VL and LLaVA-1.5 perform significantly worse than specialized ICC methods despite having much larger parameter counts.
Limitations
- Gains on the more diverse Image-Editing-Request and natural-scene Birds-to-Words datasets are smaller in absolute terms and performance remains low.
- Hard-sample generation relies on several gradient-based perturbation steps, adding training complexity.
Experimental setup
Datasets: CLEVR-Change Image-Editing-Request Spot-the-Change Birds-to-Words
Benchmarks: CLEVR-Change Image-Editing-Request Spot-the-Change Birds-to-Words
Metrics: BLEU-4 METEOR CIDEr-D ROUGE-L
Baselines: VARD-Trans SCORER DIRL SMART NCT Qwen-VL LLaVA-1.5 PCL IDC-PCL
Builds on: CLIP Diffusion-LM contrastive diffusion model (Zhong et al. 2023b) conditional discrete contrastive diffusion (Zhu et al. 2023) FGSM-style gradient sign perturbation
Closest papers by meaning
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning (AAAI 2025 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.87)
- VIXEN: Visual Text Comparison Network for Image Difference Captioning (AAAI 2024 · similarity 0.87)
- Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- Describing Differences in Image Sets with Natural Language (CVPR 2024 · similarity 0.86)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.