Research map / Vision-language and multimodal / Human and embodied modeling
Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RDD, a region-aware difference distilling network that progressively distills object then action regional difference features from global differences via learnable vectors, plus ACR with N2O and V2A contrastive losses aligning them to CLIP-encoded noun/verb tokens, for change captioning.
Problem
Change captioning relies on global difference features, missing fine-grained regional changes and resisting viewpoint and illumination disturbances, and suffers from semantically hallucinated captions due to incomplete linguistic-visual alignment.
Approach
Progressively distill regional object and action difference features from global difference features using learnable vectors in a concrete-to-abstract order, and regularize them with contrastive losses aligning noun/verb attribute tokens (from spaCy-extracted segments encoded by CLIP with learnable prompts) to object/action regional features.
Key results
- On CLEVR-Change, RDD+ACR reaches CIDEr 128.1 and SPICE 33.5, above prior SOTA (SMART/MURAT+GCM at 127.0).
- On LEVIR-CC, RDD+ACR achieves CIDEr 138.3 vs. 137.9 for CARD (prior best).
- On Spot-the-Diff, RDD+ACR reports CIDEr 43.6 vs. 40.9 for DIRL+CCR, but lower BLEU-4 (9.2 vs 10.3) and ROUGE-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RDD+ACR instead of prior methods | Change captioning · CLEVR-Change test | CIDEr: 127.0 -> 128.1 | Helped |
| Use RDD+ACR instead of prior methods | Change captioning · LEVIR-CC test | CIDEr: 137.9 -> 138.3 | Helped |
| Use RDD+ACR instead of prior methods | Change captioning · Spot-the-Diff test | CIDEr: 40.9 -> 43.6 | Helped |
| Add global difference extraction and contextual interaction instead of direct feature subtraction (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 120.4 -> 125.2 | Helped |
| Add object regional difference features alone (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 125.2 -> 124.8 | Hurt |
| Add action regional difference features alone (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 125.2 -> 127.5 | Helped |
| Distill objects before actions rather than simultaneously (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 126.4 -> 128.1 | Helped |
| Distill objects before actions rather than actions before objects (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 127.6 -> 128.1 | Helped |
| Use both N2O and V2A losses rather than neither (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 127.0 -> 128.1 | Helped |
| Add N2O loss while retaining V2A loss (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 127.5 -> 128.1 | Helped |
| Add V2A loss while retaining N2O loss (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 127.2 -> 128.1 | Helped |
| Add a prompt for noun/verb segments rather than using no initialization (ablation) | RDD+ACR on CLEVR-Change · CLEVR-Change | CIDEr: 127.7 -> 128.1 | Helped |
Surprising findings
- On Spot-the-Diff, the method improves METEOR and CIDEr but is slightly worse on BLEU-4 and ROUGE-L, which the authors attribute to short, variable-length captions.
- Increasing the number of learnable vectors does not improve performance, since more non-changing interference is distilled.
Limitations
- Performance on Spot-the-Diff is mixed: lower BLEU-4 and ROUGE-L than some baselines.
- Increasing learnable vector count degrades performance due to distilled non-change interference.
- Attribute extraction relies on spaCy parsing of ground-truth captions, so alignment depends on parser quality.
Experimental setup
Datasets: CLEVR-Change LEVIR-CC Spot-the-Diff
Benchmarks: CLEVR-Change LEVIR-CC Spot-the-Diff
Metrics: BLEU-4 METEOR ROUGE-L CIDEr SPICE
Baselines: DUDA M-VAM R3Net+SSP MCCFormers-D MCCFormers-S IFDC NCT I3N-TD VARD-Trans SCORER+CBR MURAT+GCM SMART
Builds on: InfoNCE CoOp CLIP Transformer
Closest papers by meaning
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning (AAAI 2025 · similarity 0.89)
- Revisiting Change Captioning from Self-supervised Global-Part Alignment (AAAI 2025 · similarity 0.89)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.87)
- VIXEN: Visual Text Comparison Network for Image Difference Captioning (AAAI 2024 · similarity 0.87)
- Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- Describing Differences in Image Sets with Natural Language (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.