Research map / Vision-language and multimodal / Human and embodied modeling
Revisiting Change Captioning from Self-supervised Global-Part Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSGPA, a change captioning model that uses Global-Part optimal transport alignment, paired-image part alignment, and a self-supervised fusion change adapter on a frozen EVA-ViT-g/Vicuna-7B backbone to separate true content changes from viewpoint pseudo-changes.
Problem
Image change captioning must describe true content differences between two images while ignoring pseudo changes from viewpoint and global image structure, which existing matching-based methods handle poorly.
Approach
Transfers global image features to local part tokens via optimal transport (Sinkhorn), aligns parts across paired images with unsupervised graph matching, and adds cross-attention change adapters to a frozen vision-language encoder trained with a contrastive consistency constraint separating true and pseudo changes.
Key results
- On CLEVR-Change, SSGPA reaches B4 60.9, METEOR 44.2, CIDEr 159.1, ROUGE-L 80.2, improving over VIR-VLFM by 5.7 CIDEr.
- On Birds-to-Words, SSGPA achieves CIDEr 43.6 versus 25.3 for PLC (+18.3).
- On Spot-the-Diff, SSGPA achieves CIDEr 63.4 versus 56.6 for OneDiff, and on Image Editing Request CIDEr 43.4 versus 35.7 for VARD-T.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSGPA instead of VIR-VLFM | CLEVR-Change · CLEVR-Change | CIDEr: 153.4 -> 159.1 | Helped |
| Use SSGPA instead of PLC | Birds-to-Words · Birds-to-Words | CIDEr: 25.3 -> 43.6 | Helped |
| Use SSGPA instead of VIR-VLFM | Spot-the-Diff · Spot-the-Diff | CIDEr: 48.9 -> 63.4 | Helped |
| Use SSGPA instead of DIRL+CCR | Image Editing Request · Image Editing Request | CIDEr: 41.0 -> 43.4 | Helped |
| Add Global-Part Optimal Transport (ablation) | SSGPA baseline · CLEVR-Change | BLEU-4: 56.7 -> 58.7 | Helped |
| Add Paired Images Part Alignment (ablation) | SSGPA baseline · CLEVR-Change | BLEU-4: 56.7 -> 57.0 | Helped |
| Add Global-Part Transport Alignment (GPOT+PIPA) (ablation) | SSGPA baseline · CLEVR-Change | BLEU-4: 56.7 -> 59.3 | Helped |
| Add Fusion Change Adapter Encoder (ablation) | SSGPA baseline · CLEVR-Change | BLEU-4: 56.7 -> 59.0 | Helped |
| Add Consistency Constraint (ablation) | SSGPA baseline · CLEVR-Change | BLEU-4: 56.7 -> 56.9 | Helped |
| Add Self-supervised Fusion Change Encoding (FCAE+CC) (ablation) | SSGPA baseline · CLEVR-Change | BLEU-4: 56.7 -> 59.8 | Helped |
| Add Global-Part Optimal Transport (ablation) | SSGPA baseline · Birds-to-Words | CIDEr: 24.2 -> 32.1 | Helped |
| Add Paired Images Part Alignment (ablation) | SSGPA baseline · Birds-to-Words | CIDEr: 24.2 -> 31.5 | Helped |
Limitations
- Relies on frozen pretrained encoders and LLM; adapter gains on fine-grained datasets vary by dataset.
- Gains over baselines are modest on some metrics (e.g., Spot-the-Diff BLEU-4 +1.3).
Experimental setup
Datasets: CLEVR-Change Birds-to-Words Spot-the-Diff Image Editing Request
Benchmarks: CLEVR-Change Birds-to-Words Spot-the-Diff Image Editing Request
Metrics: BLEU-4 METEOR CIDEr ROUGE-L
Baselines: DUDA VAM+ DUDA+Aux VARD-T NCT SCORER+CBR DIRL+CCR PLC CLIP4IDC VIR-VLFM Neural Naturalist Relational Speaker
Builds on: BLIP-2 Sinkhorn distances Transformer NCE loss
Closest papers by meaning
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning (AAAI 2025 · similarity 0.89)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.85)
- VIXEN: Visual Text Comparison Network for Image Difference Captioning (AAAI 2024 · similarity 0.85)
- GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.84)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.84)
- Change3D: Revisiting Change Detection and Captioning from A Video Modeling Perspective (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.