Research map / Image generation / Text-to-image diffusion
VIXEN: Visual Text Comparison Network for Image Difference Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIXEN, a two-image soft-prompt network that projects CLIP or BLIP-2 Q-Former features of an image pair into GPT-J to generate text summaries of manipulations, trained on a GPT-3-captioned augmentation of InstructPix2Pix.
Problem
Image difference captioning for manipulation detection needs comprehensible text summaries of edits, but existing IDC datasets are small and lack manipulation variety, and prior methods rely on benchmark-specific properties.
Approach
Pairwise linear projection of image features from each of two images into a frozen-style large language model soft prompt (concatenated), trained on synthetic prompt-to-prompt edits from InstructPix2Pix with GPT-3-generated change summaries and distractor same-image pairs.
Key results
- On the IP2P subset (threshold 3), VIXEN-C reaches MPNet 59.3 and CIDEr 96.6 versus CLIP4IDC 56.8 and 58.8.
- On PSBattles, VIXEN-Q scores highest on MPNet (45.1) and B@4 (5.8), indicating better generalization than baselines.
- In the MTurk preference study on IP2P, VIXEN-Q and VIXEN-C win a majority vote in 32% and 26% of cases versus 24% for CLIP4IDC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIXEN-C instead of CLIP4IDC | image difference captioning · Instruct Pix2Pix @3 | CIDEr: 58.8 -> 96.6 | Helped |
| Use VIXEN-Q instead of CLIP4IDC | image difference captioning · PSBattles | MPNet: 32.7 -> 45.1 | Helped |
| Fine-tune VIXEN-C on Image Editing Request | VIXEN-C · Image Editing Request | B@4: 10.0 -> 8.6 | Hurt |
| Train with same-image distractor pairs (p=0.5 rather than p=0) (ablation) | VIXEN-C · Instruct Pix2Pix | MPNet: 54.4 -> 59.3 | Helped |
| Train with same-image distractor pairs (p=0.5 rather than p=0) (ablation) | VIXEN-C · PSBattles | MPNet: 37.8 -> 40.3 | Helped |
| Use concatenation instead of subtraction for image feature fusion (ablation) | VIXEN-C · Instruct Pix2Pix | B@4: 16.4 -> 16.8 | Helped |
| Use concatenation instead of addition for image feature fusion (ablation) | VIXEN-C · Instruct Pix2Pix | B@4: 16.2 -> 16.8 | Helped |
| Use concatenation instead of multiplication for image feature fusion (ablation) | VIXEN-C · Instruct Pix2Pix | B@4: 14.4 -> 16.8 | Helped |
| Use concatenation instead of mean for image feature fusion (ablation) | VIXEN-C · Instruct Pix2Pix | B@4: 10.7 -> 16.8 | Helped |
| Use VIXEN-C instead of VARD | image difference captioning · Image Editing Request | CIDEr: 35.7 -> 38.1 | Helped |
Surprising findings
- Participants chose 'none of the above' in 75% of PSBattles cases versus 2% on IP2P, showing poor transfer to real-domain data.
- Mean fusion of the two image features causes a large performance drop, while concatenation performs best.
Limitations
- Fails on very minor differences, falling back to image captioning
- Language model runoff generates unsupported details
- Large synthetic-to-real performance gap; trained only on generated manipulations (no insertion, deletion or text edits)
Experimental setup
Datasets: InstructPix2Pix (augmented with GPT-3 summaries) PSBattles Image Editing Request
Benchmarks: InstructPix2Pix subset PSBattles Image Editing Request
Metrics: MPNet semantic similarity BLEU-4 CIDEr METEOR ROUGE-L user preference rate
Baselines: IDC CLIP4IDC VARD NCT BiDiff DUDA rel-att
Builds on: LLaVA-style linear projection of visual features into LLM BLIP-2 InstructPix2Pix prompt-to-prompt CLIP4IDC
Closest papers by meaning
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.88)
- Describing Differences in Image Sets with Natural Language (CVPR 2024 · similarity 0.88)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.87)
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning (AAAI 2025 · similarity 0.87)
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning (AAAI 2025 · similarity 0.87)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.