Research map / Image generation / Editing and style transfer
Difference Inversion: Interpolate and Isolate the Difference with Token Consistency for Image Analogy Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Difference Inversion learns special "DIFF" tokens that capture the A-to-A' difference (via Slerp-interpolated CLIP image/text deltas, a token consistency loss, and zero initialization) and applies them to B within general Stable Diffusion models for image analogy generation.
Problem
Image analogy generation: produce B' satisfying A:A'::B:B' from an image triplet without text instructions, while prior methods are tied to specific models (InstructPix2Pix, inpainting models) and leak residual information from A and A'.
Approach
Extract only the bidirectional difference between A and A' as optimized discrete-prompt "DIFF" tokens, using an interpolated CLIP delta (Slerp of image and caption-text deltas), a cycle-style token consistency loss, and zero-initialized embeddings, so the tokens plug into any Stable Diffusion prompt.
Key results
- DINO-v2 directional score of 0.5732 vs 0.5414 for DIA and 0.5723 for VISII (best among image-only methods).
- Human evaluation with 60 participants: 72% preference for Difference Inversion over the other baselines combined in four-option selection.
- VLM pairwise win rates against baselines range from 60% to 92% (e.g., 92% vs DIA with Qwen2-VL-72B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Difference Inversion instead of the image-only baselines | Image analogy generation · Image analogy generation | DINO-v2 directional score: 0.5114 (Analogist), 0.5414 (VISII) -> 0.5732 | Helped |
| Use Difference Inversion instead of the image-only baselines | Image analogy generation · Image analogy generation | human preference: 4%, 15%, 9% -> 72% | Helped |
| Use Difference Inversion instead of DIA | Image analogy generation · Image analogy generation vs DIA | VLM win rate: 90%, 92%, 72% | Helped |
| Add Delta Interpolation combining image and text deltas (ablation) | Difference Inversion · Image analogy generation | difference capture | Helped |
| Add Token Consistency Loss (ablation) | Difference Inversion · Image analogy generation | consistency with the query image | Helped |
| Zero-initialize Difference Token embeddings instead of random initialization (ablation) | Difference Inversion · Image analogy generation | difference reflection | Helped |
| Add CLIP loss (ablation) | Difference Inversion · Image analogy generation | difference capture | Helped |
| Use 5 Difference Tokens (ablation) | Difference Inversion · Image analogy generation | difference capture | Helped |
| Set the interpolation ratio alpha to 0.8 (ablation) | Difference Inversion · Image analogy generation | difference capture | Helped |
Surprising findings
- CLIP-based directional score of VISII is nearly comparable to the proposed method, attributed to VISII being trained on the evaluation data distribution.
- Random initialization of DIFF tokens lets unwanted information from the original images leak into the difference tokens.
Limitations
- Struggles with global differences such as weather shifts or image-to-sketch conversions.
- Struggles with detailed differences (e.g., expressions, small added objects) because CLIP space may not align with human perception.
Experimental setup
Datasets: InstructPix2Pix dataset
Benchmarks: InstructPix2Pix dataset subset (300 samples)
Metrics: CLIP directional score DINO-v2 directional score human preference VLM pairwise win rate
Baselines: SDEdit InstructPix2Pix DIA Analogist VISII
Builds on: Textual Inversion VISII DIA PEZ Slerp ControlNet
Closest papers by meaning
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- DCI: Dual-Conditional Inversion for Boosting Diffusion-Based Image Editing (NeurIPS 2025 · similarity 0.87)
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning (AAAI 2025 · similarity 0.86)
- Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion (ICML 2024 · similarity 0.86)
- Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.86)
- AID: Attention Interpolation of Text-to-Image Diffusion (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.