Research map / Image generation / Text-to-image diffusion
Origin Identification for Text-Guided Image-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces origin identification (ID²) for text-guided image-to-image diffusion outputs and a dataset (ID²-Dataset-style, built from DISC21) and a linear-transformation-on-VAE-embedding method trained with CosFace that generalizes across diffusion models.
Problem
Identifying the original image of a query edited by text-guided image-to-image diffusion models, where similarity-based retrieval models trained on one diffusion model fail to generalize to outputs from other diffusion models due to visual discrepancy.
Approach
Learn a linear transformation W on VAE embeddings (trained with a CosFace metric loss on one diffusion model's outputs) that brings translations close to their origins; the authors argue, and empirically show, that W transfers to outputs of other diffusion models because the denoising error term is approximately zero and the W singular-value structure is stable across models.
Key results
- The linear-transformed VAE embedding method reaches 86.6% mAP on unseen diffusion models versus 52.2% for the best CosFace similarity baseline (+31.6% over best similarity-based method per paper's comparison), and 88.8% mAP on seen SD2 queries.
- Per-model mAP on the test set: 88.8 (SD2), 81.5 (SDXL), 87.3 (OpenDalle), 89.3 (ColorfulXL), 85.7 (Kandinsky-3), 85.7 (SD3), 90.3 (Kolors).
- Training is 8.6x faster than the nearest competitor and matching is about 875x faster than QAConv-GS, using 0.53e-10 s/pair versus 464e-10 s/pair.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a learned linear transformation on VAE embeddings instead of the strongest similarity-based baseline | ID² unseen retrieval · ID² unseen (average of six models) | mAP: 55.0 -> 86.6 | Helped |
| Use a learned linear transformation on VAE embeddings instead of QAConv-GS | ID² unseen retrieval · ID² unseen (average of six models) | mAP: +10.8% mAP | Helped |
| Replace Stable Diffusion 2 VAE encoder with Open-Sora VAE (ablation) | Linear Trans. · ID² seen | mAP: 88.8 -> 86.3 | Hurt |
| Replace Stable Diffusion 2 VAE encoder with Open-Sora VAE (ablation) | Linear Trans. · ID² unseen | mAP: 86.6 -> 86.5 | No clear effect |
| Replace Stable Diffusion 2 VAE encoder with Open-Sora-Plan VAE (ablation) | Linear Trans. · ID² seen | mAP: 88.8 -> 88.8 | No clear effect |
| Replace Stable Diffusion 2 VAE encoder with Open-Sora-Plan VAE (ablation) | Linear Trans. · ID² unseen | mAP: 86.6 -> 86.1 | No clear effect |
| Replace CosFace supervision with Circle loss (ablation) | Linear Trans. · ID² seen | mAP: 88.8 -> 84.9 | Hurt |
| Replace CosFace supervision with Circle loss (ablation) | Linear Trans. · ID² unseen | mAP: 86.6 -> 82.5 | Hurt |
| Replace CosFace supervision with SoftMax (ablation) | Linear Trans. · ID² seen | mAP: 88.8 -> 76.1 | Hurt |
| Replace CosFace supervision with SoftMax (ablation) | Linear Trans. · ID² unseen | mAP: 86.6 -> 62.4 | Hurt |
| Replace linear transformation with a convolutional layer (ablation) | Linear Trans. · ID² unseen | mAP: 86.6 -> 32.5 | Hurt |
| Replace linear transformation with attention (ablation) | Linear Trans. · ID² seen | mAP: 88.8 -> 89.0 | Helped |
Surprising findings
- A linear transformation of VAE embeddings generalizes across diffusion models even when their VAEs have different parameters and architectures, and across a video VAE (Open-Sora) with less than 1% mAP drop.
- Non-linear transformations (MLPs with more layers) overfit: seen-model mAP rises while unseen-model mAP drops, and a convolutional layer underfits.
Limitations
- Fails on IP-Adapter, which uses CLIP encodings rather than VAE; linear CLIP transform reaches only 36.6% mAP.
- Fails on hard negatives that are visually very similar to the query; VAE compression loses fine-grained detail.
- Theoretical guarantee covers only the image-to-image SDEdit paradigm; the generalization to other paradigms is empirical and the authors state Hypothesis 1 is unproven.
Experimental setup
Datasets: DISC21 YFCC100M SEED-Data-Edit
Benchmarks: ID2 dataset test set (7 diffusion models) SEED-Data-Edit
Metrics: mAP Top-1 accuracy
Baselines: Circle loss SoftMax CosFace IBN-Net TransMatcher QAConv-GS AnyPattern SSCD Swin-B ViT-B DINOv2 MAE
Builds on: SDEdit CosFace Stable Diffusion 2
Closest papers by meaning
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.88)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.87)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.87)
- Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.87)
- $\text{I}^2\text{AM}$: Interpreting Image-to-Image Latent Diffusion Models via Bi-Attribution Maps (ICLR 2025 · similarity 0.87)
- Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.