Research map / Image generation / Text-to-image diffusion
Exposing Text-Image Inconsistency Using Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-TIIL, which uses Stable Diffusion's text embedding alignment, diffusion-based image editing and denoising to localize text-image inconsistencies at pixel and word level, and introduces the TIIL dataset of about 14K pairs.
Problem
Existing text-image inconsistency detectors are classification-based and give no explainable evidence, and existing datasets lack word- and region-level inconsistency annotations.
Approach
Uses a text-to-image diffusion model as an omniscient knowledge source: aligns text embeddings to the image and to a diffusion-edited image via embedding optimization, derives inconsistency masks from differences in UNet noise predictions, and scores consistency from CLIP similarity of the masked image.
Key results
- D-TIIL reaches 47.12% mIoU on TIIL localization versus 27.35% for GAE and 14.24% for DetCLIP.
- On binary detection, D-TIIL achieves 87.00% AUC and 79.46% accuracy, above CCN (82.80% AUC, 74.10% accuracy) and CLIP (84.18% AUC).
- Text denoising step improves mIoU from 38.43% (coarse mask M') to 47.12% (final mask M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use D-TIIL instead of DetCLIP or GAE for localization | TIIL · TIIL | mIoU (%): 14.24, 27.35 -> 47.12 | Helped |
| use D-TIIL instead of CLIP, CLIP*, or CCN for detection | TIIL · TIIL | AUC (%): 84.18, 78.58, 82.80 -> 87.00 | Helped |
| use D-TIIL instead of CLIP, CLIP*, or CCN for detection | TIIL · TIIL | Accuracy (%): 76.77, 71.87, 74.10 -> 79.46 | Helped |
| add image-regulated text denoising rather than using M′ from Step 2 (ablation) | D-TIIL · TIIL | mIoU (%): 38.43 -> 47.12 | Helped |
| initialize the aligned text embedding with E₀ instead of random noise (ablation) | D-TIIL · TIIL subset (1,000 image-text pairs) | mIoU (%): 41.27 -> 47.50 | Helped |
| add the text embedding constraint loss (ablation) | D-TIIL · TIIL subset (1,000 image-text pairs) | mIoU (%): 43.47 -> 47.50 | Helped |
| use adaptive average-based mask thresholding instead of fixed thresholds (ablation) | D-TIIL · TIIL subset (1,000 image-text pairs) | mIoU (%): 39.66, 42.81, 43.23, 42.66 -> 47.50 | Helped |
| evaluate D-TIIL on text-changed rather than image-changed examples | D-TIIL · TIIL image-changed vs text-changed | mIoU: 50.28 -> 45.57 | Hurt |
| use D-TIIL instead of CLIP for detection on image-changed examples | TIIL image-changed · TIIL image-changed | AUC: 86.50 -> 88.80 | Helped |
| use D-TIIL instead of CLIP for detection on text-changed examples | TIIL text-changed · TIIL text-changed | AUC: 78.58 -> 82.29 | Helped |
Surprising findings
- D-TIIL does not rely on reverse image search, and its accuracy on manipulated-image subsets (79.15%) stays close to its accuracy on original-image subsets (82.79%), while CCN drops from 80.12% to 68.0%.
Limitations
- Relies on the general-purpose diffusion model's limited external or domain knowledge, which may miss inconsistencies needing specialized knowledge.
- Diffusion models can take words too literally and struggle with precise local edits, causing failure cases.
- TIIL is built with DALL-E 2 edits and manual annotation by six annotators, so its inconsistencies are synthetic edits rather than natural misinformation.
Experimental setup
Datasets: Visual News NewsCLIPpings
Benchmarks: TIIL
Metrics: mIoU AUC Accuracy
Baselines: DetCLIP GAE CLIP CLIP* CCN
Builds on: Stable Diffusion CLIP DDIM classifier-free guidance DALL-E 2
Closest papers by meaning
- Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search (ICLR 2024 · similarity 0.86)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.86)
- Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation (NeurIPS 2025 · similarity 0.85)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.85)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.85)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.