Research map / Vision-language and multimodal / Human and embodied modeling
Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Disentangles diffusion backbone features into semantic and visual components using an auto-generated inconsistency dataset and proposes the VSM metric to quantify and localize visual inconsistencies in subject-driven generation.
Problem
Existing metrics for subject-driven image generation (CLIP, DINO, VLM judges) are global and cannot reliably quantify or localize fine-grained visual inconsistencies of a subject across images with pose/scale changes.
Approach
Build an automated pipeline that inpaints matched semantic regions of consistent image pairs to create controlled visual inconsistencies, then train a contrastive aggregation network on frozen diffusion features to separate semantic from visual feature spaces, and score consistency via the fraction of semantically matched points that are also visually matched.
Key results
- On controlled inconsistency, VSM reaches Pearson 0.448 and Spearman 0.582 with the oracle, versus CLIP (-0.053 / -0.005), DINO (0.087 / 0.120), and ChatGPT-4o (0.072 / 0.091).
- On subject-driven generation outputs, VSM reaches Pearson 0.405 and Spearman 0.369, outperforming CLIP (0.156 / 0.112), DINO (0.164 / 0.146), and ChatGPT-4o (0.079 / 0.073).
- Ablation: setting alpha=1 drops Pearson to 0.118; the visual similarity threshold of 0.6 performs best among tested values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VSM instead of global image-level metrics | Visual inconsistency evaluation · Controlled Inconsistency | Pearson correlation: 0.087 -> 0.448 | Helped |
| Use VSM instead of global image-level metrics | Visual inconsistency evaluation · Controlled Inconsistency | Spearman correlation: 0.120 -> 0.582 | Helped |
| Use VSM instead of global image-level metrics | Visual inconsistency evaluation · Subject-Driven Generation | Pearson correlation: 0.164 -> 0.405 | Helped |
| Use VSM instead of global image-level metrics | Visual inconsistency evaluation · Subject-Driven Generation | Spearman correlation: 0.146 -> 0.369 | Helped |
| Set the VSM visual similarity threshold Tv to 0.5 (ablation) | VSM · Controlled Inconsistency | Pearson correlation: 0.448 -> 0.465 | Mixed |
| Set the VSM visual similarity threshold Tv to 0.7 (ablation) | VSM · Controlled Inconsistency | Pearson correlation: 0.448 -> 0.352 | Hurt |
| Set the visual-loss scaling factor alpha to 1 (ablation) | VSM training · Controlled Inconsistency | Pearson correlation: 0.448 -> 0.118 | Hurt |
| Use a dataset-generation skewness threshold of 1.0 (ablation) | VSM training data pipeline · Controlled Inconsistency | Pearson correlation: 0.448 -> 0.232 | Hurt |
| Use a dataset-generation skewness threshold of 1.5 (ablation) | VSM training data pipeline · Controlled Inconsistency | Pearson correlation: 0.448 -> 0.224 | Hurt |
Surprising findings
- ChatGPT-4o correlates almost no better with the oracle than global CLIP or DINO similarity, and its scores cluster between 75 and 95 regardless of visual discrepancy.
Limitations
- Visual features may still carry semantic information, so disentanglement is only partial.
- Training data from Subjects200k is automatically validated and may contain noisy pairs.
- Spatial resolution of diffusion features limits detection of fine-grained inconsistencies.
Experimental setup
Datasets: Subjects200k SPair-71k
Benchmarks: Controlled Inconsistency testset (100 samples) Subject-driven generation evaluation (Diptych, DSD-Diffusion, EasyControl outputs)
Metrics: Pearson correlation Spearman correlation
Baselines: CLIP-I DINO ChatGPT-4o (VLM judge)
Builds on: CleanDIFT SAM Grounded-SAM CLIP Subjects200k
Closest papers by meaning
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.86)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.86)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.86)
- MMLongCite: A Benchmark for Evaluating Faithfulness of Long-Context Vision-Language Models (NeurIPS 2026 · similarity 0.85)
- Exposing Text-Image Inconsistency Using Diffusion Models (ICLR 2024 · similarity 0.85)
- TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.