Research map / Image generation / Editing and style transfer
Model Diagnosis and Correction via Linguistic and Implicit Attribute Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MDC, an automated pipeline that discovers linguistic and implicit visual attributes behind classifier errors, validates them via a diffusion-based attribute editor using soft-token learning, and retrains with counterfactual samples depicting verified causal attributes.
Problem
Diagnosing why a deep visual classifier makes errors and correcting it, especially when causal attributes are implicit and cannot be described in natural language, without requiring user-specified attributes.
Approach
Automatically discover candidate attributes from error clusters using an MLLM plus DINOv2 clustering, convert implicit image groups into learned soft text tokens aligned with CLIP directional loss, and train a DiffAE-based editor with per-attribute mapping networks to generate counterfactual validation and training sets.
Key results
- MDC Top-2 improves CUB ViT-L/14 Acc@1 from 90.80% to 92.98% and Stanford Dogs from 90.54% to 91.56%.
- In face anti-spoofing with CLIP ViT-B/16, MDC Top-2 reduces HTER on MSU-MFSD from 6.19% to 2.46%, reported as a 62.01% average relative improvement over LANCE.
- Editing FID on Idiap Replay Attack drops to 40.81 for MDC versus 76.00-87.04 for language-guided baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDC correction with Top 2 verified attributes instead of LANCE correction | ViT-L/14 · Stanford Dogs test | Acc@1: 90.98% -> 91.56% | Helped |
| Use MDC correction with Top 2 verified attributes instead of LANCE correction | ViT-L/14 · NABirds test | Acc@1: 91.89% -> 92.98% | Helped |
| Use MDC correction with Top 1 verified attribute instead of LANCE correction | ViT-L/14 · Tiny ImageNet test | Acc@1: 91.74% -> 93.75% | Helped |
| Use MDC correction with Top 2 verified attributes instead of LANCE correction | CLIP ViT-B/16 · Idiap Replay Attack | HTER: 3.43% -> 1.23% | Helped |
| Use MDC correction with Top 2 verified attributes instead of LANCE correction | CLIP ViT-B/16 · MSU-MFSD | HTER: 5.74% -> 2.46% | Helped |
| Use MDC attribute editing instead of language-guided editing | Image editing models · Stanford Dogs | FID: 95.66 -> 78.24 | Helped |
| Use MDC attribute editing instead of language-guided editing | Image editing models · Idiap Replay Attack | FID: 87.04 -> 40.81 | Helped |
Surprising findings
- Language-guided editing baselines conditioned on MLLM-generated descriptions fail to capture implicit patterns such as overall style or color, motivating image-based guidance.
Limitations
- Relies on a closed-source MLLM (Claude-3-Sonnet) for attribute discovery.
- Editing model requires large-scale pretraining (400M steps on ImageNet-22K), which is costly.
- Linguistic attribute editing performance is only on par with baselines.
Experimental setup
Datasets: Stanford Dogs CUB200-2011 Tsinghua Dogs NABirds Tiny ImageNet OULU-NPU CASIA-MFSD Idiap Replay Attack MSU-MFSD ImageNet-22K
Benchmarks: Stanford Dogs CUB200-2011 Tsinghua Dogs NABirds Tiny ImageNet OULU-NPU/CASIA-MFSD to Idiap Replay Attack/MSU-MFSD
Metrics: Acc@1 Acc@5 HTER AUC TPR@FPR=1% FID CLIP score unchanged ratio FNR/FPR deltas
Baselines: LANCE InstructPix2Pix LEDITS CycleDiffusion FreePromptEditing
Builds on: DiffAE LANCE InstructPix2Pix CLIP DINOv2
Closest papers by meaning
- SDGAN: Disentangling Semantic Manipulation for Facial Attribute Editing (AAAI 2024 · similarity 0.85)
- Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.85)
- Counterfactual Image Editing (ICML 2024 · similarity 0.84)
- CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing (AAAI 2025 · similarity 0.84)
- Automated Detection of Visual Attribute Reliance with a Self-Reflective Agent (NeurIPS 2025 · similarity 0.84)
- UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.