Research map / Image generation / Text-to-image diffusion
ArtiFade: Learning to Generate High-quality Subject from Blemished Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a pre-trained latent diffusion model (key/value attention weights plus an artifact-free text embedding) on paired unblemished/blemished subject images so that Textual Inversion and DreamBooth subject-driven generation can work from artifact-laden inputs.
Problem
Subject-driven text-to-image generation degrades sharply when the few reference images contain visible artifacts (watermarks, stickers, glass effects) or invisible adversarial noise, since existing methods assume clean training images.
Approach
Build a dataset pairing each unblemished subject image with artificially blemished versions (10 watermark types), learn blemished Textual Inversion embeddings per subset, then fine-tune only the attention key/value weights plus an extra artifact-free embedding so that the blemished embedding maps to the clean subject; the same recipe is also applied to DreamBooth-LoRA.
Key results
- On WM-ID-test, ArtiFade (TI-based) raises IDINO from 0.217 (blemished TI) to 0.337 and RDINO from 0.852 to 1.300.
- On WM-OOD-test, ArtiFade improves IDINO from 0.229 to 0.356 and RDINO from 0.858 to 1.237 over blemished Textual Inversion.
- DreamBooth-based ArtiFade reaches IDINO 0.589 and ICLIP 0.795 on WM-ID-test, the highest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ArtiFade with Textual Inversion instead of vanilla Textual Inversion to blemished inputs | Textual Inversion · WM-ID-test | IDINO: 0.217 -> 0.337 | Helped |
| Apply ArtiFade with Textual Inversion instead of vanilla Textual Inversion to blemished inputs | Textual Inversion · WM-ID-test | RDINO: 0.852 -> 1.300 | Helped |
| Apply ArtiFade with Textual Inversion instead of vanilla Textual Inversion to blemished inputs | Textual Inversion · WM-ID-test | ICLIP: 0.576 -> 0.649 | Helped |
| Apply ArtiFade with Textual Inversion instead of vanilla Textual Inversion to blemished inputs | Textual Inversion · WM-OOD-test | IDINO: 0.229 -> 0.356 | Helped |
| Apply ArtiFade with Textual Inversion instead of vanilla Textual Inversion to blemished inputs | Textual Inversion · WM-OOD-test | RDINO: 0.858 -> 1.237 | Helped |
| Apply ArtiFade with DreamBooth instead of vanilla DreamBooth | DreamBooth · WM-ID-test | IDINO: 0.503 -> 0.589 | Helped |
| Apply ArtiFade with DreamBooth instead of vanilla DreamBooth | DreamBooth · WM-ID-test | RDINO: 0.874 -> 1.308 | Helped |
| Apply ArtiFade with DreamBooth instead of vanilla DreamBooth | DreamBooth · WM-ID-test | TCLIP: 0.272 -> 0.284 | Helped |
| Add partial fine-tuning of key and value weights to artifact-free-embedding-only training (ablation) | ArtiFade (VarA) · WM-ID-test | IDINO: 0.154 -> 0.337 | Helped |
| Fine-tune query weights along with the artifact-free embedding rather than key and value weights with the embedding (ablation) | ArtiFade (VarB) · WM-ID-test | RDINO: 1.230 -> 1.300 | Helped |
| Add the artifact-free embedding while fine-tuning key and value weights (ablation) | ArtiFade (VarC) · WM-ID-test | RCLIP: 1.019 -> 1.020 | Helped |
| Add the artifact-free embedding while fine-tuning key and value weights (ablation) | ArtiFade (VarC) · WM-ID-test | TCLIP: 0.280 -> 0.282 | Helped |
Surprising findings
- Fine-tuning only the artifact-free embedding (VarA) overfits and yields the lowest subject fidelity, showing partial weight fine-tuning is necessary.
- Blemished Textual Inversion with DreamBooth (DB blemished) achieves higher IDINO than ArtiFade (TI-based) but lower artifact-removal ratio RDINO.
Limitations
- Evaluation relies on CLIP and DINO similarity metrics with only 16 test subjects and 80 test sets.
- Main experiments focus on Textual Inversion and DreamBooth; the approach requires training a separate fine-tuned model per artifact family (WM-model).
- Artifact types in training are synthetic watermarks; the method's generalization to real-world artifacts is only illustrated qualitatively.
Experimental setup
Datasets: Textual Inversion subject images DreamBooth subject images VGGFace2
Benchmarks: WM-ID-test (16 novel subjects, 5 in-distribution watermarks) WM-OOD-test (5 out-of-distribution watermarks)
Metrics: IDINO ICLIP TCLIP RDINO RCLIP
Baselines: Textual Inversion DreamBooth Anti-DreamBooth
Builds on: Textual Inversion DreamBooth Latent Diffusion Model Anti-DreamBooth
Closest papers by meaning
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.86)
- Yuan: Yielding Unblemished Aesthetics Through a Unified Network for Visual Imperfections Removal in Generated Images (AAAI 2025 · similarity 0.85)
- OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance (NeurIPS 2024 · similarity 0.85)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.85)
- Alchemist: Turning Public Text-to-Image Data into Generative Gold (NeurIPS 2025 · similarity 0.84)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.