Research map / Image generation / Text-to-image diffusion
Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the-Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CATOD, an active-learning framework that iteratively selects high-quality training images for OOD concepts using a weighted aesthetic and concept-matching score, and trains LoRA/DreamBooth/Textual Inversion adaptors on Stable Diffusion 2.0.
Problem
Adaptor modules (LoRA, DreamBooth, Textual Inversion) fail to accurately depict out-of-distribution concepts in text-to-image diffusion models, largely because of low-quality training data.
Approach
Treat adaptor training as an active learning loop in which samples are scored by a dynamically weighted combination of a personalized aesthetic scorer and a VLAD-style concept-matching scorer, and the training schedule is governed by a trigonometric balance indicator over generated images.
Key results
- CATOD with LoRA improves average CLIP score by up to 11.10 over random sampling (RAND) and 4.91 over CLIP-score sampling on single-concept adaption (Table 1).
- CATOD reduces CMMD by about 0.12 to 0.54 relative to baselines in single-concept adaption, with LoRA reaching 0.84 average CMMD versus 1.47 for RAND.
- In multi-concept adaption, CATOD yields a 1.56 to 5.07 CLIP score gain and 0.12 to 0.86 CMMD decrease versus baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CATOD instead of CLIP-based sample selection | LoRA · Single-concept OOD (25 concepts; category average) | CLIP score: 73.06 -> 77.97 | Helped |
| Use CATOD instead of CLIP-based sample selection | LoRA · Single-concept OOD (25 concepts; category average) | CMMD: 1.18 -> 0.84 | Helped |
| Use CATOD instead of CLIP-based sample selection | LoRA · Multi-concept OOD (category average) | CLIP score: 71.23 -> 74.29 | Helped |
| Use CATOD instead of CLIP-based sample selection | LoRA · Multi-concept OOD (category average) | CMMD: 1.54 -> 1.24 | Helped |
| Add an aesthetic scorer to concept-matching-only CATOD (ablation) | LoRA CATOD · Lizard and penguin | CLIP score: 68.45 -> 75.35 (lizard); 70.16 -> 73.24 (penguin) | Helped |
| Add a concept-matching scorer to aesthetic-only CATOD (ablation) | LoRA CATOD · Lizard and penguin | CLIP score: 73.19 -> 75.35 (lizard); 72.85 -> 73.24 (penguin) | Mixed |
| Add weighted scoring to CATOD with both scorers (ablation) | LoRA CATOD · Lizard and penguin | CLIP score: 75.35 -> 77.00 (lizard); 73.24 -> 74.11 (penguin) | Helped |
| Add weighted scoring to CATOD with both scorers (ablation) | LoRA CATOD · Lizard and penguin | CMMD: 0.94 -> 0.89 (lizard); 0.93 -> 0.71 (penguin) | Helped |
| Use the personalized aesthetic scorer instead of ReLIC (ablation) | LoRA CATOD · Lizard and penguin | CLIP score: 71.35 -> 73.19 (lizard); 72.39 -> 72.85 (penguin) | Helped |
| Use the personalized aesthetic scorer instead of TANet (ablation) | LoRA CATOD · Lizard and penguin | CMMD: 1.32 -> 1.14 (lizard); 1.35 -> 1.28 (penguin) | Helped |
Surprising findings
- Recent state-of-the-art aesthetic scorers (ReLIC, TANet, BAID) did not improve OOD adaption and sometimes lowered performance versus the personalized scorer.
- Simply adding aesthetic and concept-matching scores without dynamic weighting caused consistent performance loss on both CLIP and CMMD.
Limitations
- Evaluation limited to 25 concepts in five categories using SD 2.0.
- Some generated images still show confused visual details that hurt aesthetics (Figure 7 discussion).
- Dataset release is conditional on code release; performance saturates after a few cycles and extra samples may not help.
Experimental setup
Datasets: ImageNet iNaturalist 2018 IP102
Benchmarks: 25 OOD concepts across insect, lizard, penguin, seafish, snake categories
Metrics: CLIP score CMMD
Baselines: DreamBooth + RAND DreamBooth + CLIP TI + RAND TI + CLIP LoRA + RAND LoRA + CLIP ReLIC TANet BAID
Builds on: LoRA DreamBooth Textual Inversion Latent Diffusion Models PA-IAA CMMD
Closest papers by meaning
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.87)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.87)
- Ambient Diffusion Omni: Training Good Models with Bad Data (NeurIPS 2025 · similarity 0.87)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
- ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning (ICLR 2025 · similarity 0.86)
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.