Research map / Image generation / Text-to-image diffusion
SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the CCUB dataset of culturally representative image-caption pairs from five countries and SCoFT, a fine-tuning method for Stable Diffusion that adds memorization and self-contrastive perceptual losses to reduce stereotypes and improve cultural relevance.
Problem
Text-to-image models trained on web-crawled data such as LAION produce stereotypical and culturally misrepresentative images, and standard fine-tuning on small culture-specific datasets overfits and struggles to encode high-level cultural concepts.
Approach
Fine-tune Stable Diffusion with LoRA on a small culturally curated dataset, adding a memorization penalty and a self-contrastive perceptual triplet loss computed on pixel-decoded images, using the model's own ControlNet-conditioned generations as negatives.
Key results
- User survey (51 participants, 5 countries): SCoFT+MPC average rank 1.83 for most culturally representative vs 3.07 for Stable Diffusion; least offensive 1.78 vs 3.07.
- Automatic KID on CCUB test set drops from 30.355 (Stable Diffusion) to 19.621 (SCoFT+MPC) (x10^-3), while Stable Diffusion scores higher on CLIP score (0.813 vs 0.799).
- Participants preferred SCoFT+MP over SCoFT+M and SCoFT+MPC over SCoFT+MP 62% of the time for offensiveness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on CCUB using SCoFT+MPC | Stable Diffusion · CCUB test | KID (×10³): 30.355 -> 19.621 | Helped |
| Fine-tune on CCUB using SCoFT+MPC | Stable Diffusion · User survey, five cultures | Average rank: Most Culturally Representative: 3.07 -> 1.84 | Helped |
| Fine-tune on CCUB using SCoFT+MPC | Stable Diffusion · User survey, five cultures | Average rank: Least Stereotypical: 2.98 -> 1.91 | Helped |
| Fine-tune on CCUB using SCoFT+MPC | Stable Diffusion · User survey, five cultures | Average rank: Least Offensive: 3.07 -> 1.78 | Helped |
| Fine-tune on CCUB using SCoFT+MPC | Stable Diffusion · MS COCO | KID (×10³): 4.396 -> 4.819 | Hurt |
| Fine-tune on CCUB using SCoFT+MPC | Stable Diffusion · Generated images | CLIP Score: 0.813 -> 0.799 | Hurt |
| Add memorization loss during fine-tuning (ablation) | Stable Diffusion fine-tuned on CCUB · CCUB training prompts | DIV: 0.302 -> 0.356 | Helped |
| Add memorization loss during fine-tuning (ablation) | Stable Diffusion fine-tuned on CCUB · CCUB test prompts | DIV: 0.317 -> 0.379 | Helped |
| Add perceptual loss to memorization loss (ablation) | SCoFT+M · User survey, five cultures | Average rank: Least Offensive: 2.59 -> 2.30 | Helped |
| Add self-contrastive loss to memorization and perceptual losses (ablation) | SCoFT+MP · User survey, five cultures | Average rank: Least Offensive: 2.30 -> 1.78 | Helped |
| Add perceptual and self-contrastive losses to memorization loss (ablation) | SCoFT+M · User survey, five cultures | Average rank: Least Offensive: 2.59 -> 1.78 | Helped |
Surprising findings
- Original Stable Diffusion scores higher on CLIP score and MS COCO KID, while human raters strongly prefer SCoFT, suggesting CLIP score is inadequate for cultural competence.
- Recording gradients on the first denoising step worked best, contrary to concurrent work that recorded the last step.
Limitations
- Focused on accurate cultural depiction; generating diverse images for generic prompts is not fully addressed.
- CCUB collected by experienced residents without rigorous verification of dataset quality.
- Automatic metrics (CLIP score, KID on COCO) disagree with human judgments.
Experimental setup
Datasets: CCUB LAION MS COCO
Benchmarks: CCUB MS COCO
Metrics: KID CLIP Score user survey average rank CLIP-I DINO DreamSim DIV
Baselines: Stable Diffusion SCoFT+M SCoFT+MP
Builds on: Stable Diffusion LoRA ControlNet DreamSim DiffusionCLIP
Closest papers by meaning
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.85)
- Fair Text-to-Image Diffusion via Fair Mapping (AAAI 2025 · similarity 0.85)
- No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- Instruct Where the Model Fails: Generative Data Augmentation via Guided Self-contrastive Fine-tuning (AAAI 2025 · similarity 0.85)
- Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.84)
- Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.