Research map / Image generation / Text-to-image diffusion
Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Focus-N-Fix, a region-aware reward fine-tuning method for text-to-image diffusion models that constrains changes to outside problematic regions (from heatmaps or saliency maps) to fix artifacts, safety issues and misalignment with minimal collateral change.
Problem
Reward-based fine-tuning of text-to-image models improves targeted quality aspects (safety, artifacts, alignment) but often changes images globally, degrades other aspects, causes catastrophic forgetting, and leads to reward hacking.
Approach
Build on DRaFT differentiable reward fine-tuning with a regional constraint that penalizes changes outside a mask of problematic regions predicted from the pre-trained model's image; masks are used only during fine-tuning, so inference is a standard forward pass.
Key results
- On over-sexualization, Focus-N-Fix achieved the highest safety human preference score (0.479) vs DRaFT (0.361) and SLD (0.439) on SD v1.4.
- For artifacts, Focus-N-Fix achieved an artifact score of 0.294 versus 0.207 for DRaFT, with T2I alignment 0.100 vs 0.012.
- On SDXL, Focus-N-Fix preserved image structure better than DRaFT (PSNR 20.92 vs 12.40, LPIPS 0.33 vs 0.66) while keeping VNLI change positive (0.013 vs -0.031).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Focus-N-Fix regional constraints for safety fine-tuning | Stable Diffusion v1.4 · Over-sexualization human evaluation | Safety score: 0.479 | Helped |
| Use Focus-N-Fix regional constraints for safety fine-tuning | Stable Diffusion v1.4 · Over-sexualization human evaluation | T2I alignment score: 0.004 | No clear effect |
| Use Focus-N-Fix regional constraints for safety fine-tuning | Stable Diffusion v1.4 · Over-sexualization human evaluation | Artifact score: 0.042 | No clear effect |
| Use Focus-N-Fix regional constraints for artifact fine-tuning | Stable Diffusion v1.4 · Artifact human evaluation | Artifact score: 0.294 | Helped |
| Use Focus-N-Fix regional constraints for artifact fine-tuning | Stable Diffusion v1.4 · Artifact human evaluation | T2I alignment score: 0.100 | Helped |
| Add regional constraints to DRaFT (ablation) | Stable Diffusion v1.4 safety fine-tuning · Over-sexualization human evaluation | Safety score: 0.361 -> 0.479 | Helped |
| Add regional constraints to DRaFT (ablation) | Stable Diffusion v1.4 safety fine-tuning · Over-sexualization human evaluation | T2I alignment score: -0.146 -> 0.004 | Helped |
| Add regional constraints to DRaFT (ablation) | Stable Diffusion v1.4 artifact fine-tuning · Artifact human evaluation | Artifact score: 0.207 -> 0.294 | Helped |
| Use Focus-N-Fix instead of DraFT | SDXL safety fine-tuning · SDXL safety evaluation | ΔVNLI score: -0.031 -> 0.013 | Helped |
| Use Focus-N-Fix instead of DraFT | SDXL safety fine-tuning · SDXL safety evaluation | PSNR: 12.40 -> 20.92 | Helped |
| Use Focus-N-Fix instead of DraFT | SDXL safety fine-tuning · SDXL safety evaluation | ΔSafety Reward: 0.33 -> 0.21 | Hurt |
Surprising findings
- DRaFT fine-tuned for artifacts may reward-hack by changing image structure (e.g., drill shape, adding hands) instead of fixing the artifact region.
- SLD achieved the lowest artifact but poor T2I alignment, showing that reducing one issue can degrade another.
Limitations
- Predicting misalignment regions (e.g., missing objects) is hard, so applicability to misalignment is limited.
- Non-localizable aspects like aesthetics or style cannot use the region constraint directly.
- Most experiments use SD v1.4 and DRaFT only.
Experimental setup
Datasets: HPDv2 PartiPrompts WeLI images (PaLI captions)
Benchmarks: HPDv2 evaluation prompts PartiPrompts
Metrics: human preference MOS (safety, artifact, T2I alignment) VNLI alignment score safety reward PSNR SSIM LPIPS
Baselines: DRaFT Reward Guidance Reward Guidance + RC Safe Latent Diffusion
Builds on: DRaFT Reward Guidance LoRA Safe Latent Diffusion
Closest papers by meaning
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.91)
- Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models (ICLR 2024 · similarity 0.90)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.89)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.89)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.88)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.