Research map / Image generation / Text-to-image diffusion
Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Refine-by-Align, a diffusion-based two-stage (alignment + refinement) model that uses cross-attention correspondence to refine localized artifacts in generated images using a reference image, without test-time tuning.
Problem
Personalized or reference-guided generated images often contain localized artifacts (e.g., incorrect logos, text, fine textures) that reduce identity fidelity; prior artifact refinement methods lack reference-guided control and handle arbitrary-shaped regions poorly.
Approach
Uses diffusion cross-attention maps between the artifact-region query and DINOv2 reference tokens to locate the corresponding reference region (alignment), then reuses the same network with that region as reference to inpaint the artifact (refinement); trained self-supervised on Pixabay and MVObj.
Key results
- On GenArtifactBench, the method reaches DINO-I 75.31 and CLIP-I 86.63, the best among compared methods (ObjectStitch DINO-I 72.62, AnyDoor 71.34).
- User study: the method wins identity preference against all six baselines, e.g., 74.17% vs Cross-Image Attention and 57.5% vs AnyDoor.
- Ablation: replacing DINOv2 with CLIP drops DINO score from 75.31 to 70.77, and single-stage training drops it to 71.76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Refine-by-Align instead of PAL | GenArtifactBench artifact refinement · GenArtifactBench | CLIP-T: 25.8906 -> 25.4063 | Hurt |
| use Refine-by-Align instead of MimicBrush | GenArtifactBench artifact refinement · GenArtifactBench | CLIP-I: 85.0625 -> 86.6250 | Helped |
| use Refine-by-Align instead of ObjectStitch | GenArtifactBench artifact refinement · GenArtifactBench | DINO-I: 72.6152 -> 75.3135 | Helped |
| use Refine-by-Align instead of Paint-by-Example | GenArtifactBench artifact refinement · GenArtifactBench | Identity preference win rate: 70.83% vs 29.17% | Helped |
| use Refine-by-Align instead of Paint-by-Example | GenArtifactBench artifact refinement · GenArtifactBench | Realism preference win rate: 71.67% vs 28.33% | Helped |
| add refinement-mode training to alignment-mode training (ablation) | Refine-by-Align · GenArtifactBench | CLIP-T: 24.4375 -> 25.4063 | Helped |
| add refinement-mode training to alignment-mode training (ablation) | Refine-by-Align · GenArtifactBench | CLIP-I: 84.5000 -> 86.6250 | Helped |
| add refinement-mode training to alignment-mode training (ablation) | Refine-by-Align · GenArtifactBench | DINO: 71.7609 -> 75.3135 | Helped |
| use DINOv2 instead of CLIP as the visual encoder (ablation) | Refine-by-Align · GenArtifactBench | CLIP-T: 24.3750 -> 25.4063 | Helped |
| use DINOv2 instead of CLIP as the visual encoder (ablation) | Refine-by-Align · GenArtifactBench | CLIP-I: 84.3750 -> 86.6250 | Helped |
| use DINOv2 instead of CLIP as the visual encoder (ablation) | Refine-by-Align · GenArtifactBench | DINO: 70.7714 -> 75.3135 | Helped |
Surprising findings
- PAL achieves the highest CLIP-T score (25.89) despite having the lowest DINO-I (53.81), showing CLIP-T fails to measure detail preservation.
- Test-time timestep is set to t=0 for speed, although grid search found t=24 optimal.
Limitations
- Performance degrades when the object in the generated image differs substantially from the reference.
- Alignment accuracy is limited by the 16x16 DINOv2 patch token resolution.
- Requires user-provided artifact masks; no automatic artifact detection.
Experimental setup
Datasets: Pixabay MVObj DreamBooth PAL artifacts dataset
Benchmarks: GenArtifactBench
Metrics: CLIP-T CLIP-I DINO-I user preference win rate mIoU
Baselines: Paint-by-Example ObjectStitch AnyDoor PAL Cross-Image Attention MimicBrush DIFT DHF
Builds on: AnyDoor IMPRINT Stable Diffusion DINOv2 zoom-in inpainting
Closest papers by meaning
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.87)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.86)
- One-Shot Reference-based Structure-Aware Image to Sketch Synthesis (AAAI 2025 · similarity 0.85)
- TFCustom: Customized Image Generation with Time-Aware Frequency Feature Guidance (CVPR 2025 · similarity 0.85)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.85)
- MagiCapture: High-Resolution Multi-Concept Portrait Customization (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.