Research map / Image generation / Autoregressive and latent generation
PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PatchDPO, an extra training stage for finetuning-free personalized image generation that scores patch-level quality via self-supervised-tuned ViT patch matching and applies patch-weighted diffusion losses to IP-Adapter-Plus on SDXL.
Problem
Finetuning-free personalized image generation often yields images inconsistent with reference images in local patches, and standard single-stage reconstruction training and whole-image DPO do not address these local mismatches.
Approach
Estimate per-patch quality by max cosine similarity between patch features of generated and reference images (from a self-supervised-finetuned ImageNet ViT's shallow layer), then train with patch-weighted diffusion losses that reward good patches in the generated image and reweight the reference-image reconstruction toward its low-quality patches.
Key results
- On DreamBench single-object setting, PatchDPO reaches Avg. 0.619 (DINO 0.727, CLIP-I 0.838), above the best listed baseline IP-Adapter-Plus at 0.600.
- Under the Kosmos-G single-image setting, PatchDPO reaches Avg. 0.666 versus OmniGen at 0.650.
- Self-supervised finetuning raises patch matching Spatch from 68.4% to 83.7% on HPatches.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace natural training images with the constructed dataset (ablation) | IP-Adapter-Plus trained with Lmse · DreamBench | DINO: 0.658 -> 0.708 | Helped |
| replace Lmse training with traditional DPO (ablation) | IP-Adapter-Plus with Dours · DreamBench | DINO: 0.708 -> 0.676 | Hurt |
| add PatchDPO training (ablation) | IP-Adapter-Plus · DreamBench | DINO: 0.692 -> 0.727 | Helped |
| add self-supervised training of the vision model for patch features (ablation) | PatchDPO on IP-Adapter-Plus · DreamBench | DINO: 0.719 -> 0.727 | Helped |
| self-supervised train ViT-Base for patch feature extraction (ablation) | ViT-Base · HPatches | Spatch: 68.4 -> 83.7 | Helped |
| use layer 7 rather than layer 12 for ViT-Base patch features (ablation) | ViT-Base after self-supervised training · HPatches | Spatch: 75.8 -> 83.7 | Helped |
| apply PatchDPO to IP-Adapter-Plus | IP-Adapter-Plus · DreamBench | DINO: 0.608 -> 0.727 | Helped |
| apply PatchDPO to IP-Adapter-Plus | IP-Adapter-Plus · DreamBench | CLIP-I: 0.809 -> 0.838 | Helped |
| apply PatchDPO to IP-Adapter-Plus | IP-Adapter-Plus · DreamBench | CLIP-T: 0.281 -> 0.292; MS-Diffusion: 0.321 | Mixed |
| apply PatchDPO to IP-Adapter-Plus | IP-Adapter-Plus · DreamBench (Kosmos-G evaluation setting) | Avg.: 0.650 -> 0.666 | Helped |
Surprising findings
- Traditional Diffusion-DPO on the same data fails to improve over the original model (Avg-level DINO 0.676 vs original 0.692).
Limitations
- Patch quality is estimated only for the object region, with background excluded.
- Training data is synthetic, generated by Stable Diffusion with cleaner backgrounds, and natural images were found to hurt training.
Experimental setup
Datasets: DreamBench Concept101 MultiDreamBench HPatches SA-1B ImageNet
Benchmarks: DreamBench Concept101 MultiDreamBench HPatches
Metrics: DINO CLIP-I CLIP-T Avg. Spatch
Baselines: DreamBooth Textual Inversion Custom Diffusion ELITE IP-Adapter IP-Adapter-Plus SSR-Encoder BLIP-Diffusion MS-Diffusion Subject-Diffusion JeDi Kosmos-G
Builds on: Diffusion-DPO DPO IP-Adapter-Plus ProtoPNet SFD2
Closest papers by meaning
- Patched Denoising Diffusion Models For High-Resolution Image Synthesis (ICLR 2024 · similarity 0.85)
- Boost Your Human Image Generation Model via Direct Preference Optimization (CVPR 2025 · similarity 0.85)
- DrPO: Drifting Preference Optimization for One-Step Generative Models (NeurIPS 2026 · similarity 0.84)
- Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation (AAAI 2025 · similarity 0.84)
- CPO: Condition Preference Optimization for Controllable Image Generation (NeurIPS 2025 · similarity 0.84)
- TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.