Research map / Image generation / Editing and style transfer
EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
EasyRef conditions SDXL on multiple reference images and a text prompt by having Qwen2-VL-2B encode them via learned reference tokens, injected through IP-Adapter-style cross-attention, and introduces the MRBench benchmark.
Problem
Personalized diffusion generation from multiple reference images: tuning-free methods average image embeddings, losing cross-image consistency, while LoRA requires per-group finetuning and fails to generalize zero-shot.
Approach
Use an MLLM (Qwen2-VL-2B) with an instruction that jointly encodes all reference images and the text prompt; append learnable reference tokens at the final LLM layer with bi-directional attention to aggregate them into a compact condition injected into SDXL via adapters, trained with a three-stage progressive scheme.
Key results
- On MRBench held-out split, EasyRef reaches CLIP-I 0.833 and DINO-I 0.614 versus IP-Adapter-SDXL's 0.795 and 0.579, while LoRA fails zero-shot on that split.
- On MRBench held-in split, EasyRef (CLIP-I 0.843, CLIP-T 0.726, DINO-I 0.672) beats LoRA (0.831, 0.715, 0.654).
- On COCO single-image references, EasyRef achieves DINO-I 0.873 versus IP-Adapter-SDXL's 0.650.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EasyRef instead of IP-Adapter-SDXL for single-reference generation | SDXL image generation · COCO 2017 validation | DINO-I: 0.650 -> 0.873 | Helped |
| Use EasyRef instead of LoRA for multi-reference generation | MRBench held-in split · MRBench held-in split | CLIP-I: 0.831 -> 0.843 | Helped |
| Use EasyRef instead of IP-Adapter-SDXL with averaged image embeddings | MRBench held-out split · MRBench held-out split | CLIP-T: 0.645 -> 0.709 | Helped |
| Use 64 reference tokens rather than 32 tokens (ablation) | EasyRef · MRBench held-out split | CLIP-I: 0.813 -> 0.833 | Helped |
| Use 64 reference tokens rather than 128 tokens (ablation) | EasyRef · MRBench held-out split | DINO-I: 0.611 -> 0.614 | Helped |
| Insert reference tokens at the final layer rather than position -2 (ablation) | EasyRef · MRBench held-out split | CLIP-I: 0.831 -> 0.833 | Helped |
| Insert reference tokens at the final layer rather than position -3 (ablation) | EasyRef · MRBench held-out split | CLIP-T: 0.702 -> 0.709 | Helped |
| Use MLLM reference-token aggregation rather than averaging reference embeddings (ablation) | EasyRef · MRBench held-out split | DINO-I: 0.584 -> 0.614 | Helped |
| Use MLLM reference-token aggregation rather than concatenating reference embeddings (ablation) | EasyRef · MRBench held-out split | DINO-I: 0.579 -> 0.614 | Helped |
| Include multimodal instructions in the MLLM input (ablation) | EasyRef · Multi-reference image generation | generation performance | Helped |
| Include the text image prompt in the MLLM input (ablation) | EasyRef · Multi-reference image generation | text-image alignment | Helped |
| Add alignment pretraining and single-reference finetuning to the progressive training scheme (ablation) | EasyRef · Identity preservation and complex compositions | visual alignment | Helped |
Surprising findings
- Concatenating reference representations gives no clear gain over averaging on CLIP-I and CLIP-T and is 5x costlier in tokens, while the learned reference-token aggregation performs best with 64 tokens.
Limitations
- Performance degrades when the number of reference images exceeds the training constraint (about 16 images per group).
- Reference images are resized so MLLM input cannot exceed 336x336, limiting fine-grained detail.
- LoRA baseline fails to generalize zero-shot to the held-out split, so comparisons on that split rely on IP-Adapter only.
Experimental setup
Datasets: LAION-5B COCO 2017 validation MRBench
Benchmarks: COCO 2017 validation MRBench
Metrics: CLIP-I CLIP-T DINO-I CLIPScore human preference winrate
Baselines: IP-Adapter-SDXL LoRA Open unCLIP Kandinsky 2.1 Versatile Diffusion SD unCLIP Uni-ControlNet T2I-Adapter ControlNet Shuffle
Builds on: IP-Adapter Stable Diffusion XL LoRA LI-DiT
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- UniFL: Improve Latent Diffusion Model via Unified Feedback Learning (NeurIPS 2024 · similarity 0.88)
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.88)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.87)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.87)
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.