Research map / Image generation / Editing and style transfer
RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that concatenated-attention consistency methods implicitly apply a rank-1 guidance, and proposes RefDrop, a training-free plug-and-play reference feature guidance with a scalar coefficient for controllable consistency and diversity in image and video generation.
Problem
Concatenated-attention methods for consistent image generation are popular but their underlying mechanism is unclear, and existing controllable consistency methods either need encoder/fine-tuning or cannot easily control reference strength or suppress features.
Approach
Reformulates concatenated attention as a linear interpolation between self-attention and cross-attention to reference features with a constant rank-1 coefficient, then replaces it with a simple scalar-weighted reference feature guidance (RFG) that can be negative (suppression) or multi-reference, applied to every self-attention layer.
Key results
- On personalized video generation, RefDrop improves VQA_A (94.27 vs 93.25), VQA_T (89.91 vs 86.20), face consistency (99.46 vs 99.20) and lowers warping error (0.0058 vs 0.0077) relative to SVD, at the cost of flow score (2.62 vs 5.80).
- Human evaluation with 44 users (6,160 ratings) shows participants preferred RefDrop for consistent image generation, with IP-Adapter ranking slightly behind, and clearly preferred RefDrop for personalized video quality.
- Quantitatively, RefDrop is reported to lie on the Pareto front of consistency vs. text alignment and diversity vs. text alignment against SDXL and IP-Adapter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | VQA_A: 93.25 -> 94.27 | Helped |
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | VQA_T: 86.20 -> 89.91 | Helped |
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | CLIP score: 20.76 -> 20.84 | Helped |
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | CLIP Temp: 99.83 -> 99.91 | Helped |
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | Face consistency: 99.20 -> 99.46 | Helped |
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | Warping error: 0.0077 -> 0.0058 | Helped |
| Add reference feature guidance to SVD | SVD-img2vid-xt-1-1 · Personalized video generation | Flow score: 5.80 -> 2.62 | Hurt |
| Use negative reference strength c=-0.3 to discourage similarity to the reference | RefDrop · Diverse image generation | Subject diversity | Helped |
| Use negative reference strength c=-0.3 to discourage similarity to the reference | RefDrop · Diverse image generation | Text alignment | Mixed |
| Use c=0.2 reference strength for subsequent video frames | RefDrop with SVD · Video generation | Temporal consistency | Helped |
| Increase reference strength beyond the recommended video setting (ablation) | RefDrop with SVD · Video generation | Motion quality | Hurt |
| Lower reference strength for diverse image generation (ablation) | RefDrop · Diverse image generation | Visual quality | Hurt |
Surprising findings
- Naively breaking the linear interpolation (adding the cross-attention term without weighting self-attention) disrupts image generation.
- Negative reference coefficients allow suppressing reference features to encourage diversity, which IP-Adapter achieves only with a larger text-alignment cost.
Limitations
- Sometimes fails to exactly replicate specific object appearance, hallucinating instead (e.g., lotion bottles).
- Tends to produce generated individuals with similar poses; mitigation requires extra techniques not fully integrated.
- Video consistency coefficient is sensitive: larger values make the video totally static.
Experimental setup
Benchmarks: EvalCrafter metrics on 220 personalized videos human evaluation (44 users)
Metrics: CLIP score DreamSim consistency LPIPS diversity VQA_A VQA_T CLIP-Temp face consistency warping error flow score human preference
Baselines: SDXL IP-Adapter BLIP-Diffusion ConsiStory Stable Video Diffusion Cross-Frame Attention Concatenated Attention Temporal low-pass frequency filter
Builds on: concatenated attention cross-frame attention classifier-free guidance
Closest papers by meaning
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.87)
- Re-Attentional Controllable Video Diffusion Editing (AAAI 2025 · similarity 0.86)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.86)
- CCM: Real-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models (ICML 2024 · similarity 0.86)
- Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model (ICLR 2025 · similarity 0.86)
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.