Research map / Image generation / Editing and style transfer
Zero-shot Image Editing with Reference Imitation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes imitative editing, where a masked region of a source image is filled by automatically finding and imitating the corresponding part of a reference image, trained self-supervised on video frame pairs with a dual U-Net diffusion model.
Problem
Local image editing typically relies on text prompts (inpainting) or carefully cropped reference objects (composition), which fail to capture fine-grained local parts or patterns; the paper targets editing a masked region using an unsegmented in-the-wild reference image without paired masks.
Approach
Train dual diffusion U-Nets (imitative and reference) self-supervised on two frames from the same video, masking one frame and recovering it from the other; reference U-Net keys/values are concatenated into the imitative U-Net attention, with SIFT-guided masking, strong augmentation, and optional depth conditioning for texture transfer.
Key results
- On the inner-ID part composition track, MimicBrush reaches SSIM 0.70, PSNR 17.54, LPIPS 0.28, versus best AnyDoor w/ Mask at 0.44, 14.09, 0.50.
- In the user study, MimicBrush was picked best for fidelity 55.8% of the time versus 30.6% for the next best (AnyDoor w/ Mask).
- On inter-ID part composition, AnyDoor w/ Mask scores higher DINO-I (61.30 vs 56.48) and CLIP-I (86.08 vs 84.30), while MimicBrush has higher CLIP-T (30.08 vs 29.39).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace reference U-Net with CLIP Encoder (ablation) | MimicBrush · Part composition | SSIM: 0.70 -> 0.66 | Hurt |
| replace reference U-Net with DINOv2 Encoder (ablation) | MimicBrush · Part composition | SSIM: 0.70 -> 0.67 | Hurt |
| replace reference U-Net with CLIP Encoder (ablation) | MimicBrush · Texture transfer | DINO-I: 49.83 -> 37.86 | Hurt |
| replace reference U-Net with DINOv2 Encoder (ablation) | MimicBrush · Texture transfer | DINO-I: 49.83 -> 46.34 | Hurt |
| train using image data only (ablation) | MimicBrush · Part composition | DINO-I: 56.48 -> 39.68 | Hurt |
| train using image data only (ablation) | MimicBrush · Texture transfer | DINO-I: 49.83 -> 41.30 | Hurt |
| use weak augmentation (ablation) | MimicBrush · Part composition | DINO-I: 56.48 -> 50.55 | Hurt |
| use weak augmentation (ablation) | MimicBrush · Texture transfer | DINO-I: 49.83 -> 50.92 | Helped |
| use single-box masking (ablation) | MimicBrush · Part composition | SSIM: 0.70 -> 0.66 | Hurt |
| use Mask Grid 0.25 (ablation) | MimicBrush · Part composition | SSIM: 0.70 -> 0.68 | Hurt |
| use Mask Grid 0.50 (ablation) | MimicBrush · Part composition | SSIM: 0.70 -> 0.68 | Hurt |
| use Mask Grid 0.75 (ablation) | MimicBrush · Part composition | SSIM: 0.70 -> 0.67 | Hurt |
Surprising findings
- On inter-ID part composition, AnyDoor with reference masks beats MimicBrush on DINO-I and CLIP-I, the paper's own comparison acknowledging the gap.
- CLIP and DINOv2 encoders as reference feature extractors achieve competitive results, lagging the U-Net mainly in fine detail preservation.
Limitations
- Fails to locate the reference region when it is too small or when multiple candidate regions exist in the reference image.
- Inter-ID part composition is harder; AnyDoor with reference masks is competitive on semantic metrics.
- Benchmark is small (120 inter-ID part composition samples, 30 per track) and evaluated partly with manual annotation and a 10-annotator user study.
Experimental setup
Datasets: Pexels videos (100k) SAM dataset
Benchmarks: MimicBrush benchmark (part composition, texture transfer) DreamBooth inner-ID pairs
Metrics: SSIM PSNR LPIPS DINO-I CLIP-I CLIP-T user preference rank
Baselines: Paint-by-Example AnyDoor Firefly
Builds on: Stable Diffusion 1.5 inpainting Depth Anything AnyDoor Paint-by-Example
Closest papers by meaning
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.87)
- VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling (NeurIPS 2026 · similarity 0.87)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.87)
- Referring Image Editing: Object-level Image Editing via Referring Expressions (CVPR 2024 · similarity 0.87)
- EchoShot: Multi-Shot Portrait Video Generation (NeurIPS 2025 · similarity 0.86)
- Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.