Research map / Detection and segmentation / Open-set and few-shot detection
AnyDoor: Zero-shot Object-level Image Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnyDoor, a zero-shot diffusion-based object teleportation model that places a target object into a scene at a user-specified location and shape, using DINOv2 identity tokens plus frequency-aware high-pass detail maps, trained on video and image data.
Problem
Zero-shot object-level image customization: placing an unseen target object into a specified location of a scene image while preserving its identity and fine details, without per-object finetuning.
Approach
Represent the object with complementary identity features (DINOv2 global and patch tokens injected via cross-attention) and detail maps (high-frequency Sobel maps stitched into the scene collage fed to a ControlNet-style encoder), and train on video frame pairs of the same object to learn appearance variation, with adaptive timestep sampling for video vs. image data.
Key results
- On the constructed DreamBooth-based benchmark, the full model reaches CLIP score 82.1 and DINO score 67.8, versus 81.0/64.1 for DINOv2 (G+P) without segmentation and 71.7/27.7 for VGG.
- In a 15-annotator user study, AnyDoor scored 3.06 fidelity versus 2.10 for Paint-by-Example and 2.11 for Graphit, and 3.04 quality versus 2.71 and 2.65.
- Ablation: replacing the high-frequency map with an all-zero map drops DINO score from 67.8 to 63.2 on the benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DINOv2 instead of VGG/CLIP as ID extractor backbone (ablation) | ID extractor · DreamBooth benchmark | DINO Score: DINO 27.7 (VGG) / 31.5 (CLIP G+P) -> 64.1 (DINOv2 G+P) | Helped |
| add patch tokens to global token in ID extractor (DINOv2 G+P vs G only) (ablation) | ID extractor · DreamBooth benchmark | DINO Score: 35.4 -> 64.1 | Helped |
| filter background with frozen segmentation model before ID extraction (ablation) | ID extractor · DreamBooth benchmark | DINO Score: 64.1 -> 67.8 (CLIP 81.0 -> 82.1) | Helped |
| add high-frequency map collage to detail extractor (vs no collage) (ablation) | detail extractor · DreamBooth benchmark | DINO Score: 63.2 -> 67.8 (CLIP 80.4 -> 82.1) | Helped |
| use original-image collage instead of high-frequency map in detail extractor (ablation) | detail extractor · DreamBooth benchmark | DINO Score: DINO 68.8 vs 67.8 (higher); paper notes it yields copy-paste-like low diversity | Mixed |
| use noised or patch-shuffled collage instead of high-frequency map (ablation) | detail extractor · DreamBooth benchmark | DINO Score: Noise 68.1 / Shuffle 66.9 vs HF-Map 67.8 (CLIP 81.6 and 82.0 vs 82.1) | Hurt |
| replace adaptive timestep sampling with even timestep sampling (ablation) | training strategy · Qualitative ablation (Fig. 7) | image quality and fidelity (qualitative) | Hurt |
| user study: AnyDoor vs Paint-by-Example and Graphit | AnyDoor · User study (15 annotators) | Fidelity (1-4): 3.06 vs 2.10 (Paint-by-Example), 2.11 (Graphit) | Helped |
Surprising findings
- Using the original object image as the collage gives the highest fidelity but produces near copy-paste outputs with low diversity, motivating the high-frequency map trade-off.
Limitations
- Struggles with fine details such as small characters or logos.
- Relies on segmentation for background removal and resolution limits of 512x512 with VAE decoder quality.
Experimental setup
Datasets: YouTubeVOS YouTubeVIS UVO MOSE VIPSeg BURST MVImgNet VitonHD FashionTryon MSRA-10K DUT HFlickr
Benchmarks: DreamBooth-based benchmark (30 concepts x 80 COCO-Val scenes, 2,400 images) VitonHD-test (qualitative)
Metrics: CLIP-Score DINO-Score user study fidelity/quality/diversity
Baselines: Paint-by-Example Graphit IP-Adapter Stable Diffusion inpainting DreamBooth Custom Diffusion Cones
Builds on: Stable Diffusion ControlNet DINOv2 Paint-by-Example ObjectStitch
Closest papers by meaning
- HomeDiffusion: Zero-Shot Object Customization with Multi-View Representation Learning for Indoor Scenes (AAAI 2025 · similarity 0.87)
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image (CVPR 2024 · similarity 0.86)
- CustAny: Customizing Anything from A Single Example (CVPR 2025 · similarity 0.84)
- ZeroShape: Regression-based Zero-shot Shape Reconstruction (CVPR 2024 · similarity 0.84)
- Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering (NeurIPS 2024 · similarity 0.84)
- A Diffusion-Based Framework for Occluded Object Movement (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.