Research map / Image generation / Text-to-image diffusion
InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free customized concept swapping method (InstantSwap) that uses an automatically derived bounding box for background gradient masking, semantic-enhanced cross-attention for foreground, and step-skipping gradient updates for speed, plus a new ConSwapBench benchmark.
Problem
Customized concept swapping in images struggles with foreground/background inconsistency when source and target shapes differ greatly, and existing methods are slow due to training or redundant forward passes at every timestep.
Approach
Derive a bbox from enhanced cross-/self-attention maps, mask score-distillation gradients outside the bbox to protect background, inject semantic cross-attention inside the bbox for foreground, and compute gradients only at periodic anchor timesteps (SSGU) to reuse them in between.
Key results
- InstantSwap achieves the best scores on all seven metrics on ConSwapBench, e.g., CLIP-I 75.00 and PSNR 27.39 versus 73.70 and 24.63 for the next-best methods.
- Inference time drops to 19.83 s versus 140.26 s for CDS and 140.34 s for PhotoSwap, with SSGU factor lambda=5.
- Applying SSGU to SDS and DDS cuts their inference time from 40.37 s to 8.62 s and 66.89 s to 13.90 s, respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BGM to mask background gradients (ablation) | InstantSwap · ConSwapBench | PSNR: 18.03 -> 27.39 | Helped |
| add BGM to mask background gradients (ablation) | InstantSwap · ConSwapBench | LPIPS: 249.24 -> 47.68 | Helped |
| add BGM to mask background gradients (ablation) | InstantSwap · ConSwapBench | MSE: 184.19 -> 27.87 | Helped |
| add BGM to mask background gradients (ablation) | InstantSwap · ConSwapBench | CLIP-T: 23.08 -> 25.74 | Helped |
| add source-branch SECR (ablation) | InstantSwap · ConSwapBench | CLIP-I: 73.70 -> 75.00 | Helped |
| add source-branch SECR (ablation) | InstantSwap · ConSwapBench | CLIP-T: 25.47 -> 25.74 | Helped |
| add target-branch SECR (ablation) | InstantSwap · ConSwapBench | CLIP-I: 73.40 -> 75.00 | Helped |
| add target-branch SECR (ablation) | InstantSwap · ConSwapBench | CLIP-T: 25.52 -> 25.74 | Helped |
| add SECR to both source and target branches (ablation) | InstantSwap · ConSwapBench | CLIP-I: 72.42 -> 75.00 | Helped |
| add SECR to both source and target branches (ablation) | InstantSwap · ConSwapBench | CLIP-T: 25.21 -> 25.74 | Helped |
| use automatically generated bbox instead of GT bbox for generation (ablation) | InstantSwap · ConSwapBench | CLIP-I: 75.79 -> 77.72 | Helped |
| increase SSGU factor from λ=1 (no SSGU) to λ=5 (ablation) | InstantSwap · ConSwapBench | swapping consistency | Mixed |
Surprising findings
- Ground-truth bboxes prevent full swapping of the source concept, whereas the automatically generated, slightly larger bboxes cover the concept fully and perform better.
Limitations
- Excessively large SSGU factor lambda causes artifacts and loss of foreground consistency.
- Relies on DreamBooth-customized checkpoints for target concepts, requiring per-concept fine-tuning.
- Benchmark is small (62 ConceptBench images, 160 SwapBench images) and SwapAnything could not be compared since it is not public.
Experimental setup
Datasets: SwapBench ConceptBench
Benchmarks: ConSwapBench
Metrics: CLIP-I CLIP-T PSNR LPIPS MSE SSIM inference time
Baselines: SDS DDS CDS PhotoSwap PnPInv+P2P P2P
Builds on: SDS DDS PhotoSwap DreamBooth Prompt-to-Prompt
Closest papers by meaning
- Attention Calibration for Disentangled Text-to-Image Personalization (CVPR 2024 · similarity 0.86)
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.85)
- Mind the Gap Between Prototypes and Images in Cross-domain Finetuning (NeurIPS 2024 · similarity 0.84)
- Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis (AAAI 2025 · similarity 0.84)
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.84)
- Concept Lancet: Image Editing with Compositional Representation Transplant (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.