Research map / Image generation / Editing and style transfer
Click2Mask: Local Editing with Dynamic Mask Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Click2Mask performs local image addition from a single clicked point by dynamically evolving a mask during Blended Latent Diffusion, guided by an Alpha-CLIP semantic loss.
Problem
Local image editing for adding new content usually requires a precise mask or a detailed textual location description, which is cumbersome and error-prone.
Approach
Start from a large Gaussian potential around a user click and iteratively contract a mask during Blended Latent Diffusion, elevating potential in regions with high Alpha-CLIP loss gradients so the mask adapts to the generated object's shape, requiring no fine-tuning or precise mask.
Key results
- Human study (149 participants): Click2Mask preferred in 57.14% of items vs Emu Edit (42.86%), and 83.70% vs MagicBrush (16.30%).
- Automatic metrics: Click2Mask CLIP_direct 0.204 vs Emu Edit 0.150 and MagicBrush 0.095; L1 0.027 vs 0.046 and 0.049.
- Runtime about 70 seconds per edit on a Nvidia T4 Medium GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Click2Mask instead of Emu Edit | Local image addition · Emu Edit addition subset | CLIP_direct: 0.150 -> 0.204 | Helped |
| Use Click2Mask instead of MagicBrush | Local image addition · Emu Edit addition subset | CLIP_direct: 0.095 -> 0.204 | Helped |
| Use Click2Mask instead of Emu Edit | Local image addition · Emu Edit addition subset | CLIP_out: 0.331 -> 0.334 | Helped |
| Use Click2Mask instead of MagicBrush | Local image addition · Emu Edit addition subset | CLIP_out: 0.324 -> 0.334 | Helped |
| Use Click2Mask instead of Emu Edit | Local image addition · Emu Edit addition subset | alphaCLIP_edit: 0.186 -> 0.195 | Helped |
| Use Click2Mask instead of MagicBrush | Local image addition · Emu Edit addition subset | alphaCLIP_edit: 0.166 -> 0.195 | Helped |
| Use Click2Mask instead of Emu Edit | Local image addition · Emu Edit addition subset | L1: 0.046 -> 0.027 | Helped |
| Use Click2Mask instead of MagicBrush | Local image addition · Emu Edit addition subset | L1: 0.049 -> 0.027 | Helped |
| Use Click2Mask instead of Emu Edit | Local image addition · Human evaluation vs Emu Edit | % Majority: 42.86% -> 57.14% | Helped |
| Use Click2Mask instead of MagicBrush | Local image addition · Human evaluation vs MagicBrush | % Majority: 16.30% -> 83.70% | Helped |
| Start with a sufficiently large, enlarged mask (ablation) | Click2Mask · Local image addition; white dog guided to prompt “Huge bone” | edit capture (qualitative) | Helped |
| Evolve the mask rather than keep the initial mask fixed (ablation) | Click2Mask · Local image addition | edit quality (qualitative) | Helped |
Surprising findings
- Final background preservation via VAE decoder fine-tuning from BLD was found unnecessary; Gaussian feathering sufficed.
Limitations
- Struggles to converge to small, fine-detailed masks such as a dog collar.
- BLD may fail to add an object when a similar object is nearby in the unmasked region because text guidance is not spatially driven.
- Evaluation relies on a 100-sample subset of the Emu Edit addition benchmark and a proxy edited-region metric.
Experimental setup
Datasets: Emu Edit test benchmark (addition subset, 100 sampled)
Benchmarks: Emu Edit benchmark
Metrics: user preference majority vote CLIP_direct CLIP_out Edited Alpha-CLIP L1
Baselines: Emu Edit MagicBrush InstructPix2Pix
Builds on: Blended Latent Diffusion Blended Diffusion Alpha-CLIP Stable Diffusion
Closest papers by meaning
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.89)
- SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control (CVPR 2024 · similarity 0.89)
- Improving Editability in Image Generation with Layer-wise Memory (CVPR 2025 · similarity 0.88)
- FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation (NeurIPS 2025 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.87)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.