Research map / Image generation / Editing and style transfer
Improving Editability in Image Generation with Layer-wise Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free iterative, mask-ordered image editing with PixArt-α using layer-wise memory, background consistency guidance, and multi-query disentangled cross-attention, evaluated on a new Multi-Edit Benchmark.
Problem
Sequential image editing with multiple object additions struggles to preserve earlier edits and to naturally integrate new objects into existing content, especially under rough mask inputs and mask-order occlusion.
Approach
Stores per-step latents, masks and prompt embeddings in a layer-wise memory; blends preserved background latents (BCG) and applies multi-query disentangled cross-attention over previous mask orders so new objects adapt to existing content, without any additional training.
Key results
- On Multi-Edit Bench at 1024x1024, the method reaches BLEU-2/3/4 of 64.99/47.69/36.59 and METEOR 0.1513, versus 55.30/40.38/29.58 for Blended Latent Diffusion and 63.29/47.63/36.28 for HD-Painter.
- Ablation: adding BCG raises BLEU-2 from 56.29 to 60.74 over the PixArt-α inpainting baseline; full MQD with memory reaches 64.99.
- BCG reduces single-step editing time from 4.12 s to 3.90 s (about 10%) and the object deletion procedure is reported as 60% faster with tau=8 of 20 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Background Consistency Guidance (ablation) | PixArt-α inpainting · Multi-Edit Bench | BLEU-2: 56.29 -> 60.74 | Helped |
| add Background Consistency Guidance (ablation) | PixArt-α inpainting · Multi-Edit Bench | METEOR: 0.1586 -> 0.1585 | No clear effect |
| add Background Consistency Guidance (ablation) | PixArt-α inpainting · Multi-Edit Bench | CLIPc: 64.05 -> 64.10 | Helped |
| add Query Disentanglement without memory (ablation) | PixArt-α inpainting · Multi-Edit Bench | BLEU-2: 56.29 -> 62.68 | Helped |
| add Query Disentanglement without memory (ablation) | PixArt-α inpainting · Multi-Edit Bench | CLIPc: 64.05 -> 63.99 | No clear effect |
| extend QD to Multi-Query Disentanglement with layer-wise memory (ablation) | PixArt-α inpainting with QD · Multi-Edit Bench | BLEU-2: 62.68 -> 64.99 | Helped |
| extend QD to Multi-Query Disentanglement with layer-wise memory (ablation) | PixArt-α inpainting with QD · Multi-Edit Bench | METEOR: 0.1530 -> 0.1513 | Hurt |
| use BCG instead of latent blending | PixArt-α inpaint pipeline · Single-step editing | Time (sec): 4.1218 ±0.0521 -> 3.8992±0.0142 | Helped |
| use the proposed iterative editing method instead of HD-Painter | PixArt-α · Multi-Edit Bench | BLEU-4: 36.28 -> 36.59 | Helped |
Surprising findings
- SD3-ControlNet-Inpaint performs far worse on the semantic metrics (BLEU-2 29.90), which the authors attribute to ControlNet inpainting being suboptimal for multi-step editing.
- NoiseCollage adapted with ordering improves visual alignment but gains little on semantic metrics.
Limitations
- Editing multiple objects takes longer and scales with the number of edits.
- Layer-wise memory adds memory cost.
- Evaluation relies on a self-constructed benchmark with LLaVA-based captioning metrics.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: Multi-Edit Bench
Metrics: BLEU-2 BLEU-3 BLEU-4 METEOR CLIP score (cropped)
Baselines: LayoutGuidance NoiseCollage LooseControl Blended Latent Diffusion HD-Painter SD3-ControlNet-Inpaint PixArt-α
Builds on: PixArt-α Blended Latent Diffusion NoiseCollage
Closest papers by meaning
- CAMILA: Context-Aware Masking for Image Editing with Language Alignment (NeurIPS 2025 · similarity 0.89)
- Click2Mask: Local Editing with Dynamic Mask Generation (AAAI 2025 · similarity 0.88)
- PairEdit: Learning Semantic Variations for Exemplar-based Image Editing (NeurIPS 2025 · similarity 0.88)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.88)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.87)
- LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.