Research map / Image generation / Editing and style transfer
PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIXELS, a training-free inference-time method that progressively blends source and exemplar latents under a non-binary edit map, enabling pixel-level control over exemplar-driven editing with off-the-shelf SDXL.
Problem
Exemplar-based image editing typically requires training specialized architectures, supports only one exemplar and binary masks (uniform change within the edited region), and loses the text-to-image model's original capabilities.
Approach
Build a surgical latent by copy-pasting source and exemplar latents, then run a diffusion denoising loop where each spatial location is released from the source at a timestep set by a non-binary edit map (per-pixel denoising strength), so edits can be graded spatially and any number of exemplars can be added in one pass, with no retraining.
Key results
- PIXELS achieves CLIP-I 91.71 and FID 5.412, versus MimicBrush's 84.95 and 7.915 on 3000 ImageNet pairs.
- In a 26-participant user study, PIXELS gets the best (lowest) average rank for adherence (1.40) and realism (1.40), versus MimicBrush's 2.27 and 2.97.
- Inference memory overhead is under 7MB (0.04%) over SDXL img2img, with about 3.86% inference time overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PIXELS instead of MimicBrush | Exemplar-driven editing on ImageNet validation pairs · ImageNet validation pairs | CLIP-I Score: 84.95 -> 91.71 | Helped |
| Use PIXELS instead of MimicBrush | Exemplar-driven editing on ImageNet validation pairs · ImageNet validation pairs | FID: 7.915 -> 5.412 | Helped |
| Use PIXELS instead of Versatile Diffusion | Exemplar-driven editing in the user study · User study | Adherence ranking: 4.93 -> 1.40 | Helped |
| Use PIXELS instead of Versatile Diffusion | Exemplar-driven editing in the user study · User study | Realism ranking: 4.03 -> 1.40 | Helped |
| Increase maximum denoising strength as exemplar distance from the source grows (ablation) | PIXELS · Exemplar-driven editing with increasingly distant exemplars | Realism | Helped |
| Increase maximum denoising strength as exemplar distance from the source grows (ablation) | PIXELS · Exemplar-driven editing with increasingly distant exemplars | Exemplar fidelity | Hurt |
| Keep maximum denoising strength constant at 0.2 for distant exemplars rather than increasing it (ablation) | PIXELS · Exemplar 1 and exemplar 6 edits | Realism | Hurt |
| Use Log instead of Linear thresholding (ablation) | PIXELS · Exemplar-driven editing | Exemplar adherence | Hurt |
| Use Cubic or Quadratic instead of Linear thresholding (ablation) | PIXELS · Exemplar-driven editing | Exemplar adherence | Helped |
| Use Cubic or Quadratic instead of Linear thresholding (ablation) | PIXELS · Exemplar-driven editing | Transition seamlessness | Hurt |
| Use Sigmoid instead of Linear thresholding (ablation) | PIXELS · Exemplar-driven editing | Edit quality | No clear effect |
| Multiply added noise by the edit map | Spatially adjusted noise baseline · Exemplar-driven editing | Image quality | Hurt |
Surprising findings
- Spatially multiplying the added noise by the edit map tends to make the model converge to random single-color images.
- A naive binary latent-surgery copy-paste produces strong transition artifacts at edit boundaries, which motivates the progressive scheme.
Limitations
- The edit map must be supplied manually (drawn masks, depth or segmentation maps) rather than generated automatically.
- Quantitative comparison binarizes the edit map for baselines, which are not designed for non-binary maps.
- Trade-off between adherence and realism requires choosing denoising strength per edit.
Experimental setup
Datasets: ImageNet validation set
Benchmarks: ImageNet validation (3000 random image pairs)
Metrics: FID CLIP-I user ranking score for adherence and realism
Baselines: Versatile Diffusion Paint-By-Example IP-Adapter MimicBrush
Builds on: Latent Diffusion Models SDEdit SDXL RePaint
Closest papers by meaning
- PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement (NeurIPS 2025 · similarity 0.89)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.89)
- PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation (AAAI 2025 · similarity 0.89)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.88)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.88)
- EditInfinity: Image Editing with Binary-Quantized Generative Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.