Research map / Image generation / Editing and style transfer
PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAIR Diffusion, which conditions diffusion models on per-object structure (panoptic segmentation) and appearance (VGG and DINOv2 pooled features) to enable inversion-free object-level image editing, plus multimodal classifier-free guidance for combining text and reference images.
Problem
Existing image editing methods lack fine-grained, object-level control over structure and appearance; text is too coarse for appearance and many mask/sketch-based methods cannot manipulate multiple objects independently or need inversion.
Approach
Represent each object by a structure (category and shape from panoptic segmentation) and an appearance vector (pooled low-level VGG and mid-level DINOv2 features), condition a diffusion model on these spatially, and extend classifier-free guidance factorization so reference-image appearance and text can be weighted independently; no inversion is needed at edit time.
Key results
- On LSUN Bedroom appearance control, PAIR Diffusion gets FID 12.81, L1 0.02, SSIM 0.51, outperforming E2EVE (FID 13.59, L1 0.05, SSIM 0.34) on all metrics.
- On CelebA-HQ structure control, PAIR Diffusion reaches 0.67 mIoU and 0.52 SSIM versus SEAN's 0.64 mIoU and 0.32 SSIM.
- Combining VGG and DINOv2 appearance features gives the best L1 (0.1891) and LPIPS (0.545) in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PAIR Diffusion instead of E2EVE for appearance control | appearance editing · LSUN Bedroom validation | SSIM: 0.34 -> 0.51 | Helped |
| use PAIR Diffusion instead of E2EVE for appearance control | appearance editing · LSUN Bedroom validation | FID: 13.59 -> 12.81 | Helped |
| use PAIR Diffusion instead of SEAN for structure control | structure control · CelebA-HQ validation | mIoU: 0.64 -> 0.67 | Helped |
| use PAIR Diffusion instead of SEAN for structure control | structure control · CelebA-HQ validation | SSIM: 0.32 -> 0.52 | Helped |
| combine VGG and DINOv2 features for appearance representation instead of using VGG features only (ablation) | PAIR Diffusion · COCO validation | LPIPS: 0.555 -> 0.545 | Helped |
| combine VGG and DINOv2 features for appearance representation instead of using DINOv2 features only (ablation) | PAIR Diffusion · COCO validation | L1: 0.1953 -> 0.1891 | Helped |
| use standard classifier-free guidance rather than multimodal classifier-free guidance (ablation) | PAIR Diffusion · text-guided object appearance editing | text control | Hurt |
| increase text guidance strength under standard classifier-free guidance (ablation) | PAIR Diffusion · text-guided object appearance editing | text control | No clear effect |
Surprising findings
- Standard classifier-free guidance factorization ignores the text prompt even when its guidance scale is increased, motivating the multimodal guidance formulation.
Limitations
- Appearance and structure formulation is simple (pooled vectors and panoptic maps); more sophisticated designs left for future work.
- Identity preservation of edited objects is weak; appearance is defined as visual characteristics rather than exact identity.
- Quantitative evaluation is limited to LSUN Bedroom, CelebA-HQ and COCO ablations rather than state-of-the-art comparisons.
Experimental setup
Datasets: LSUN Bedroom LSUN Church CelebA-HQ COCO
Benchmarks: LSUN Bedroom CelebA-HQ COCO
Metrics: FID L1 SSIM mIoU LPIPS
Baselines: Prompt-Free-Diffusion Paint-By-Example E2EVE SEAN Copy-Paste LDM inpainting CP + Denoise
Builds on: latent diffusion models ControlNet classifier-free guidance DINOv2 Prompt-Free-Diffusion
Closest papers by meaning
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.90)
- PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention (CVPR 2025 · similarity 0.89)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.89)
- FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Models (AAAI 2025 · similarity 0.89)
- Image Editing As Programs with Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.