Research map / Image generation / Editing and style transfer
BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BARET, a training-free-of-diffusion-finetuning real image editing method that fine-tunes only the target text embedding (target-text inversion) and balances self- and cross-attention maps from reconstruction and transition processes to support non-rigid edits.
Problem
Text-guided real image editing with diffusion models is limited in non-rigid edits, often needs image captions, or requires costly fine-tuning of the diffusion model.
Approach
Fine-tune the target text embedding (rather than the null-text embedding) for fast reconstruction without captions, interpolate between the fine-tuned and target embeddings with a decreasing schedule to create a transition embedding, and inject the reconstruction self-attention maps and transition cross-attention maps into the editing process (Balanced Attention Module).
Key results
- User study mean score 3.77 for BARET vs 2.93 for Null-text+p2p, 2.42 for SINE, 2.31 for MasaCtrl, 1.93 for SDEdit.
- BARET reports 1-LPIPS 0.764 and CLIPScore 0.729; Null-text has 1-LPIPS 0.766 but CLIPScore 0.681; SINE 0.619 and 0.7.
- Target-text inversion converges in about 16.1s (100 steps) vs null-text inversion needing over 400 steps and 48-66s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Target-text inversion schedule (fine-tune target text embedding) instead of null-text embedding fine-tuning | real image reconstruction inversion · reconstruction convergence | steps / time to convergence: converges at 100 steps in 16.1s vs null-text needing >400 steps and 48-66s | Helped |
| Balanced Attention Module plus progressive transition (BARET) vs. null-text inversion with prompt2prompt | real image editing · User study (100 pairs) | mean user score (1-5): 3.77 vs 2.93 (Null-text+p2p) | Helped |
| Null-text inversion (with prompt2prompt) vs. BARET on 1-LPIPS | real image editing · Objective evaluation | 1-LPIPS: Null-text 0.766 vs BARET 0.764 (Null-text slightly higher) | Mixed |
| BARET vs. null-text inversion on text alignment | real image editing · Objective evaluation | CLIPScore: 0.729 vs 0.681 | Helped |
| Interpolation parameter lowered from 0.8-0.1 to 0.6-0.1 and below (editing fidelity vs. text alignment tradeoff) (ablation) | progressive transition scheme · real image editing | CLIPScore and 1-LPIPS | Mixed |
| Self-attention injection step increased from 0.1 to 0.2-0.3 (vs. 0.4-0.5) (ablation) | Balanced Attention Module · real image editing | qualitative balance of fidelity and non-rigid change | Mixed |
| Cross-attention injection step varied between 0.4 and 0.8 (ablation) | Balanced Attention Module · real image editing | qualitative editing result | No clear effect |
| Fine-tuning target text embedding instead of null-text (removing caption requirement) | real image editing pipeline · inference time | inversion time: ~16s vs ~1m (Null-text) and ~20m (SINE) | Helped |
Surprising findings
- Null-text inversion attains slightly higher 1-LPIPS (0.766) than BARET (0.764), while BARET has much higher CLIPScore.
Limitations
- User study relies on 62 valid responses and a small set of 100 image-text pairs; quantitative metrics are limited to LPIPS and CLIPScore on top three methods.
- Non-rigid editing depends on hand-tuned interpolation and injection hyperparameters chosen empirically.
- Transition result alone fails the editing goal and background features are lost without self-attention injection.
Experimental setup
Datasets: TEdBench (referenced for user study prompts)
Benchmarks: TEdBench (user study pairs)
Metrics: user score (1-5) 1-LPIPS CLIPScore inference time
Baselines: SDEdit MasaCtrl Null-text inversion SINE Imagic
Builds on: DDIM inversion null-text inversion prompt-to-prompt Imagic SINE
Closest papers by meaning
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.89)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.89)
- LEDITS++: Limitless Image Editing using Text-to-Image Models (CVPR 2024 · similarity 0.89)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (ICLR 2024 · similarity 0.88)
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.