Research map / Image generation / Editing and style transfer
Doubly Abductive Counterfactual Inference for Text-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates text-based image editing as counterfactual inference and proposes DAC, which abducts two LoRA exogenous variables (UNet LoRA for image details and CLIP text-encoder LoRA for the semantic change) on Stable Diffusion 2.1 so that subtracting the second LoRA performs the edit.
Problem
Text-based editing of a single real image must balance editability (following the target prompt) against fidelity (preserving the source), and single-image fine-tuning overfits and loses editability.
Approach
Two doubly abducted exogenous variables: a UNet LoRA U encodes image details, and a text-encoder LoRA Δ, fit with an annealed U-weighting, encodes the visual transition from P' to P; negating Δ at inference edits the image while keeping fidelity.
Key results
- In a user study on AMT (110 participants, 5,940 answers), 75.3% preferred DAC over DDS, Imagic, and SINE.
- DAC takes about 15 minutes per image (6 min first abduction, 9 min second) on one A100, versus 120 min for SINE, 0.33 min for DDS and 12 min for Imagic.
- CLIP-score and LPIPS are reported per editing type; DAC gets higher text alignment for removal, manipulation, addition and face manipulation, but LPIPS is argued to be an unreliable fidelity measure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DAC instead of DDS | Text-based image editing · User study (54 image-prompt pairs) | preference rate: 4.4 -> 75.3% | Helped |
| Use DAC instead of Imagic | Text-based image editing · User study (54 image-prompt pairs) | preference rate: 13.7 -> 75.3% | Helped |
| Use DAC instead of SINE | Text-based image editing · User study (54 image-prompt pairs) | preference rate: 6.6 -> 75.3% | Helped |
| Increase Abduction-1 optimization iterations (ablation) | DAC · Dog manipulation | editability | Hurt |
| Increase Abduction-1 optimization iterations (ablation) | DAC · Dog manipulation | fidelity | Helped |
| Set β to vary from -1 to 1 for the subtracted text LoRA (ablation) | DAC · Image editing | semantic change strength: from -1 to 1 | Helped |
| Increase annealing parameter η (ablation) | DAC · Image editing | fidelity: η = 0.2; η = 0.4; η = 0.6; η = 0.8 | Helped |
| Increase annealing parameter η (ablation) | DAC · Image editing | editability: η = 0.2; η = 0.4; η = 0.6; η = 0.8 | Hurt |
| Add text-encoder LoRA T during Abduction-1 (ablation) | DAC · Image editing | fidelity | Helped |
Surprising findings
- LPIPS favors methods that make no change to the source image (DDS, SINE), so the authors argue it fails to reflect fidelity and rely on a user study instead.
Limitations
- Per-image fine-tuning takes about 15 minutes, far slower than inference-only editing methods.
- Relies on per-image overfitting hyperparameters (eta annealing, LoRA ranks), with eta chosen per image.
- Fidelity is evaluated mainly by a user study since LPIPS and CLIP-score are imperfect proxies.
Experimental setup
Datasets: Unsplash images (source images)
Benchmarks: in-house TBIE image-prompt set (54 images, 6 editing types)
Metrics: CLIP-score LPIPS user preference rate
Baselines: Imagic SINE DDS InstructPix2Pix SEED-LLaMA Emu2 P2P PnP MasaCtrl DDIM inversion
Builds on: Stable Diffusion LoRA Imagic DDS SINE
Closest papers by meaning
- Counterfactual Image Editing (ICML 2024 · similarity 0.86)
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.86)
- Diffusion Counterfactual Generation with Semantic Abduction (ICML 2025 · similarity 0.86)
- EditInfinity: Image Editing with Binary-Quantized Generative Models (NeurIPS 2025 · similarity 0.86)
- Text-Driven Fashion Image Editing with Compositional Concept Learning and Counterfactual Abduction (CVPR 2025 · similarity 0.85)
- Benchmarking Counterfactual Image Generation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.