Research map / Image generation / Editing and style transfer
Counterfactual Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes counterfactual image editing with augmented SCMs, proves non-identifiability results, and proposes counterfactually consistent estimators implemented as ANCMs (NCM-based VAEs) evaluated on Colored MNIST with bars and CelebA-HQ.
Problem
Counterfactual image editing that edits one feature while preserving causal relationships among generative factors; existing methods ignore causal structure, and image counterfactual distributions are generally non-identifiable from observational data and causal diagrams.
Approach
Model image generation as an augmented SCM, show image counterfactuals are non-identifiable, and define counterfactual-consistent estimators (within optimal bounds over a care set of features) that are achieved by training graph-constrained neural causal models (ANCM) to fit the observational distribution.
Key results
- On Colored MNIST backdoor digit editing, ANCM estimates of P(C_{D=6}=1, B_{D=6}=0 | ...) lie within the optimal bounds, while baseline CVAE, CGN and DEAR estimates fall outside them.
- In the frontdoor setting, ANCM gives exactly 0 for the query P(C_{D=2}=0, B_{D=2}=0 | D=7, C=0, B=1) matching the true model, while CVAE and DEAR give values greater than 0.
- On CelebA-HQ age editing, ANCM preserves gender while CVAE and DEAR change gender in counterfactual images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ANCM instead of the non-causal baselines for digit editing | Colored MNIST Backdoor · Edit a red 3 with a bar to digit 6 | Ctf-consistency of the feature counterfactual query | Helped |
| Use ANCM or CGN rather than CVAE or DEAR for bar removal | Colored MNIST Backdoor · Remove the bar from a green 1 | P(green 1 without bar): 1 | Helped |
| Use ANCM rather than CVAE or DEAR for digit editing | Colored MNIST Frontdoor · Edit a green 7 with a bar to digit 2 | P(green 2 without bar): 0 | Helped |
| Use ANCM rather than CVAE or CGN for color editing | Colored MNIST Frontdoor · Edit a red 4 without a bar to green | P(green 4 with a bar) | Helped |
| Use ANCM rather than CVAE for mouth editing | CelebA-HQ Smiling setting · Open mouth | Preservation of smiling | Helped |
| Use ANCM rather than CVAE or DEAR for age editing | CelebA-HQ Age setting · Make the person older | Preservation of gender | Helped |
| Use a causal diagram that permits bidirected edges rather than enforcing Markovianity | DEAR · Colored MNIST Backdoor | Fit to the observational distribution | Helped |
Surprising findings
- Image counterfactual distributions are almost never identifiable from observational i.i.d. samples, and even with the causal diagram they remain non-identifiable.
- Markovian models (such as DEAR) cannot in general both fit the observational distribution and provide counterfactual-consistent estimates when the true model is semi-Markovian.
Limitations
- Requires labeled care-set features and a known causal diagram as inductive bias.
- Scalability and efficiency of inference to high-dimensional images are noted as open challenges.
- CelebA-HQ results rely on DiffuseVAE refinement, and the ground-truth causal diagram is not available.
Experimental setup
Datasets: Colored MNIST (modified with bars) CelebA-HQ
Benchmarks: Colored MNIST with bars (backdoor and frontdoor settings) CelebA-HQ
Metrics: empirical probability of counterfactual feature queries whether estimates fall within optimal bounds visual inspection of counterfactual images
Baselines: CVAE CGN DEAR
Builds on: Neural Causal Models Structural Causal Models Pearl Causal Hierarchy DiffuseVAE CGN DEAR
Closest papers by meaning
- Benchmarking Counterfactual Image Generation (NeurIPS 2024 · similarity 0.93)
- Counterfactual Generative Modeling with Variational Causal Inference (ICLR 2025 · similarity 0.90)
- Diffusion Counterfactual Generation with Semantic Abduction (ICML 2025 · similarity 0.90)
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.88)
- Towards Characterizing Domain Counterfactuals for Invertible Latent Causal Models (ICLR 2024 · similarity 0.87)
- Doubly Abductive Counterfactual Inference for Text-based Image Editing (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.