Research map / Image generation / Editing and style transfer
Benchmarking Counterfactual Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks SCM-based (Deep-SCM) counterfactual image generation model families (VAE, HVAE, GAN, with normalising flows for attributes) on MorphoMNIST, CelebA (two causal graphs) and ADNI brain MRI using composition, effectiveness, realism (FID) and minimality (CLD) metrics, and releases a Python package.
Problem
Counterfactual image generation lacks ground truth and a unified comparison; existing methods and metrics have not been evaluated together under causal constraints.
Approach
A unified benchmark framework that evaluates all Deep-SCM style amortized explicit (VAE, HVAE) and amortized implicit (GAN) mechanisms with a shared set of causal, effectiveness, realism and minimality metrics across several datasets and causal graphs, including new graphs for CelebA and new GAN setup for ADNI.
Key results
- HVAE achieves the lowest composition LPIPS on MorphoMNIST (0.006 at 1 cycle vs 0.026 VAE and 0.049 GAN) and on ADNI (0.101 vs 0.306 VAE).
- HVAE attains the best FID on MorphoMNIST (5.362 vs 10.124 VAE and 35.568 GAN) and on ADNI (74.696 vs 278.245 VAE).
- On CelebA (simple graph) HVAE achieves the best effectiveness for smiling (F1 0.998) and does worse on bald F1 across complex-graph interventions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HVAE instead of VAE | Counterfactual image generation · MorphoMNIST | LPIPS, 1 cycle: $0.026_{0.003}$ -> $\mathbf{0.006_{0.001}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · CelebA simple and complex | LPIPS, 1 cycle: $0.295_{0.004}$ -> $\mathbf{0.063_{0.003}}$; $0.282_{0.061}$ -> $\mathbf{ 0.122_{0.033}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · ADNI | LPIPS, 1 cycle: $0.306_{0.008}$ -> $\mathbf{0.101_{0.012}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · MorphoMNIST | Thickness MAE, do(t): $0.109_{0.01}$ -> $\mathbf{0.086_{0.09}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · MorphoMNIST | Digit accuracy, do(y): $0.775_{0.02}$ -> $\mathbf{0.972_{0.02}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · CelebA simple | Eyeglasses F1, do(e): $0.810_{0.02}$ -> $\mathbf{0.981_{0.02}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · CelebA complex | Age F1, do(a): $0.35_{0.04}$ -> $\mathbf{0.654_{0.1}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · ADNI | Ventricular volume MAE, do(v): $0.20_{0.04}$ -> $\mathbf{0.04_{0.01}}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · MorphoMNIST | FID: $10.124$ -> $\mathbf{5.362}$ | Helped |
| Use HVAE instead of VAE | Counterfactual image generation · CelebA simple | FID: $66.412$ -> $\mathbf{22.047}$ | Helped |
| Use GAN instead of HVAE | Counterfactual image generation · CelebA complex | FID: $35.712$ -> $\mathbf{27.861}$ | Helped |
| Add counterfactual fine-tuning to HVAE (ablation) | HVAE on ADNI · ADNI | model performance | Hurt |
Surprising findings
- HVAE ignored the conditioning on CelebA, requiring a counterfactual fine-tuning step to make interventions effective.
- HVAE fails to scale to 256x256 CelebA-HQ, with counterfactual training producing unrealistic artifacts.
Limitations
- Model-dependent effectiveness metrics require training anti-causal predictors whose number grows with graph size.
- Methods do not scale to higher resolution (CelebA-HQ 256x256) or to more complex graphs for single-variable models.
Experimental setup
Datasets: MorphoMNIST CelebA ADNI
Benchmarks: MorphoMNIST CelebA ADNI
Metrics: l1 distance LPIPS MAE F1 accuracy FID CLD
Baselines: VAE GAN HVAE
Builds on: Deep-SCM Hierarchical VAE Conditional GAN Counterfactual Latent Divergence LPIPS
Closest papers by meaning
- Counterfactual Image Editing (ICML 2024 · similarity 0.93)
- Diffusion Counterfactual Generation with Semantic Abduction (ICML 2025 · similarity 0.89)
- Counterfactual Generative Modeling with Variational Causal Inference (ICLR 2025 · similarity 0.88)
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.88)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.87)
- ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.