Research map / Label-efficient and robust learning / Recognition and long-tail
Rethinking Visual Counterfactual Explanations Through Region Constraint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes region-constrained visual counterfactual explanations (RVCEs) generated by adapting Image-to-Image Schrödinger Bridges with classifier guidance (RCSB), reaching new SOTA on ImageNet VCE benchmarks.
Problem
Visual counterfactual explanations modify images in entangled, dispersed ways, so it is unclear which factors cause the classifier's decision change; the paper argues a region constraint is needed for causal reasoning.
Approach
Constrain counterfactual edits to a predefined image region by framing VCE generation as classifier-guided conditional inpainting with an I²SB (Image-to-Image Schrödinger Bridge) prior, plus Tweedie-based gradient correction, ADAM stabilization, adaptive normalization, trajectory truncation, and grid-based automated region extraction from attribution maps.
Key results
- On Zebra-Sorrel, RCSB^A achieves FID 8.0 and sFID 16.2 versus 33.1 and 43.9 for DVCE (the best prior method in the table).
- On Cheetah-Cougar, RCSB^A reaches COUT 0.92 and FR 100.0, while ACE and LDCE baselines have COUT at or below 0.51.
- With LangSAM-extracted exact object regions, RCSB keeps FR around 99% and S3 of 0.81-0.91 across the three tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RCSB A instead of DVCE | ImageNet visual counterfactual generation with ResNet50 · Zebra – Sorrel | FID: 33.1 -> 8.0 | Helped |
| Use RCSB A instead of DVCE | ImageNet visual counterfactual generation with ResNet50 · Cheetah – Cougar | FID: 46.9 -> 17.2 | Helped |
| Use RCSB A instead of DVCE | ImageNet visual counterfactual generation with ResNet50 · Egyptian Cat – Persian Cat | FID: 46.6 -> 23.0 | Helped |
| Replace the I²SB prior with RePaint (ablation) | RCSB with automated region extraction · Zebra – Sorrel | FID: 8.0 -> 63.8 | Hurt |
| Replace the I²SB prior with RePaint (ablation) | RCSB with automated region extraction · Cheetah – Cougar | FID: 17.2 -> 129.3 | Hurt |
| Replace the I²SB prior with RePaint (ablation) | RCSB with automated region extraction · Egyptian Cat – Persian Cat | FID: 23.0 -> 148.7 | Hurt |
| Replace the I²SB prior with MCG (ablation) | RCSB with automated region extraction · Zebra – Sorrel | FID: 8.0 -> 43.2 | Hurt |
| Replace the I²SB prior with MCG (ablation) | RCSB with automated region extraction · Cheetah – Cougar | FID: 17.2 -> 76.6 | Hurt |
| Replace the I²SB prior with MCG (ablation) | RCSB with automated region extraction · Egyptian Cat – Persian Cat | FID: 23.0 -> 93.7 | Hurt |
| Replace the I²SB prior with DDRM (ablation) | RCSB with automated region extraction · Zebra – Sorrel | FID: 8.0 -> 42.5 | Hurt |
| Replace the I²SB prior with DDRM (ablation) | RCSB with automated region extraction · Cheetah – Cougar | FID: 17.2 -> 60.5 | Hurt |
| Replace the I²SB prior with DDRM (ablation) | RCSB with automated region extraction · Egyptian Cat – Persian Cat | FID: 23.0 -> 59.2 | Hurt |
Surprising findings
- Adapting RePaint, MCG, and DDRM to the same guidance and region pipeline matches or exceeds prior VCE SOTA, but they are far less compute-efficient (about 24x more NFEs than RCSB for RePaint).
Limitations
- Region-constrained explanations depend on the quality of the region extraction (attribution maps or LangSAM masks).
- Foundation-model generative priors trained on larger data may still produce artifacts; the method relies on I2SB trained on ImageNet-scale data.
- Area constraints above about 30-40% of the image degrade performance, since the I2SB checkpoint was trained on masks up to 30%.
Experimental setup
Datasets: ImageNet freeform masks from I2SB training data
Benchmarks: ImageNet Zebra-Sorrel ImageNet Cheetah-Cougar ImageNet Egyptian Cat-Persian Cat
Metrics: FID sFID S3 COUT Flip Rate
Baselines: ACE LDCE DVCE RePaint MCG DDRM
Builds on: I2SB (Image-to-Image Schrödinger Bridge) classifier guidance DiME RePaint Integrated Gradients
Closest papers by meaning
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.89)
- Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ (Abstract Reprint) (AAAI 2024 · similarity 0.88)
- LeapFactual: Reliable Visual Counterfactual Explanation Using Conditional Flow Matching (NeurIPS 2025 · similarity 0.88)
- Towards Fine-Grained Interpretability: Counterfactual Explanations for Misclassification with Saliency Partition (CVPR 2025 · similarity 0.88)
- Counterfactual Maps: What They Are and How to Find Them (NeurIPS 2026 · similarity 0.87)
- Structure Your Data: Towards Semantic Graph Counterfactuals (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.