Research map / Vision-language and multimodal / Human and embodied modeling

V-CECE: Visual Counterfactual Explanations via Conceptual Edits

NeurIPS 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Abstract visual reasoning benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes V-CECE, a zero-training, black-box counterfactual explanation framework that applies optimal semantic concept edits (via WordNet bipartite matching) with a frozen Stable Diffusion inpainting model to compare how CNN, ViT and LVLM classifiers reason versus humans.

Problem

Counterfactual image explanations often use non-semantic or dispersed edits and assume classifiers reason at a human semantic level; there is little investigation of where neural classifiers and humans disagree on which concept edits change a label.

Approach

Computes minimum-cost semantic edits (insert, delete, substitute) between concept sets of source and target classes using WordNet shortest-path costs and Hungarian matching, orders them with an LVLM or global bias scores, and applies them with a frozen inpainting diffusion model, stopping when the black-box classifier flips.

Key results

• On BDD100K with Claude 3.5 Sonnet classifier, V-CECE Local-Global reaches 98.1% SR with FID 42.76 and 2.44 average edits, while DenseNet-driven edits reach only 84.8-88.9% SR.
• Human survey (31 participants) indicates DenseNet label flips about three edits later than humans would judge, with 59.7% of DenseNet counterfactuals visually correct versus 81.2% for Claude 3.5 Sonnet.
• On Visual Genome, V-CECE Local-Global achieves 98.43% (ResNet18), 99.77% (Claude 3 Haiku) and 98.87% (Claude 3.5 Sonnet) success rates with about 2.5-3.0 average edits, versus about 12-13 for prior semantic counterfactual methods.

Measured effects reported in the paper

Surprising findings

• Enabling the thinking module for Claude 3.7 Sonnet required more edits and yielded lower FID than the non-thinking mode with identical weights.
• CNN classifiers' global edits were sporadic with no consistent concept that reliably flipped the label, while LVLM classifiers showed consistent concept biases (e.g., removing car, pole).

Limitations

• Human survey cohort is modest (31 participants) and limits statistical power and generalizability.
• Generated images frequently contain artifacts, especially when removing large objects or after many edit steps.
• Edits rely on a generative inpainting model that is not trained on the data, so image quality is bounded by that editor.

Experimental setup

Datasets: BDD100K Visual Genome

Benchmarks: BDD100K Visual Genome (VG-Random)

Metrics: FID CMMD SimSiam similarity (S3) success rate average number of edits label stability across runs human survey step agreement and visual correctness

Baselines: STEEX DiME ACE TIME Dervakos et al. Dimitriou et al.

Builds on: WordNet Stable Diffusion inpainting GroundingDINO SAM DPM++ 2M SDE

Closest papers by meaning

• Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ (Abstract Reprint) (AAAI 2024 · similarity 0.91)
• Structure Your Data: Towards Semantic Graph Counterfactuals (ICML 2024 · similarity 0.89)
• Rethinking Visual Counterfactual Explanations Through Region Constraint (ICLR 2025 · similarity 0.89)
• Benchmarking Counterfactual Image Generation (NeurIPS 2024 · similarity 0.88)
• CounterFlowNet: From Minimal Changes to Meaningful Counterfactual Explanations (NeurIPS 2026 · similarity 0.88)
• LeapFactual: Reliable Visual Counterfactual Explanation Using Conditional Flow Matching (NeurIPS 2025 · similarity 0.88)