Research map / Learning theory and algorithms / Prediction and decision losses
Identifying Important Group of Pixels using Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoXI, a greedy game-theoretic pixel/patch-group explanation method for image classifiers that uses self-context Shapley values and interactions to select patch groups with quadratic forward passes.
Problem
Visual explanations of image classifiers usually rank pixels by individual contribution, ignoring collective/redundant contributions; exact Shapley values and interactions need exponentially many forward passes.
Approach
Greedy selection of patches maximizing confidence, shown equivalent to maximizing self-context Shapley value plus interaction with already selected set (insertion), and a weighted full-context variant for deletion; reduces cost from exponential to quadratic forward passes using attention masking.
Key results
- Insertion: with 4% of patches visible, MoXI reaches 90% accuracy versus 2% (Grad-CAM), 4% (Attention rollout), 25% (Shapley value).
- Deletion: at 10% of image concealed, MoXI drops accuracy to 16% while Grad-CAM and Attention rollout stay near 79%.
- Computational cost reduced from O(|N|^2 ^|N|) to O(|N|^2) forward passes in worst case.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add interactions to self-context Shapley values (ablation) | MoXI · Visual SET-SUM task on synthetic MNIST | insertion accuracy | Helped |
| Add interactions to self-context Shapley values (ablation) | MoXI · ImageNet, ViT-T insertion (4% visible) | top-1 accuracy: 90% vs 25%, 4%, and 2% | Helped |
| Add interactions to self-context Shapley values (ablation) | MoXI · ImageNet, ViT-T deletion (10% concealed) | top-1 accuracy: 16% vs approximately 79% | Helped |
| Use MoXI patch ranking to select patches corrupted with Gaussian noise | MoXI · ImageNet, ViT-T, Gaussian noise level 5 | top-1 accuracy | Mixed |
Surprising findings
- Pixels with highest individual Shapley values can be redundant; collecting them does not yield the most informative set, motivating interaction-aware selection.
Limitations
- Shapley value baseline computed approximately via random sampling of 200 subsets
- Model is vulnerable to Gaussian noise corruptions in deletion setting (accuracy drops sharply)
- Evaluated only on image classifiers with patch-level granularity rather than pixels
Experimental setup
Datasets: ImageNet MNIST
Benchmarks: ImageNet
Metrics: top-1 accuracy under insertion curve top-1 accuracy under deletion curve accuracy under common corruptions
Baselines: Grad-CAM Grad-CAM++ Attention rollout Shapley value MoXI(-)
Builds on: Shapley values interaction index SHAP RISE
Closest papers by meaning
- Unlocking the Game: Estimating Games in Möbius Representation for Explanation and High-Order Interaction Detection (AAAI 2025 · similarity 0.85)
- shapiq: Shapley Interactions for Machine Learning (NeurIPS 2024 · similarity 0.85)
- I-CEE: Tailoring Explanations of Image Classification Models to User Expertise (AAAI 2024 · similarity 0.82)
- Support Vector-based Estimation of Multilinear Games for Feature Selection and Explanation (AAAI 2025 · similarity 0.82)
- Exploring the cloud of feature interaction scores in a Rashomon set (ICLR 2024 · similarity 0.82)
- Discover and Mitigate Multiple Biased Subgroups in Image Classifiers (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.