Research map / Label-efficient and robust learning / Recognition and long-tail
Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes label and feature leakage in local feature selection, proves necessary and sufficient conditions for no leakage, and proposes SUWR, a sequential stop-or-select procedure proven leakage-free and trained with REINFORCE.
Problem
Local feature selection methods can encode information about the label or non-selected features in their selection masks, producing misleading explanations and inflated performance.
Approach
Defines label and feature leakage formally, derives conditions under which a selector has no leakage, and proposes SUWR: sequential feature selection where each step depends only on already-selected feature values and selections are never reversed, guaranteeing no leakage by construction.
Key results
- On the toy Pareto analysis, L2X, INVASE, TabNet and REAL-X exceed the leakage-free Pareto front, while SUWR stays close to it.
- On the synthetic benchmark, SUWR reaches oracle AUROC on Syn1-3 (e.g., 0.700, 0.895, 0.903) and maintains TPR above 97% in all settings.
- On fashion-MNIST, SUWR with six 3x3 patches outperforms CAE with ten patches; REAL-X performs considerably worse than both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SUWR instead of INVASE | Local feature selection · Syn1 | AUROC: .692 -> .700 | Helped |
| Use SUWR instead of TabNet | Local feature selection · Syn2 | AUROC: .885 -> .895 | Helped |
| Use SUWR instead of TabNet | Local feature selection · Syn3 | AUROC: .903 -> .903 | No clear effect |
| Use SUWR instead of INVASE | Local feature selection · Syn4 | AUROC: .792 -> .810 | Helped |
| Use SUWR instead of TabNet | Local feature selection · Syn5 | AUROC: .791 -> .816 | Helped |
| Use SUWR instead of INVASE | Local feature selection · Syn6 | AUROC: .877 -> .896 | Helped |
Surprising findings
- Existing local selection methods (including REAL-X, which adds noise to masks) exceed the leakage-free Pareto front, e.g., TabNet reaches perfect prediction with two features.
- Under the full-support assumption, the first selection step must be blind (independent of feature values), which the authors argue is unavoidable for leakage-free selection.
Limitations
- The conjecture that SUWR is the only leakage-free policy under full-support assumptions is not fully proven.
- SUWR requires sequential inference and may be computationally costlier, with potential scaling challenges to high-dimensional data.
- The linear programming method only applies to small, fully known discrete problems.
Experimental setup
Datasets: synthetic benchmark (Syn1-6, 11 Gaussian features) digits-MNIST fashion-MNIST
Benchmarks: synthetic benchmark Syn1-6 digits-MNIST fashion-MNIST toy binary sum-of-products Pareto analysis
Metrics: MSE AUROC TPR FDR CFSR accuracy number of selected features
Baselines: L2X INVASE TabNet REAL-X CAE local-optimal linear programming global-optimal brute force w/o FS oracle selector
Builds on: REINFORCE L2X INVASE REAL-X TabNet
Closest papers by meaning
- An Additive Instance-Wise Approach to Multi-class Model Interpretation (ICLR 2023 · similarity 0.86)
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.85)
- Contextual Feature Selection with Conditional Stochastic Gates (ICML 2024 · similarity 0.85)
- Interpretability with full complexity by constraining feature information (ICLR 2023 · similarity 0.84)
- A Learning Theoretic Perspective on Local Explainability (ICLR 2021 · similarity 0.83)
- Leveraging Local Variance for Pseudo-Label Selection in Semi-supervised Learning (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.