Research map / Learning theory and algorithms / Prediction and decision losses
Variable Importance in High-Dimensional Settings Requires Grouping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BCPI, a block-based conditional permutation importance framework with p-values for single and grouped variables, plus an internal stacking DNN layer that reduces computation for high-cardinality groups.
Problem
Variable importance in high-dimensional, highly correlated settings: permutation and conditional permutation importance become unreliable, inflate type-I error, and are computationally costly for many variables or large groups.
Approach
Combine conditional permutation importance computed at the group level with statistical p-values (Wald test), and embed stacking as an initial linear projection sub-layer inside the DNN so each group is summarized before importance is computed, reducing cost for high-cardinality groups.
Key results
- BCPI-DNN controls type-I error at the 5% target under inter-block correlations up to 0.8 while BPI-DNN fails in high-correlation settings.
- Internal stacking reduces computation time by around a factor of 2 for high-cardinality groups (100 variables per group) while keeping inferential behavior comparable.
- On UK Biobank brain age prediction, the reduced model after removing non-significant groups reached R2=0.8 and MAE=2.9 years, in line with CNN benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conditional permutation rather than standard permutation | DNN group importance (BCPI-DNN versus BPI-DNN) · Synthetic groups with high inter-group correlations | Type-I error | Helped |
| Add internal stacking (ablation) | BPI-DNN · Synthetic grouping benchmark | Type-I error | Helped |
| Use a Random Forest instead of a DNN as the BCPI learner | BCPI · Synthetic groups with strong inter-group correlations | Type-I error | Hurt |
| Add internal stacking (ablation) | BCPI-DNN · Synthetic groups of high cardinality | Inference performance | No clear effect |
| Add internal stacking (ablation) | BCPI-DNN · Synthetic groups of high cardinality | Computation time: around a factor of 2 | Helped |
| Remove non-significant groups | BCPI-DNN UKBB brain-age model · UKBB brain age prediction | R²: R^2=0.8 | No clear effect |
| Remove non-significant groups | BCPI-DNN UKBB brain-age model · UKBB brain age prediction | MAE: MAE=2.9 | No clear effect |
| Increase inter-group correlation | GOPFI and LOGI · Synthetic grouping benchmark | AUC | Hurt |
| Use grouped Shapley (SAGE) importance | Grouped SAGE · Synthetic data, low versus high correlation | AUC: mean $\approx 0.95$; mean $\approx 0.76$ | Mixed |
| Use BCPI-DNN rather than BCPI-RF | BCPI · Nonlinear synthetic case with high correlations | AUC | Helped |
Surprising findings
- Standard group permutation (BPI-DNN) fails to control type-I error under high inter-group correlation, while conditional version does.
- Random Forest-based BCPI-RF does not fully reach the target type-I error rate under strong correlations.
Limitations
- DNN base learner may memorize with limited training data; simpler learner like Random Forest may be preferable.
- Relies on predefined groups; statistically defined groups may mix heterogeneous variables and reliance on groups may fail if important variables are spread across groups.
- Only linear projections tested for internal stacking.
Experimental setup
Datasets: synthetic Gaussian block-correlated data UK Biobank (n=8357)
Benchmarks: synthetic block-correlated simulations UK Biobank brain age prediction
Metrics: AUC type-I error power computation time R2 MAE
Baselines: LOGI LOGO GOPFI GPFI Marginal Effects SAGE
Builds on: Conditional Permutation Importance Permutation Importance LOCO Leave-One-Group-Out Stacking
Closest papers by meaning
- Optimizing importance weighting in the presence of sub-population shifts (ICLR 2025 · similarity 0.85)
- Measuring Variable Importance in Heterogeneous Treatment Effects with Confidence (ICML 2025 · similarity 0.85)
- Testing Semantic Importance via Betting (NeurIPS 2024 · similarity 0.83)
- Permutation-Based Hypothesis Testing for Neural Networks (AAAI 2024 · similarity 0.83)
- Conditional Feature Importance with Generative Modeling Using Adversarial Random Forests (AAAI 2025 · similarity 0.83)
- Total Variation Floodgate for Variable Importance Inference in Classification (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.