Research map / Label-efficient and robust learning / Recognition and long-tail
Feature Importance Metrics in the Presence of Missing Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distinguishes full-data versus observed-data feature importance under missing data, shows estimation pitfalls, and proposes the feature measurement importance gradient (FMIG) to guide which features to measure more often, validated on synthetic time-series experiments.
Problem
Feature importance metrics are ill-defined under missing data because the estimand (full-data vs observed-data) is rarely specified, identification assumptions are unstated, and common imputation-based estimators are biased; no metric tells analysts which features to measure more often.
Approach
Separates full-data and observed-data LOCO estimands, analyzes identification (positivity, MCAR/MAR/MNAR) and estimation bias, and introduces FMIG: the gradient of expected negative loss with respect to an incremental-propensity-score perturbation of a feature's measurement probability, which is identified without positivity.
Key results
- In MCAR synthetic experiments, full-data LOCO rises from X1 to X3 while observed-data LOCO stays flat because higher missingness offsets predictive gains.
- Under MNAR, observed-data LOCO of X3 drops further as its information leaks into correlated missingness indicators, while full-data LOCO is unchanged.
- Complete case analysis is unbiased under valid positivity but shows strong bias when positivity is violated (p(R=1) about 0.1 to 0.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compare features with increasing predictive importance but decreasing measurement probability | Full data and observed data LOCO · Synthetic Experiment 1 (MCAR) | LOCO | Mixed |
| Introduce MNAR missingness in which X3 influences missingness of later features | Observed data LOCO for X3 · Synthetic Experiment 2 (MNAR) | LOCO | Hurt |
| Violate positivity by strongly increasing missingness | Complete case analysis for full data LOCO · Synthetic Experiment 3 | Estimation bias | Hurt |
| Use complete case analysis under MCAR with valid positivity | Full data LOCO · Synthetic Experiment 1 | Estimation bias | No clear effect |
| Increase the measurement probability of X2 | FMIG · Synthetic Experiment 4 (MCAR) | Prediction loss | Helped |
| Further increase measurement of X1, which is already fully observed | FMIG · Synthetic Experiment 4 (MCAR) | Prediction loss | No clear effect |
| Measure the constant feature X3 more frequently | FMIG · Synthetic Experiment 4 (MCAR) | Prediction loss | Hurt |
| Use mean imputation within an impute-then-regress classifier while retaining missingness indicators | Observed data feature importance · Observed data | Estimation bias | No clear effect |
| Use conditional mean imputation instead of modeling the full density of missing features | Full data and observed data feature importance · Missing data | Estimation bias | Hurt |
| Exclude the label from the multiple-imputation model | Multiple imputation for full data feature importance · Missing data | Feature importance estimation | Hurt |
Surprising findings
- A feature with high full-data and observed-data importance can have near-zero FMIG if measuring it more often adds no predictive information.
- Mean imputation is unbiased for observed-data importance but biased for full-data importance.
Limitations
- Evaluation is limited to synthetic data.
- FMIG does not prescribe an optimal measurement policy.
- Full-data LOCO requires strong, often unrealistic assumptions and positivity.
Experimental setup
Datasets: synthetic time-series data
Benchmarks: synthetic time-series experiments
Metrics: LOCO (full data and observed data) FMIG loss difference bootstrap confidence intervals
Baselines: complete case analysis mean imputation conditional mean imputation multiple imputation
Builds on: LOCO (Lei et al., 2018) incremental propensity score interventions (Kennedy, 2019) impute-then-regress classifiers (Le Morvan et al., 2021) temporal convolutional network (Bai et al., 2018)
Closest papers by meaning
- Feature Importance Disparities for Data Bias Investigations (ICML 2024 · similarity 0.85)
- Random features models: a way to study the success of naive imputation (ICML 2024 · similarity 0.84)
- Conditional Feature Importance with Generative Modeling Using Adversarial Random Forests (AAAI 2025 · similarity 0.84)
- Prediction models that learn to avoid missing values (ICML 2025 · similarity 0.83)
- Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data (ICML 2025 · similarity 0.83)
- Interpretable Generalized Additive Models for Datasets with Missing Values (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.