Research map / Learning theory and algorithms / Prediction and decision losses
Statistical Test for Feature Selection Pipelines by Selective Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a selective-inference-based statistical test (with a line-search computation and auto-conditioning package si4automl) giving valid p-values for feature selection pipelines combining imputation, outlier detection, and feature selection components in linear models.
Problem
Quantifying the statistical significance of features selected by a data analysis pipeline that includes missing-value imputation, outlier detection, and feature selection steps, since resampling/CV cannot validly evaluate pipelines with unsupervised steps and data splitting reduces power.
Approach
Extends selective inference to arbitrary DAG-structured pipelines of linear MVI, OD, and FS components: the truncation set is computed by a parametric line search propagating update rules node by node, yielding a truncated-normal selective p-value valid for any configuration without per-pipeline derivations.
Key results
- The proposed selective test controls type I error at alpha=0.05 across n in {100,200,300,400} while the naive test fails to control it.
- On synthetic data the proposed method has much higher power than the oc baseline across all pipelines and effect sizes.
- On eight UCI datasets the proposed method achieves power such as 0.57 vs 0.07 (oc) for Data1 at n=100, and power rises with n (e.g., 0.91 at n=200 for Data1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Airfoil Self-Noise (Data1), n=200 | power: .11 -> .91 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Concrete Compressive Strength (Data2), n=200 | power: .08 -> .80 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Energy Efficiency (heating load, Data3), n=200 | power: .15 -> .78 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Energy Efficiency (cooling load, Data4), n=200 | power: .12 -> .66 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Gas Turbine CO and NOx Emission (Data5), n=200 | power: .13 -> .76 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Real Estate Valuation (Data6), n=200 | power: .08 -> .82 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Wine Quality (red wine, Data7), n=200 | power: .07 -> .49 | Helped |
| Use proposed selective inference instead of oc (over-conditioning) | all_cv pipeline · Wine Quality (white wine, Data8), n=200 | power: .06 -> .40 | Helped |
Surprising findings
- The naive test fails to control type I error for pipelines even under the null, showing the pipeline selection inflates false positives.
Limitations
- Limited to linear MVI, OD, and FS components (three each) and linear models with Gaussian noise assumed with known or estimated variance.
- Conditioning on the whole selection event (oc) is over-conservative; proposed method still depends on computing truncation intervals per component.
- Cross-validation-based pipeline selection is only handled in an extension, and computation time grows substantially with sample size.
Experimental setup
Datasets: Airfoil Self-Noise Concrete Compressive Strength Energy Efficiency Gas Turbine CO and NOx Emission Real Estate Valuation Wine Quality
Benchmarks: synthetic null and alternative datasets UCI real datasets (8 subsampled)
Metrics: type I error rate power
Baselines: oc naive test
Builds on: selective inference polyhedral lemma lasso selective inference stepwise selective inference
Closest papers by meaning
- Conformalized Multiple Testing after Data-dependent Selection (NeurIPS 2024 · similarity 0.84)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.84)
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.83)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.83)
- Towards a statistical theory of data selection under weak supervision (ICLR 2024 · similarity 0.83)
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.