Research map / Probabilistic and scientific modeling / Variational and generative sampling
Total Variation Floodgate for Variable Importance Inference in Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines the expected total variation (ETV) as a model-free variable importance measure for categorical responses and extends the floodgate method to give cross-validated lower confidence bounds for it under model-X assumptions.
Problem
Quantifying variable importance for a feature X on a categorical outcome Y conditional on confounders Z, in a model-free way with valid statistical inference (lower confidence bounds).
Approach
Define ETV as the normalized expected total variation between L(Y|X,Z) and L(Y|Z); lower-bound it by reducing to classification accuracy between real samples and resampled-X samples drawn from L(X|Z), then apply a CLT-based floodgate lower confidence bound with cross-validated classifiers and a threshold c for abstention.
Key results
- On the US election conjoint data, the floodgate ETV lower bound is around 0.08 and the estimate around 0.1 with cross-validated abstention threshold c.
- In the synthetic conjoint setting, ETV is (1-q)|p-0.5| and with p=1 and q≈0.27 the maximum achievable ETV is about 0.365, far below the general upper bound of 1.
- Simulations show the oracle classifier gives the highest floodgate bound, the logistic classifier is a close second, and even random forest performs reasonably well.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the oracle classification function instead of the logistic model | cross-validated Floodgate · Synthetic simulations, p=4 or 10 | ETV lower bound | Helped |
| use a logistic classifier instead of a random forest | cross-validated Floodgate · Synthetic simulations, p=4 or 10 | ETV lower bound | Helped |
| choose threshold c with cross-validation instead of setting c=0 | logistic Floodgate classifier · Synthetic simulations, p=10 | ETV lower bound | Helped |
| choose threshold c with the oracle procedure instead of cross-validation | logistic Floodgate classifier · Synthetic simulations, p=10 | ETV lower bound | Helped |
| activate threshold c in the classifier | Floodgate with HierNet · US general election conjoint analysis | ETV estimate | Helped |
| activate threshold c in the classifier | Floodgate with HierNet · US general election conjoint analysis | ETV lower bound | Helped |
Surprising findings
- Even when the ETV reaches its ideal maximum in the election-style setting, it is only about 0.365, far below the theoretical upper bound of 1, so the natural bound is loose in practice.
- The theorem shows no valid upper confidence bound on ETV can be constructed without strong assumptions, only lower bounds.
Limitations
- Requires the ability to sample from L(X|Z) (design-based/model-X assumption).
- Only lower confidence bounds are provided; a valid upper bound is shown to be impossible in general without strong assumptions.
- Abstention threshold c discards ambiguous samples, and performance depends on the quality of working models.
Experimental setup
Datasets: simulated Gaussian covariates with Bernoulli responses US general election conjoint data (n=7190)
Benchmarks: simulated floodgate lower bounds US general election conjoint data
Metrics: floodgate lower confidence bound ETV estimate coverage
Baselines: oracle classifier logistic classifier random forest Naive c=0 AMCE
Builds on: floodgate model-X knockoffs HierNet
Closest papers by meaning
- Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guarantees (NeurIPS 2022 · similarity 0.85)
- Lazy Estimation of Variable Importance for Large Neural Networks (ICML 2022 · similarity 0.84)
- Variable Importance in High-Dimensional Settings Requires Grouping (AAAI 2024 · similarity 0.83)
- Beyond the Norms: Detecting Prediction Errors in Regression Models (ICML 2024 · similarity 0.82)
- Differentiable Annealed Importance Sampling Minimizes The Jensen-Shannon Divergence Between Initial and Target Distribution (ICML 2024 · similarity 0.82)
- Forward $\chi^2$ Divergence Based Variational Importance Sampling (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.