Research map / Label-efficient and robust learning / Recognition and long-tail
Estimating Conditional Mutual Information for Dynamic Feature Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIME, which estimates conditional mutual information for greedy dynamic feature selection in a discriminative way by training a value network to regress on predictor loss improvements, and extends it to prior information, non-uniform costs, and variable budgets.
Problem
Dynamic feature selection needs both accurate predictions from arbitrary feature subsets and a policy choosing the most valuable next feature; existing CMI-based methods rely on generative models, while discriminative methods skip CMI estimation entirely.
Approach
Train a value network to regress onto the incremental cross-entropy loss improvement from adding a feature, which is an unbiased estimate of CMI under the Bayes-optimal predictor, so the network learns CMI discriminatively; jointly train with a predictor on masked inputs.
Key results
- On MNIST, DIME achieves above 90% accuracy with an average of about 10/784 features (1.27%).
- On Imagenette, DIME reaches nearly 97% accuracy with about 15/196 patches (7.7%).
- DIME provides consistent gains over baselines such as Argmax Direct, EDDI, CAE and RL methods across tabular and image datasets, with ViT backbones outperforming ResNets on image data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIME's penalized stopping criterion rather than the compared methods | Dynamic feature selection · Intubation, ROSMAP, and MNIST | AUROC on medical tasks; top-1 accuracy on MNIST | Helped |
| Use DIME's penalized stopping criterion rather than the compared methods | Dynamic feature selection · Imagenette, ImageNet-100, and MHIST | top-1 accuracy on image subsets; AUROC on MHIST | Helped |
| Use penalized stopping rather than budget-constrained stopping | DIME · Imagenette and ImageNet-100 | top-1 accuracy | Helped |
| Use penalized stopping rather than fixed-budget stopping | DIME · Tabular and image datasets | classification performance | Helped |
| Use true non-uniform feature costs during training and inference (ablation) | DIME · Intubation and ROSMAP | classification performance | Helped |
| Use true costs only during inference instead of ignoring costs (ablation) | DIME · ROSMAP | classification performance | Helped |
| Use true costs only during inference instead of ignoring costs (ablation) | DIME · Intubation | classification performance | No clear effect |
| Use ViT rather than ResNet (ablation) | DIME · Imagenette and ImageNet-100 | top-1 accuracy | Helped |
| Add Canny edge image as prior information for both feature selection and prediction | DIME · MHIST | AUROC | Helped |
| Add the sketch to the predictor only, omitting it from feature selection (ablation) | DIME · MHIST | AUROC | Helped |
| Use the sketch in both selection and prediction rather than predictor-only (ablation) | DIME · MHIST | AUROC | Helped |
| Use DIME rather than EDDI or CwCF with non-uniform feature costs | Dynamic feature selection · Intubation and ROSMAP | classification performance | Helped |
Surprising findings
- Predictor suboptimality yields CMI estimates biased in either direction, since the bias is a difference of KL terms.
- EDDI usually does not perform well despite being a generative approach, and RL methods (CwCF, OL) are not competitive.
Limitations
- No performance guarantees since DFS is not adaptive submodular for non-uniform costs; the cost-ratio criterion is a heuristic.
- Theoretical CMI recovery assumes infinitely expressive networks and Bayes-optimal predictor; predictor suboptimality biases estimates.
- Intubation dataset is private, limiting reproducibility.
Experimental setup
Datasets: Intubation (emergency medicine, d=112) ROSMAP MNIST Imagenette ImageNet-100 MHIST
Benchmarks: Intubation ROSMAP MNIST Imagenette ImageNet-100 MHIST
Metrics: AUROC top-1 accuracy average feature acquisition cost
Baselines: Concrete Autoencoder mRMR CMICOT Argmax Direct EDDI Hard Attention CwCF Opportunistic learning
Builds on: Argmax Direct EDDI greedy CMI selection Concrete Autoencoder
Closest papers by meaning
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.87)
- Contextual Feature Selection with Conditional Stochastic Gates (ICML 2024 · similarity 0.87)
- Distributionally Robust Feature Selection (NeurIPS 2025 · similarity 0.86)
- Hypernetworks for Dynamic Feature Selection (NeurIPS 2026 · similarity 0.85)
- Stochastic Encodings for Active Feature Acquisition (ICML 2025 · similarity 0.84)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.