Research map / Architectures and efficiency / RNNs and normalization
SAND: One-Shot Feature Selection with Additive Noise Distortion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAND, a noise-distortion feature-selection layer with trainable gains normalized to sum of squares k, enabling one-shot feature selection during neural network training without retraining.
Problem
Embedded feature selection for neural networks typically requires modifying the loss, changing architectures, post-selection retraining, or hyperparameter sweeps to reach a target number of features k.
Approach
A non-intrusive input layer that multiplies each feature by a trainable gain a_i and adds Gaussian noise scaled by (1-a_i), with gains constrained by an l_alpha norm equal to k^(1/alpha); the constraint induces polarization of gains toward 1 (selected) and 0 (discarded), so the trained network is used directly for inference.
Key results
- On 9 benchmark datasets over 10 trials, SAND matches or is close to the best baselines on most datasets (e.g., MNIST 0.962 vs SL 0.963; Mice Protein 0.988) but is clearly worse on Madelon (0.732 vs SA 0.786 / LLY 0.862) and HAR70 (0.897 vs LLY 0.910).
- On CA Housing, SAND achieves MAE 0.521 vs 0.589-0.639 for baselines (lower is better).
- On the novel MSI Grain dataset with k=9, SAND reaches 0.919 accuracy vs 0.911-0.918 for baselines, with no retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAND instead of the strongest reported baseline | neural-network feature selection · Mice Protein | accuracy: 0.988 ± 0.007 -> 0.988 ± 0.006 | No clear effect |
| Use SAND instead of the strongest reported baseline | neural-network feature selection · MNIST-Fashion | accuracy: 0.856 ± 0.002 -> 0.836 ± 0.005 | Hurt |
| Use SAND instead of the strongest reported baseline | neural-network feature selection · CA Housing | MAE: 0.639 ± 0.063 -> 0.521 ± 0.027 | Helped |
| Use SAND instead of the strongest reported baseline | neural-network feature selection · MSI Grain | accuracy: 0.918 ± 0.002 -> 0.919 ± 0.002 | Helped |
| Vary σ from 1.0 to 3.0 instead of fixing it at 1.5 (ablation) | SAND · Madelon | accuracy: 0.741 ± 0.022; 0.732 ± 0.021; 0.708 ± 0.023; 0.689 ± 0.017; 0.646 ± 0.025 | Mixed |
| Vary σ from 1.0 to 3.0 instead of fixing it at 1.5 (ablation) | SAND · Mice Protein | accuracy: 0.988 ± 0.006; 0.988 ± 0.006; 0.988 ± 0.007; 0.988 ± 0.006; 0.987 ± 0.006 | No clear effect |
| Set α = 1.0 rather than α = 2.0 (reported σ settings) (ablation) | SAND · MNIST | accuracy: 0.959 ± 0.002; 0.956 ± 0.002; 0.962 ± 0.002 | No clear effect |
| Set α = 1.0 rather than α = 2.0 (reported σ settings) (ablation) | SAND · HAR70 | accuracy: 0.854 ± 0.036; 0.814 ± 0.038; 0.897 ± 0.005 | Hurt |
Surprising findings
- SAND's selected features are obtained without any retraining, yet on several datasets it matches methods that retrain after selection.
- Stochastic Gates fails to polarize beyond 37 features on MICE Protein (77 features), while SAND reaches the target k without a grid search.
Limitations
- Underperforms on Madelon and HAR70 relative to some baselines (e.g., LLY, SA), and the paper attributes this to lack of retraining.
- Theoretical analysis is limited to linear regression; no general guarantee for deep networks.
- Results rely on fixed sigma=1.5 and alpha=2 chosen heuristically; sensitivity is only tested on a subset of settings.
Experimental setup
Datasets: Mice Protein MNIST MNIST-Fashion ISOLET COIL-20 Activity Madelon HAR70 CA Housing MSI Grain
Benchmarks: Mice Protein MNIST MNIST-Fashion ISOLET COIL-20 Activity Madelon HAR70 CA Housing MSI Grain
Metrics: accuracy mean absolute error selection frequency
Baselines: Sequential Attention Sequential LASSO Batch-wise Attenuation Group LASSO Stochastic Gates
Builds on: Concrete Stochastic Gates Sequential Attention LassoNet Group LASSO
Closest papers by meaning
- Contextual Feature Selection with Conditional Stochastic Gates (ICML 2024 · similarity 0.87)
- Estimating Conditional Mutual Information for Dynamic Feature Selection (ICLR 2024 · similarity 0.87)
- Hypernetworks for Dynamic Feature Selection (NeurIPS 2026 · similarity 0.87)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.86)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.85)
- Balancing Feature Similarity and Label Variability for Optimal Size-Aware One-shot Subset Selection (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.