Research map / Learning theory and algorithms / Prediction and decision losses
Sequential Attention for Feature Selection
Taisuke Yasuda, Mohammadhossein Bateni, Lin Chen, Matthew Fahrbach, Gang Fu, Vahab Mirrokni
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select features adaptively over multiple rounds (fewer features per round) instead of selecting many at once (ablation) | Sequential Attention · ISOLET, 64 features | accuracy: 0.934 (1 per round) vs 0.782 (64 per round) | Improved |
| Select features adaptively (fewer per round) instead of in large batches (ablation) | Sequential Attention · MNIST, 64 features | accuracy: 0.963 (1 per round) vs 0.932 (64 per round) | Improved |
| Adaptivity (more rounds) on mice protein (ablation) | Sequential Attention · Mice Protein, 64 features | accuracy: 0.990 vs 0.990 (differences within noise) | No clear effect |
| Replace softmax overparameterization with other Hadamard parameterizations (l1, l2, normalized variants) (ablation) | Sequential Attention · Small-scale benchmarks (MNIST, ISOLET, Activity, etc.) | accuracy: Mostly within ~0.01 of softmax; e.g., MNIST 0.957-0.958 vs softmax 0.958 | No clear effect |
| Replace softmax with unnormalized l1 |w| Hadamard parameterization (ablation) | Sequential Attention · ISOLET | accuracy: 0.894 vs 0.920 (softmax) | Hurt |
| Use normalized l1 instead of softmax overparameterization (ablation) | Sequential Attention · ISOLET and Activity | accuracy: ISOLET 0.921 vs 0.920; Activity 0.933 vs 0.922 | No clear effect |
| Sequential Attention vs Group LASSO / Sequential LASSO on small-scale benchmarks | Sequential Attention · Small-scale benchmark suite (k=50) | accuracy: SA best on ISOLET (0.920 vs GL 0.906) but OMP higher on Mice Protein (0.994 vs 0.993) and GL higher on Activity (0.933 v | Mixed |
| Sequential Attention vs LASSO-based methods on Criteo at larger k | Sequential Attention · Criteo large, k>=15 AUC | AUC: SA 0.72659 vs GL(1e-1) 0.72392 at k=15; GL(1e-4) 0.73512 vs SA 0.73495 at k=35 | Mixed |
Closest papers by meaning
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.88)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.86)
- Estimating Conditional Mutual Information for Dynamic Feature Selection (ICLR 2024 · similarity 0.86)
- Fundamental limits of learning in sequence multi-index models and deep attention networks: high-dimensional asymptotics and sharp thresholds (ICML 2025 · similarity 0.86)
- Contextual Feature Selection with Conditional Stochastic Gates (ICML 2024 · similarity 0.84)
- PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
