Research map / Label-efficient and robust learning / Recognition and long-tail
Feature Selection for Latent Factor Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes per-class low-rank latent factor models (PPCA, LFA, and a new nonparametric ELF) with an SNR-based feature selection criterion, proven to recover true features asymptotically, and applies it to class-incremental multi-class classification on CIFAR-10/100 and ImageNet-1k features from CLIP.
Problem
Feature selection for high-dimensional multi-class classification is usually done jointly over all classes, which scales poorly with class count and makes class-incremental learning hard; existing SNR-style criteria lack feature recovery guarantees.
Approach
Model each class separately with a low-rank latent factor model (PPCA/LFA/ELF/HeteroPCA), compute a per-feature SNR from the estimated loadings and noise variances, select top-SNR features per class, and classify with Mahalanobis distance in the selected subspace; new classes can be added without retraining existing class models.
Key results
- On CIFAR-100 with 2,560 features, PPCA reaches 73.01% accuracy at 2,000 features versus 69.63% for FSA and TISP using all features.
- On ImageNet-1k, ELF reaches 70.24% accuracy with 300 of 640 features, while FSA and TISP get 63-64% at similar sizes.
- In simulation with n=300 and 10 noise features, LFA recovers true features with 100% accuracy versus 98% for HeteroPCA and 88% for PPCA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELF SNR selection to retain 1,500 features instead of all 2,560 (ablation) | ELF with CLIP RN50x4 · CIFAR-10 | accuracy: 91 -> 91 | No clear effect |
| Use ELF SNR selection to retain 750 features instead of all 2,560 (ablation) | ELF with CLIP RN50x4 · CIFAR-10 | accuracy: 91 -> 89.41 | Hurt |
| Use PPCA SNR selection to retain 2,000 features instead of all 2,560 (ablation) | PPCA with CLIP RN50x4 · CIFAR-100 | accuracy: 72.81 -> 73.01 | Helped |
| Use ELF SNR selection to retain 750 features instead of all 2,560 (ablation) | ELF with CLIP RN50x4 · CIFAR-100 | accuracy: 72.81 -> 65.11 | Hurt |
| Use ELF SNR selection to retain 300 features instead of all 640 (ablation) | ELF with CLIP RN50x4 · ImageNet-1k | accuracy: 73.73 -> 70.24 | Hurt |
| Use LFA rather than PPCA for SNR feature recovery | Low-rank generative feature selection · Simulated data (10 true features, 10 noisy features), n=50 | feature selection accuracy: 71.2 -> 90.6 | Helped |
| Use ELF rather than PPCA for SNR feature recovery | Low-rank generative feature selection · Simulated data (10 true features, 10 noisy features), n=50 | feature selection accuracy: 71.2 -> 87.6 | Helped |
| Use ELF SNR selection rather than FSA | Feature selection with CLIP RN50x4 · CIFAR-100 | accuracy: 69.63 -> 72.81 | Helped |
| Use ELF SNR selection rather than TISP | Feature selection with CLIP RN50x4 · ImageNet-1k | accuracy: 71.37 -> 73.73 | Helped |
Surprising findings
- Plain PPCA gives the best CIFAR-100 accuracy (73.01% at 2,000 features) despite being the weakest estimator in simulation.
- On CIFAR-10, FSA and TISP reach slightly higher peak accuracy (91.3% and 91.2%) than the proposed methods, which plateau around 91%.
Limitations
- Theoretical recovery guarantees are asymptotic with fixed d and only for PPCA and LFA; non-asymptotic bounds are left to future work.
- Evaluation uses frozen CLIP features and a single one-class simulation setting rather than end-to-end models.
- Real-data gains over FSA/TISP are mixed; on CIFAR-10 the baselines reach slightly higher peak accuracy.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k simulated latent factor data
Metrics: classification accuracy feature selection accuracy mean absolute deviation of signal, noise variance, and SNR e training time
Baselines: FSA TISP PPCA LFA HeteroPCA
Builds on: Probabilistic PCA Latent Factor Analysis HeteroPCA SELF Feature Selection Annealing
Closest papers by meaning
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.84)
- Contextual Feature Selection with Conditional Stochastic Gates (ICML 2024 · similarity 0.82)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.82)
- Towards a statistical theory of data selection under weak supervision (ICLR 2024 · similarity 0.81)
- Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond (ICML 2024 · similarity 0.81)
- Best Subset Selection: Optimal Pursuit for Feature Selection and Elimination (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.