Research map / Label-efficient and robust learning / Recognition and long-tail
Indirectly Parameterized Concrete Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Indirectly Parameterized Concrete Autoencoders (IP-CAE), which generate Gumbel-Softmax logits via a learnable embedding and shared linear map, improving stability, accuracy, reconstruction and training speed for embedded feature selection over CAE.
Problem
Concrete Autoencoders for neural embedded feature selection suffer from unstable joint optimization, duplicate feature selections, slow convergence and high variance, especially with non-linear decoders.
Approach
Instead of directly learning Gumbel-Softmax logits, learn an embedding matrix Psi and a shared linear map W (log alpha_i = W psi_i + b), which transforms gradients via W W^T and implicitly reduces duplicate selections; CAE is the special case W = I. A GJSD diversity regularizer is also proposed as a baseline.
Key results
- IP-CAE achieves the lowest reconstruction error on all six datasets, e.g. 1.36E-02 on MNIST vs 1.48E-02 for CAE and 1.98E-02 for LassoNet.
- IP-CAE reaches top-1 accuracy of 94.07% on MNIST and 97.92% on COIL-20 versus CAE's 83.10% and 80.70%.
- IP-CAE reaches CAE's validation performance up to roughly 2.5x to 25x faster depending on dataset and task (speedup table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace direct CAE selector-logit parameterization with indirect parameterization | CAE · MNIST, MNIST-Fashion, ISOLET, COIL-20, Smartphone HAR, and Mice Protein reconstruction | normalized reconstruction error: CAE: 1.48E-02, 1.36E-02, 5.42E-03, 4.21E-03, 3.03E-03, 8.71E-03; IP-CAE: 1.36E-02, 1.17E-02, 4.38E-03, 2.48E-03, 2.03E-0 | Helped |
| Replace direct CAE selector-logit parameterization with indirect parameterization | CAE · MNIST, MNIST-Fashion, ISOLET, COIL-20, Smartphone HAR, and Mice Protein classification | top-1 accuracy: CAE: 83.10, 73.19, 75.82, 80.70, 82.72, 63.10; IP-CAE: 94.07, 82.68, 91.85, 97.92, 93.71, 94.26 | Helped |
| Add GJSD regularization to CAE | CAE · Reconstruction and classification datasets | performance | Helped |
| Add GJSD regularization to IP-CAE (ablation) | IP-CAE · Datasets and tasks evaluated | performance | No clear effect |
| Train CAE for 1000 rather than 200 epochs (ablation) | CAE · ISOLET | accuracy: 75.8 -> 91.0 | Helped |
| Train CAE for 1000 rather than 200 epochs (ablation) | CAE · ISOLET | normalized reconstruction error: 0.067 -> 0.054 | Helped |
| Train IP-CAE for 1000 rather than 200 epochs (ablation) | IP-CAE · ISOLET | accuracy: 91.9 -> 91.4 | Hurt |
| Train IP-CAE for 1000 rather than 200 epochs (ablation) | IP-CAE · ISOLET | normalized reconstruction error: 0.054 -> 0.053 | Helped |
| Add learning-rate warmup over the first 25, 50, 75, or 100 epochs (ablation) | CAE training · Datasets and tasks evaluated | final performance | Hurt |
| Add weight decay with parameters 10^{-4}, 10^{-3}, 10^{-2}, or 10^{-1} (ablation) | CAE training · Datasets and tasks evaluated | performance | Hurt |
| Use a full matrix rather than scalar or diagonal IP weights (ablation) | IP-CAE · Training efficiency | training efficiency | Helped |
| Use scalar or diagonal IP weights instead of the CAE identity parameterization (ablation) | CAE · Stability and final performance | performance | Helped |
Surprising findings
- GJSD diversity regularization, which explicitly forces unique selections, improves CAE but remains inferior to IP-CAE, suggesting IP's benefit is not solely from removing duplicates.
- Extended CAE training to 1000 epochs on ISOLET improves CAE, but IP-CAE at 200 epochs still outperforms it.
Limitations
- Mechanism is largely empirical; the authors call for more formal analysis of implicit overparameterization.
- Experiments are limited to six small-to-medium tabular/image datasets with a small MLP decoder.
- Hyperparameter P is not tuned and the method's benefit vanishes when P is too small (underparameterized regime).
Experimental setup
Datasets: MNIST MNIST-Fashion ISOLET COIL-20 Smartphone HAR Mice Protein
Benchmarks: MNIST MNIST-Fashion ISOLET COIL-20 Smartphone HAR Mice Protein
Metrics: normalized Frobenius reconstruction error top-1 classification accuracy unique percentage speedup runtime
Baselines: CAE STG LassoNet GJSD regularization
Builds on: Concrete Autoencoders Gumbel-Softmax Generalized Jensen-Shannon divergence
Closest papers by meaning
- A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data (ICLR 2024 · similarity 0.85)
- Selective Deep Autoencoder for Unsupervised Feature Selection (AAAI 2024 · similarity 0.84)
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.83)
- ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection (NeurIPS 2026 · similarity 0.83)
- SASA: Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability (NeurIPS 2026 · similarity 0.83)
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.