Research map / Architectures and efficiency / RNNs and normalization
Why Regularized Auto-Encoders learn Sparse Representation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| constrain hidden weight length to one (ablation) | DAE with ReLU · MNIST and CIFAR-10 patches | achievable sparsity (avg. activation fraction) before dead units: 0.1 -> 0.05 | Helped |
| constrain weight length to one for CAE and mDAE (ablation) | CAE, mDAE with ReLU or Sigmoid · MNIST and CIFAR-10 patches | predictability of sparsity vs regularization coefficient | Helped |
| increase regularization coefficient with zero bias gradient (ReLU, unit weights) | CAE and mDAE · MNIST and CIFAR-10 patches | avg. activation fraction | No clear effect |
| increase corruption variance with ReLU | DAE · MNIST and CIFAR-10 patches | avg. activation fraction | Helped |
| use explicit first-order Taylor term of DAE (eDAE) (ablation) | auto-encoder with ReLU · MNIST and CIFAR-10 | avg. activation fraction vs sigma^2 | Helped |
| increase regularization coefficient with positive bias gradient (Sigmoid, unit weights) | AE models · MNIST and CIFAR-10 patches | sparsity | Helped |
Closest papers by meaning
- Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.86)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.86)
- Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth (ICML 2024 · similarity 0.86)
- Mechanistic Interpretability with Sparse Autoencoder Neural Operators (NeurIPS 2026 · similarity 0.85)
- Sparse Autoencoders, Again? (ICML 2025 · similarity 0.85)
- Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.