Research map / Architectures and efficiency / RNNs and normalization
Sparse dictionary representations
Extracts interpretable features with sparse autoencoders and dictionary views of implicit neural representations.
98 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “layer · activation · functions”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 63 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.2% to 6.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Why Regularized Auto-Encoders learn Sparse Representation? (ICML 2016 · 13 citations)
- 2017 · Sparse Boltzmann Machines with Structure Learning as Applied to Text Analysis (AAAI 2017 · 16 citations)
- 2018 · A Hybrid l1-l0 Layer Decomposition Model for Tone Mapping (CVPR 2018 · 161 citations)
- 2020 · Implicit Rank-Minimizing Autoencoder (NeurIPS 2020 · 16 citations)
- 2021 · What does LIME really see in images? (ICML 2021 · 2 citations)
- 2022 · On the Parameterization and Initialization of Diagonal State Space Models (NeurIPS 2022 · 73 citations)
- 2023 · TINC: Tree-Structured Implicit Neural Compression (CVPR 2023 · 21 citations)
- 2024 · Improved Implicit Neural Representation with Fourier Reparameterized Training (CVPR 2024 · 28 citations)
- 2025 · Scaling and evaluating sparse autoencoders (ICLR 2025 · 11 citations)
- 2026 · Sparse Autoencoders Trained on the Same Data Learn Different Features (ICLR 2026 · 1 citation)
Earliest papers here
- Why Regularized Auto-Encoders learn Sparse Representation? (ICML 2016 · 13 citations)
- Sparse Boltzmann Machines with Structure Learning as Applied to Text Analysis (AAAI 2017 · 16 citations)
- A Hybrid l1-l0 Layer Decomposition Model for Tone Mapping (CVPR 2018 · 161 citations)
What has been tried
55 papers here report 566 measured changes (a component added, removed or swapped, compared with a baseline): helped 55%, no clear effect 10%, hurt 30%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use random bases instead of fixed Fourier bases | Kodak 24 (first 3 images), average · PSNR | 24.71 -> 23.83 | Hurt | Improved Implicit Neural Representation with Fourier Reparameterized Training (CVPR 2024 · 28 citations) |
| add day-cycle gating and encoding | METR-LA, Average · MAE | 2.78 -> 2.71 | Helped | ModWaveMLP: MLP-Based Mode Decomposition and Wavelet Denoising Model to Defeat Complex Structures in Traffic Forecasting (AAAI 2024 · 24 citations) |
| Add tied encoder-decoder initialization | GPT-2 small, 128k latents, k=32 · MSE | Helped | Scaling and evaluating sparse autoencoders (ICLR 2025 · 11 citations) | |
| Add spectral-sampling initialization with low and high frequency regions; increase high-frequency share from 30% to 70% at l=21 | Kodak images · Signal PSNR (dB) | 35.2 -> 35.0 | Hurt | Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks (CVPR 2025 · 5 citations) |
| Place SVA after layer2 rather than elsewhere | RML2016.10a · OA (%) | ResNet18: 58.84 -> 61.49; ResNet34: 59.40 -> 61.71; ResNet50: 59.26 -> 60.41 | Helped | Harnessing the Power of SVD: An SVA Module for Enhanced Signal Classification (AAAI 2024 · 5 citations) |
| Freeze decoder directions at initialization | IOI · sparse controllability | Hurt | Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control (ICLR 2025 · 3 citations) | |
| randomly re-initialize pruned weights instead of leaving them unperturbed | CIFAR-100, ε=10% · test accuracy | Hurt | DSD²: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free? (AAAI 2024 · 3 citations) | |
| Increase kernel size to ℓ=16 or 32 | AWGN · BER | Helped | DeepPolar: Inventing Nonlinear Large-Kernel Polar Codes via Deep Learning (ICML 2024 · 2 citations) | |
| Replace the ℓ2 reconstruction loss with an ℓ1 loss | SAE training and concept interpretability · training dynamics and interpretability | Helped | Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (ICML 2025 · 1 citation) | |
| Vary perturbation size relative to the HiPPO matrix | sCIFAR · test accuracy | Mixed | Robustifying State-space Models for Long Sequences via Approximate Diagonalization (ICLR 2024 · 1 citation) | |
| Use a deep fully-connected decoder in place of the RI-GAMP-inspired decoder | Sparse Gaussian compression · MSE | No clear effect | Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth (ICML 2024 · 1 citation) | |
| vary the activation scaling parameter c | CT reconstruction · PSNR | Mixed | ReLUs Are Sufficient for Learning Implicit Neural Representations (ICML 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 37 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- OpenWebText · 8 of 37 papers
- ImageNet · 3 of 37 papers
- RAVEL · 2 of 37 papers
- COCO · 2 of 37 papers
- RedPajama · 2 of 37 papers
Baselines
- TopK SAE · 10 of 37 papers
- ReLU SAE · 7 of 37 papers
- JumpReLU SAE · 5 of 37 papers
- BatchTopK SAE · 5 of 37 papers
- Matryoshka SAE · 5 of 37 papers
- Gated SAE · 5 of 37 papers
- SIREN · 2 of 37 papers
- ReLU MLP · 2 of 37 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 84 papers; from OpenAlex and the papers' affiliations).
- Google DeepMind · 5 papers
- Google · 3 papers
- Tsinghua University · 2 papers
- Amazon · 2 papers
- The University of Adelaide · 2 papers
- University of California, Berkeley · 2 papers
- ETHZ - ETH Zurich · 2 papers
- Mohamed bin Zayed University of Artificial Intelligence · 2 papers
- Tohoku University · 2 papers
- MBZUAI · 2 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Symmetry and equivariance in loss landscapes (95 papers)
- Mechanistic circuit interpretability (93 papers)
- Width and depth of ReLU networks (53 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Differentiable optimization layers (125 papers)
- Scalable approximation and equivariant compute (120 papers)
- Non-autoregressive translation (102 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).