Research map / Architectures and efficiency / RNNs and normalization
Breaking the Activation Function Bottleneck through Adaptive Parameterization
Sebastian Flennerhag, Hujun Yin, John Keane, Mark Elliot
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive parameterization with SVA | Logistic Regression on MNIST · MNIST test | accuracy: 92.14% -> 93.86% | Improved |
| Use 3-layer SVA instead of a 3-layer feed-forward network | Feed-forward model · MNIST test | accuracy: 97.01% -> 97.65% | Improved |
| Use recurrent adaptive parameterization | LSTM on Penn Treebank · Penn Treebank test | perplexity: 68.9 -> 55.3 | Improved |
| Use recurrent adaptive parameterization | AWD-LSTM on Penn Treebank · Penn Treebank test | perplexity: 57.3 -> 55.3 | Improved |
| Use recurrent adaptive parameterization | AWD-LSTM on WikiText-2 · WikiText-2 test | perplexity: 65.8 -> 64.5 | Improved |
| Add recurrent adaptation instead of feed-forward adaptation (ablation) | aLSTM with output adaptation · Penn Treebank test | perplexity: 63.1 -> 58.2 | Improved |
| Add recurrent adaptation instead of feed-forward adaptation (ablation) | aLSTM with IO adaptation · Penn Treebank test | perplexity: 59.1 -> 56.9 | Improved |
| Use IO adaptation instead of output adaptation (ablation) | aLSTM with feed-forward adaptation · Penn Treebank test | perplexity: 63.1 -> 59.1 | Improved |
| Use IO adaptation instead of output adaptation (ablation) | aLSTM with recurrent adaptation · Penn Treebank test | perplexity: 58.2 -> 56.9 | Improved |
| Use LSTM-RHN adaptation instead of LSTM adaptation (ablation) | aLSTM with output adaptation · Penn Treebank test | perplexity: 58.2 -> 57.3 | Improved |
| Use LSTM-RHN adaptation instead of LSTM adaptation (ablation) | aLSTM with IO adaptation · Penn Treebank test | perplexity: 56.9 -> 56.5 | Improved |
Closest papers by meaning
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.85)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.84)
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.84)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.84)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.84)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
