Research map / Architectures and efficiency / RNNs and normalization
On the State of the Art of Evaluation in Neural Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Control hyperparameters with black-box tuning when comparing recurrent architectures | LSTM, RHN, and NAS · Penn Treebank | test perplexity | Helped |
| Use a shallow LSTM rather than RHN or NAS | 10M recurrent models · Wikitext-2 | test perplexity: 4.235 vs 4.376 (RHN) and 4.329 (NAS) | Helped |
| Use NAS rather than LSTM or RHN | 27M and 46M recurrent models · Enwik8 | test BPC: 0.969 vs 0.908; 0.925 vs 0.898 | Hurt |
| Add down-projection (ablation) | LSTM and RHN models · Penn Treebank | perplexity: about 2--5 perplexity points | Helped |
| Add down-projection (ablation) | LSTM and RHN models · Wikitext-2 | perplexity: about 10--18 points | Helped |
| Share input and output embeddings (ablation) | LSTM and RHN · Penn Treebank | perplexity: about 6 points | Helped |
| Add variational dropout (ablation) | RHN · Penn Treebank | perplexity: 4.151 -> 4.272; 4.131 -> 4.273 | Helped |
| Add variational dropout (ablation) | deep LSTM · Penn Treebank | perplexity: 4.065 -> 4.166 | Helped |
| Replace variational dropout with recurrent dropout (ablation) | LSTM and RHN · Penn Treebank | perplexity: LSTM: 4.088 -> 4.105; 4.065 -> 4.142; RHN: 4.151 -> 4.143; 4.131 -> 4.111 | No clear effect |
| Tie input and forget gates (ablation) | LSTM · Penn Treebank | perplexity: 4.088 -> 4.088; 4.065 -> 4.060 | No clear effect |
| Use bounded cell states through capped input gates (ablation) | deep LSTM · Penn Treebank | perplexity: 4.065 vs 4.116 | Helped |
| Tune hyperparameters on Wikitext-2 instead of reusing Penn Treebank settings | 10M LSTM · Wikitext-2 | test perplexity: 4.421 -> 4.235 | Helped |
Closest papers by meaning
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.88)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.87)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.