Research map / Architectures and efficiency / RNNs and normalization
Regularizing and Optimizing LSTM Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a continuous cache pointer | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 52.8 | Helped |
| Add a continuous cache pointer | AWD-LSTM (tied) · WikiText-2 test | perplexity: 65.8 -> 52.0 | Helped |
| Remove fine-tuning (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 58.8 | Hurt |
| Remove fine-tuning (ablation) | AWD-LSTM (tied) · WikiText-2 test | perplexity: 65.8 -> 66.0 | Hurt |
| Remove NT-ASGD (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 63.7 | Hurt |
| Remove NT-ASGD (ablation) | AWD-LSTM (tied) · WikiText-2 test | perplexity: 65.8 -> 69.7 | Hurt |
| Use fixed rather than variable sequence lengths (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 58.9 | Hurt |
| Remove embedding dropout (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 62.7 | Hurt |
| Remove weight decay (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 61.0 | Hurt |
| Remove AR/TAR (ablation) | AWD-LSTM (tied) · WikiText-2 test | perplexity: 65.8 -> 70.1 | Hurt |
| Use full-sized embedding (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 65.6 | Hurt |
| Remove weight-dropping (ablation) | AWD-LSTM (tied) · Penn Treebank test | perplexity: 57.3 -> 68.9 | Hurt |
Closest papers by meaning
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.88)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.87)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.87)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.87)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.