Research map / Probabilistic and scientific modeling / Variational and generative sampling
Variational Smoothing in Recurrent Neural Network Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use variational smoothing instead of data noising | LSTM language models · PTB and Wikitext-2 | perplexity | Helped |
| Tie input and output embedding matrices (ablation) | Vanilla LSTM · PTB and Wikitext-2 | perplexity: PTB test: 84.5 -> 74.0; Wikitext-2 test: 100.8 -> 86.6 | Helped |
| Use Kneser-Ney rather than linear interpolation smoothing (ablation) | Variational smoothing · PTB | perplexity: 512 test: 71.2 -> 70.6; 1024 test: 67.8 -> 66.9 | Helped |
| Use Kneser-Ney rather than linear interpolation smoothing (ablation) | Variational smoothing · Wikitext-2 | perplexity: 512 test: 79.6 -> 80.9; 1024 test: 76.6 -> 76.7 | Hurt |
| Use element-wise rather than per-vector smoothing (ablation) | Variational smoothing · PTB | perplexity: 512 test: 70.6 -> 66.8; 1024 test: 66.9 -> 64.5 | Helped |
| Sample weights at test time instead of using the variational mean (ablation) | VS: Kneser-Ney · PTB validation | perplexity: 74.5 -> 89.8; 70.8 -> 78.7 | Hurt |
| Sample weights at test time instead of using the unnoised prediction (ablation) | DN: Kneser-Ney · PTB validation | perplexity: 75.0 -> 96.5; 71.4 -> 85.1 | Hurt |
| Increase the noise rate γ beyond the reported effective range (ablation) | Variational smoothing · PTB validation | perplexity | Hurt |
| Use element-wise embedding smoothing (ablation) | Variational smoothing · PTB training | training time: about one day; a couple hours | Hurt |
Closest papers by meaning
- BARNN: A Bayesian Autoregressive and Recurrent Neural Network (ICML 2025 · similarity 0.87)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.86)
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising (NeurIPS 2026 · similarity 0.85)
- Training-Free Bayesianization for Low-Rank Adapters of Large Language Models (NeurIPS 2025 · similarity 0.85)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.