Research map / Large language models / Language, safety and interpretability
Data Noising as Smoothing in Neural Network Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add bigram Kneser-Ney noising | 2-layer LSTM language model, medium (512 hidden size) · Penn Treebank test | perplexity: 80.4 -> 76.9 | Helped |
| Add bigram Kneser-Ney noising | 2-layer LSTM language model, large (1500 hidden size) · Penn Treebank test | perplexity: 77.5 -> 73.4 | Helped |
| Replace no noising with blank-token noising (ablation) | 2-layer LSTM language model · Text8 test | perplexity: 123.6 -> 110.7 | Helped |
| Replace no noising with unigram noising (ablation) | 2-layer LSTM language model · Text8 test | perplexity: 123.6 -> 111.3 | Helped |
| Add bigram Kneser-Ney noising | 2-layer LSTM encoder-decoder with attention · IWSLT 2015 English-German tst2014 | BLEU: 24.6 -> 26.0 (+1.4) | Helped |
| Add blank noising (ablation) | 2-layer LSTM encoder-decoder with attention · IWSLT 2015 English-German tst2014 | BLEU: +0.7 | Helped |
| Add unigram noising (ablation) | 2-layer LSTM encoder-decoder with attention · IWSLT 2015 English-German tst2014 | BLEU: +0.9 | Helped |
| Noise only the source sequence rather than both source and target (ablation) | Bigram Kneser-Ney noising in the encoder-decoder · IWSLT 2015 English-German tst2014 | BLEU: 24.6 -> 24.8 (+0.2) | Helped |
| Noise only the target sequence rather than both source and target (ablation) | Bigram Kneser-Ney noising in the encoder-decoder · IWSLT 2015 English-German tst2014 | BLEU: 24.6 -> 25.6 (+1.0) | Helped |
| Use discount-scaled adaptive noising probability instead of fixed noising probability (ablation) | Bigram noising on Penn Treebank · Penn Treebank validation | perplexity | Helped |
| Replace unigram proposal distribution with continuation-count proposal and noise target tokens (ablation) | Noising scheme for language modeling · Penn Treebank | perplexity: 79.4 -> 76.2 | Helped |
Closest papers by meaning
- Variational Smoothing in Recurrent Neural Network Language Models (ICLR 2019 · similarity 0.92)
- Noisin: Unbiased Regularization for Recurrent Neural Networks (ICML 2018 · similarity 0.89)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.84)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.84)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.84)
- Non-Vacuous Generalization Bounds for Large Language Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.