Research map / Large language models / Language, safety and interpretability
Controlling Global Statistics in Recurrent Neural Network Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic KL regularization with repetition constraints | RNNLM · PTB repetition k=1 | absolute frequency mismatch: 7.07k -> 3.00k | Helped |
| add dynamic KL regularization with repetition constraints | RNNLM · PTB | test perplexity: 77.6 -> 78.3 | No clear effect |
| replace sampled-marginal dynamic estimate with static (model output distribution) regularizer (ablation) | dynamic KL regularization · PTB repetition k=1 | absolute frequency mismatch: 3.00k -> 8.08k | Hurt |
| drop the condition-distribution KL term (dynamic-P(C)) (ablation) | dynamic KL regularization · PTB repetition k=3 | absolute frequency mismatch: 3.89k -> 4.67k | Hurt |
| add dynamic KL regularization with bigram constraints | RNNLM · PTB unigram frequency | absolute frequency mismatch: 163k -> 125k | Helped |
| drop the condition-distribution KL term (ablation) | dynamic KL regularization · PTB bigram constraints, unigram | absolute frequency mismatch: 125k -> 147k | Hurt |
| use bigram statistics from larger WikiText-103 rather than WikiText-2 (ablation) | dynamic KL regularization · WikiText-2 | test perplexity: 91.4 -> 86.8 | Helped |
| add dynamic KL regularization with WikiText-2 own statistics | RNNLM · WikiText-2 | test perplexity: 91.6 -> 91.4 | No clear effect |
| add dynamic KL regularization with repetition constraints | definition model · WordNet greedy | BLEU: 18.5 -> 19.1 | Helped |
| add dynamic KL regularization with repetition constraints | definition model · WordNet greedy k=1 | absolute frequency mismatch: 10.73k -> 0.72k | Helped |
| train with near-greedy generation (temperature 0.1) for the sampled text (ablation) | dynamic KL regularization · WordNet greedy k=3 | absolute frequency mismatch: 14.37k -> 8.87k | Helped |
| train with near-greedy generation (temperature 0.1) for the sampled text (ablation) | dynamic KL regularization · WordNet greedy | BLEU: 19.1 -> 18.9 | Hurt |
Closest papers by meaning
- Tailoring Language Generation Models under Total Variation Distance (ICLR 2023 · similarity 0.88)
- Differentiated Distribution Recovery for Neural Text Generation (AAAI 2019 · similarity 0.88)
- Tractable Control for Autoregressive Language Generation (ICML 2023 · similarity 0.87)
- Recurrent Hierarchical Topic-Guided RNN for Language Generation (ICML 2020 · similarity 0.87)
- Noisin: Unbiased Regularization for Recurrent Neural Networks (ICML 2018 · similarity 0.87)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.