Research map / Large language models / Language, safety and interpretability
Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, Jian Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with DITTO pseudo-repetition penalization instead of plain MLE | 16-layer Transformer on Wikitext-103 · Wikitext-103 test, greedy decoding | MAUVE: 0.34 -> 0.77 | Improved |
| fine-tune with DITTO instead of MLE | 16-layer Transformer on Wikitext-103 · Wikitext-103 test, greedy decoding | Perplexity: 25.68 -> 24.33 | Improved |
| fine-tune with DITTO instead of MLE | 16-layer Transformer on Wikitext-103 · Wikitext-103 test, greedy decoding | Repetition-Sen: 14.50 -> 2.85 | Improved |
| DITTO versus UL-token+seq on repetition rate | 16-layer Transformer on Wikitext-103 · Wikitext-103 test, greedy decoding | Repetition-4: 7.60 -> 22.00 | Hurt |
| fine-tune with DITTO instead of MLE under top-k and nucleus sampling | 16-layer Transformer on Wikitext-103 · Wikitext-103 test, nucleus sampling | MAUVE: 0.94 -> 0.96 | Improved |
| fine-tune BART-large with DITTO instead of MLE | BART-large · CNN/DailyMail | ROUGE-1: 44.11 -> 44.41 | Improved |
| replace DITTO loss with MSE loss (ablation) | DITTO · Wikitext-103 test, greedy decoding | MAUVE: 0.77 -> 0.73 | Hurt |
| replace DITTO loss with margin (hinge) loss (ablation) | DITTO · Wikitext-103 test, greedy decoding | MAUVE: 0.77 -> 0.66 | Hurt |
| repeat two-sentence groups instead of one sentence for pseudo data (ablation) | DITTO · Wikitext-103 test, greedy decoding | MAUVE: 0.77 -> 0.77 | No clear effect |
| shorten pseudo data to 150 tokens (DITTO-short) (ablation) | DITTO · Wikitext-103 self-reinforcement (TP/IP/WR) | TP, IP, WR | Hurt |
| mix pseudo and real data at gamma=0.5 (versus other ratios) (ablation) | DITTO · Wikitext-103 and CNN/DailyMail | generation quality | Improved |
| stronger penalty lambda=0.5 rather than 0.9 (ablation) | DITTO · Wikitext-103 vs CNN/DailyMail | generation quality | Mixed |
Closest papers by meaning
- Understanding In-Context Learning from Repetitions (ICLR 2024 · similarity 0.87)
- Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.87)
- Interpreting the Repeated Token Phenomenon in Large Language Models (ICML 2025 · similarity 0.87)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.86)
- From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers (ICML 2024 · similarity 0.86)
- Reflection-Window Decoding: Text Generation with Selective Refinement (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
