Research map / Vision-language and multimodal / Multimodal LLMs
Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| drop attention to repetitive n-grams during training | GPT-2 · Wikitext-103, FreeLaw, and OpenWebText2 | Rep-2 (%): 47.05 -> 9.78; 51.74 -> 10.15; 73.96 -> 25.24 | Helped |
| drop attention to repetitive n-grams during training | GPT-2 · Wikitext-103, FreeLaw, and OpenWebText2 | PPL: 21.98 -> 28.26; 16.13 -> 20.68; 80.37 -> 107.02 | Hurt |
| drop attention to random tokens instead of repetitive tokens (ablation) | GPT-2 + Rep-Dropout · Wikitext-103 | Rep-2 (%): 9.78 -> 38.33 | Hurt |
| drop attention to random tokens instead of repetitive tokens (ablation) | GPT-2 + Rep-Dropout · FreeLaw and OpenWebText2 | Rep-2 (%): 10.15 -> 38.82; 25.24 -> 63.43 | Hurt |
| increase the repetition dropout rate (ablation) | GPT-2 + Rep-Dropout · Wikitext-103 | Rep-2 | Helped |
| increase the repetition dropout rate (ablation) | GPT-2 + Rep-Dropout · Wikitext-103 | PPL | Hurt |
| merge only repetitive high-inflow pairs rather than all high-inflow pairs (ablation) | HI-Re · Wikitext-103 | Rep-2 (%): 41.91 -> 43.62 | Helped |
| merge a random subset of high-inflow pairs instead of repetitive high-inflow pairs (ablation) | HI-Re · Wikitext-103 | Rep-2 (%): 43.62 -> 52.41 | Hurt |
| apply ScaleGrad only to repetitive words in the prefix (ablation) | ScaleGrad · Wikitext-103 | Rep-2 (%): 12.49 -> 17.53 | Hurt |
| apply ScaleGrad to a random prefix subset matched in size (ablation) | ScaleGrad · Wikitext-103 | Rep-2 (%): 17.53 -> 22.97 | Hurt |
| fine-tune on Alpaca rather than leave Llama2 untuned | Llama2-7B · Wikitext-103 | Rep-2 (%): 47.79 -> 15.08 | Helped |
| include WT-103 50K in instruction-tuning data | Llama2-7B · Wikitext-103 | Rep-2 (%): 15.08 -> 41.63; 15.08 -> 54.10 | Hurt |
Closest papers by meaning
- Interpreting the Repeated Token Phenomenon in Large Language Models (ICML 2025 · similarity 0.87)
- The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.86)
- Understanding In-Context Learning from Repetitions (ICLR 2024 · similarity 0.85)
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.85)
- Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.84)
- Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.