Research map / Image generation / Fast diffusion sampling
Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Error Norm Truncation (ENT), which drops tokens whose l2 error norm between predicted distribution and one-hot target is high, to make MLE training robust to noisy data.
Problem
MLE training for text generation is vulnerable to noisy training data, and existing loss-based truncation methods estimate data quality using only the target-token probability, ignoring the non-target token distribution and being sensitive to training iteration.
Approach
Estimate token quality with the l2 norm of the difference between the model's predicted distribution and the one-hot ground truth (error norm), and hard- or threshold-truncate tokens with high error norm during training, which separates noisy from clean tokens better than NLL and is less sensitive to when truncation starts.
Key results
- On WikiText-103 with GPT2-large, ENT-Threshold lowers validation perplexity from 25.88 (MLE) to 24.30.
- With 50% untranslated noise on opus-100 En-Fr, ENT-Fraction reaches 32.4 BLEU versus 28.6 for MLE, and beats the best of Loss Truncation and TaiLr by 2.1 BLEU.
- Multilingual opus-100 8-direction MT: ENT-Threshold averages 30.6 BLEU versus 30.1 for MLE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ENT-Threshold instead of MLE training | GPT2-large · WikiText-103 validation | perplexity: 25.88 -> 24.30 | Helped |
| Use ENT-Fraction instead of MLE training | Transformer multilingual MT · opus-100, average across directions | BLEU: 30.1 -> 30.3 | Helped |
| Use ENT-Fraction instead of MLE fine-tuning | T5-small · CNN/Daily Mail | ROUGE-1: 42.19 -> 42.63 | Helped |
| Use ENT-Fraction instead of MLE fine-tuning | BART-base · CNN/Daily Mail | ROUGE-1: 43.50 -> 43.48 | Hurt |
| Use ENT-Threshold instead of MLE training | Transformer · ParaCrawl En-Cs, WMT22 test | BLEU: 25.2 -> 25.7 | Helped |
| Use ENT-Fraction instead of MLE training | Transformer · ParaCrawl En-Ru, WMT22 test | BLEU: 24.6 -> 25.5 | Helped |
| Use ENT-Fraction instead of the strongest listed baseline | Transformer · ParaCrawl En-Zh, WMT22 test | BLEU: 13.2 -> 13.1 | Hurt |
| Use ENT-Fraction instead of MLE training | Transformer · opus-100 En-Fr with 50% untranslated noise | BLEU: 28.6 -> 32.4 | Helped |
| Use ENT-Fraction instead of MLE training | Transformer · opus-100 En-Fr with 50% misordered-word noise | BLEU: 35.5 -> 36.4 | Helped |
| Use ENT-Threshold instead of MLE training | Transformer multilingual MT · opus-100 En-Gl, T=1 | BLEU: 27.4 -> 28.1 | Helped |
| Use ENT-Threshold instead of MLE training | Transformer multilingual MT · opus-100 En-Fr, T=1 | BLEU: 38.1 -> 38.2 | Helped |
| Use ENT-Threshold instead of MLE training | Transformer multilingual MT · opus-100 En-Gl, T=100 | BLEU: 27.9 -> 28.5 | Helped |
Surprising findings
- On CNN/Daily Mail, TaiLr underperforms MLE on both T5-small and BART-base, and ENT-Fraction only matches MLE on BART-base.
- Loss Truncation performs at or below MLE in several multilingual MT directions, and MLE is stronger than truncation methods under mild noise levels.
Limitations
- Improvements are attributed to noisy training data, so gains on clean, curated data may be smaller.
- Hyperparameters (truncation fraction or threshold) must be tuned, and coarse-grained or dataset-level quality estimation is left to future work.
Experimental setup
Datasets: opus-100 WikiText-103 CNN/Daily Mail ParaCrawl WMT22
Benchmarks: opus-100 WikiText-103 CNN/Daily Mail ParaCrawl WMT22
Metrics: SacreBLEU perplexity ROUGE-1/2/LSum
Baselines: MLE Loss Truncation TaiLr
Builds on: Loss Truncation TaiLr Influence functions
Closest papers by meaning
- Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.88)
- Fewer Truncations Improve Language Modeling (ICML 2024 · similarity 0.87)
- Learning from others' mistakes: Finetuning machine translation models with span-level error annotations (ICML 2025 · similarity 0.86)
- How to Synthesize Text Data without Model Collapse? (ICML 2025 · similarity 0.86)
- Improving Open-Ended Text Generation via Adaptive Decoding (ICML 2024 · similarity 0.86)
- Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.