Research map / Large language models / Language, safety and interpretability
Language Generation with Strictly Proper Scoring Rules
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts non-local strictly proper scoring rules (Brier and spherical) to token-level language generation losses, with a score smoothing technique, showing gains when fine-tuning pretrained MLE models and when instruction-tuning LLaMA models.
Problem
Maximum likelihood (log score) is the default loss for language generation, but the log score is the only local strictly proper score, and its unboundedness is criticized; it is unclear whether non-local strictly proper scoring rules can be used to train language generation models.
Approach
Distribute any strictly proper scoring rule at the token level over autoregressive conditional distributions, so that per-token propriety yields a calibrated sequence model, and introduce score smoothing (plus a masked logarithmic term) to support label smoothing with arbitrary scoring rules.
Key results
- Fine-tuning a log-score Transformer on WMT14 En-De with the Brier score raises BLEU from 27.61 to 28.01 and with the spherical score to 28.07.
- Instruction-tuning LLaMA-7B with the Brier score raises average WMT22 BLEU by over 3 points (e.g., EN-DE 25.42 to 29.15).
- Training from scratch, the logarithmic score converges fastest and performs best; the Brier score is the slowest and lowest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with Brier score instead of logarithmic score | Transformer · WMT14 En-De test | BLEU: 27.61 -> 28.01 | Helped |
| fine-tune with spherical score instead of logarithmic score | Transformer · WMT14 En-Fr test | BLEU: 41.92 -> 42.09 | Helped |
| fine-tune with Brier score instead of logarithmic score | Transformer · CNN/DailyMail | ROUGE-1: 39.72 -> 40.20 | Helped |
| fine-tune with alternative strictly proper scores | Transformer · CNN/DailyMail | ROUGE-L: 36.41 -> 36.78; 36.41 -> 36.73 | Mixed |
| instruction-tune with Brier score instead of log-likelihood | LLaMA-7B · WMT22 EN-DE test | BLEU: 25.42 -> 29.15 | Helped |
| instruction-tune with spherical score instead of log-likelihood | LLaMA-7B · CNN/DailyMail | ROUGE-1: 28.66 -> 30.89 | Helped |
| use alpha=2 rather than other tested alpha values (ablation) | alpha-power and pseudo-spherical scoring rules · WMT14 En-De test | BLEU: 27.91; 28.05; 28.01; 27.84; 27.71; 28.09; 27.91; 28.07; 27.92; 27.64 | Mixed |
| apply score smoothing S^epsilon to models trained with label smoothing (ablation) | Brier score fine-tuning · TED test sets | Average BLEU: 33.63 -> 33.80 | Helped |
| add masked logarithmic score to score smoothing (ablation) | Brier score fine-tuning · TED test sets | Average BLEU: 33.80 -> 34.49 | Helped |
| apply score smoothing S^epsilon to models trained with label smoothing (ablation) | Spherical score fine-tuning · TED test sets | Average BLEU: 33.86 -> 33.89 | Helped |
| add masked logarithmic score to score smoothing (ablation) | Spherical score fine-tuning · TED test sets | Average BLEU: 33.89 -> 34.43 | Helped |
| fine-tune with unsmoothed alternative scores rather than retaining label smoothing (ablation) | Transformer w/ LS · TED test sets | Average BLEU: 33.91 -> 33.63; 33.91 -> 33.86 | Hurt |
Surprising findings
- Brier and spherical scores underperform the logarithmic score when training from scratch, yet improve models when used for fine-tuning after log-score pretraining.
- Fine-tuning with the logarithmic score lowers BLEU in the dynamics analysis, while Brier and spherical fine-tuning raises it, suggesting perplexity alone is an incomplete evaluation.
Limitations
- Non-log scores underperform the log score when training from scratch under log-score-tuned hyperparameters.
- Hyperparameters were borrowed from log-score works and not tuned per scoring rule.
- LLM summarization only evaluated for LLaMA-7B due to memory constraints.
Experimental setup
Datasets: WMT14 En-De WMT14 En-Fr TED bilingual CNN/DailyMail Alpaca
Benchmarks: WMT14 En-De WMT14 En-Fr TED CNN/DailyMail WMT22 MT-bench
Metrics: BLEU ROUGE-1 ROUGE-2 ROUGE-L
Baselines: Transformer (MLE) MixCE TaiLr Convex Transformer w/ LS
Builds on: label smoothing entmax loss MixCE TaiLr Convex loss
Closest papers by meaning
- Controlled Text Generation via Language Model Arithmetic (ICLR 2024 · similarity 0.87)
- Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo (ICLR 2025 · similarity 0.86)
- MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR 2024 · similarity 0.85)
- Generating in the Limit with Infinitely Many Hallucinations (NeurIPS 2026 · similarity 0.85)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.85)
- Model-Based Minimum Bayes Risk Decoding for Text Generation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.