Research map / Learning theory and algorithms / Prediction and decision losses
Loss Functions and Operators Generated by f-Divergences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds convex Fenchel-Young losses from f-divergence regularizers with reference measures, introduces f-softargmax with a parallelizable bisection solver, and finds the alpha=1.5 divergence loss performs well on ImageNet and LM pretraining/SFT/distillation.
Problem
Standard cross-entropy/softargmax is tied to KL divergence with uniform reference; naive composition of f-divergences with softargmax is non-convex, and the effectiveness of alternative losses for language modeling is unclear.
Approach
Use f-divergences D_f(p, q) (with arbitrary reference measure q) as the regularizer of Fenchel-Young losses, giving convex losses whose associated operator (f-softargmax) is computed by a generic parallel bisection on a one-dimensional root equation.
Key results
- alpha=1.5 divergence loss reached 77.56% ImageNet accuracy vs 76.87% for KL (cross-entropy) with ResNet50.
- On XSum SFT with T5-base, alpha=1.5 loss achieved 14.27 ROUGE-2 vs 9.77 for KL.
- Distillation with alpha=1.5 loss reached 17.43 ROUGE-2 vs 16.64 for KL; NanoDO 1.2B pretraining next-token accuracy 48.70% vs 48.66% for KL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the α-divergence loss with α=1.5 instead of KL | ResNet50 · ImageNet-2012 validation | accuracy (%): 76.87 -> 77.56 | Helped |
| Use Chi-square loss instead of KL | ResNet50 · ImageNet-2012 validation | accuracy (%): 76.87 -> 76.06 | Hurt |
| Use Jensen–Shannon loss instead of KL | ResNet50 · ImageNet-2012 validation | accuracy (%): 76.87 -> 72.24 | Hurt |
| Use Squared Hellinger loss instead of KL | ResNet50 · ImageNet-2012 validation | accuracy (%): 76.87 -> 72.81 | Hurt |
| Use the α-divergence loss with α=1.5 instead of KL | NanoDO 1.2B · C4 validation next-token prediction | accuracy (%): 48.66 -> 48.70 | Helped |
| Use the α-divergence loss with α=1.5 instead of KL | T5-base SFT · XSum SFT | ROUGE-2: 9.77 -> 14.27 | Helped |
| Use Chi-square loss instead of KL | T5-base SFT · XSum SFT | ROUGE-2: 9.77 -> 11.15 | Helped |
| Use the α-divergence loss with α=1.5 instead of KL | T5-base student with T5-XL teacher · XSum distillation | ROUGE-2: 16.64 -> 17.43 | Helped |
| Learn the reference measure q instead of using q=1 | NanoDO models on C4 · C4 pretraining | performance | No clear effect |
| Decode with classical KL softargmax instead of the training-loss-associated f-softargmax | SFT language models · XSum summarization | ROUGE-2 | No clear effect |
Surprising findings
- Decoding with standard KL-based softargmax performs nearly identically to decoding with the f-softargmax matching the training loss.
- Jensen-Shannon and squared Hellinger losses underperform KL on ImageNet and the alpha=1.5 loss is the consistent best.
Limitations
- Learning the reference measure q did not improve performance on C4.
- Loss not necessarily convex in q; some f-entropies only defined on the relative interior of the simplex.
- Newer divergences (Jensen-Shannon, squared Hellinger) did not improve accuracy.
Experimental setup
Datasets: ImageNet-2012 C4 XSum
Benchmarks: ImageNet-2012 validation XSum (ROUGE-2) C4 next-token accuracy
Metrics: top-1 accuracy ROUGE-2 next-token prediction accuracy
Baselines: Kullback-Leibler (cross-entropy) Chi-square Jensen-Shannon Squared Hellinger reverse Chi-squared
Builds on: Fenchel-Young losses sparsemax entmax logistic loss f-divergences
Closest papers by meaning
- $f$-Divergence Based Classification: Beyond the Use of Cross-Entropy (ICML 2024 · similarity 0.87)
- Learning with Fitzpatrick Losses (NeurIPS 2024 · similarity 0.86)
- Cut Your Losses in Large-Vocabulary Language Models (ICLR 2025 · similarity 0.86)
- Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · similarity 0.85)
- Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules (NeurIPS 2025 · similarity 0.84)
- Multiclass Loss Geometry Matters for Generalization of Gradient Descent in Separable Classification (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.