Research map / Label-efficient and robust learning / Recognition and long-tail
$f$-Divergence Based Classification: Beyond the Use of Cross-Entropy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes posterior-estimation classifiers from f-divergence variational bounds and a bottom-up design, yielding a new shifted log (SL) divergence whose objective replaces cross-entropy for classification.
Problem
Classification is usually trained by minimizing cross-entropy (KL divergence); the paper asks whether other f-divergence-based objectives can estimate the MAP posterior better for classification with a single training optimization.
Approach
Estimate the posterior via a discriminator trained with variational f-divergence objectives (single maximization, no min-max), plus a bottom-up objective design that yields a new shifted log (SL) f-divergence whose generator is -log(u+|Tx|) up to constants.
Key results
- On CIFAR10 with ResNet, SL reaches 95.43% vs 95.39% for cross-entropy; with MobileNet, SL reaches 93.89% vs 92.59% for CE.
- On CIFAR100 with MobileNet, SL reaches 74.78% vs 72.61% for CE, while ResNet is the one case where CE (78.29%) is best.
- SL attains the highest accuracy in almost all tested image settings, and the Pearson chi-squared divergence performs worst in most scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CE with SL | Tiny · Fashion MNIST | accuracy: 91.64 \pm 0.09 -> 91.83 \pm 0.02 | Helped |
| Replace CE with SL | Tiny · CIFAR10 | accuracy: 70.13 \pm 0.05 -> 70.87 \pm 0.26 | Helped |
| Replace GAN with SL | VGG · CIFAR10 | accuracy: 93.75 \pm 0.04 -> 93.93 \pm 0.08 | Helped |
| Replace CE with SL | DLA · CIFAR10 | accuracy: 95.04 \pm 0.02 -> 95.31 \pm 0.09 | Helped |
| Replace CE with SL | ResNet · CIFAR10 | accuracy: 95.39 \pm 0.04 -> 95.43 \pm 0.04 | No clear effect |
| Replace CE with SL | MobileNet · CIFAR10 | accuracy: 92.59 \pm 0.13 -> 93.89 \pm 0.15 | Helped |
| Replace HD with SL | VGG · CIFAR100 | accuracy: 73.51 \pm 0.03 -> 73.61 \pm 0.05 | No clear effect |
| Replace HD with SL | DLA · CIFAR100 | accuracy: 78.63 \pm 0.14 -> 78.65 \pm 0.01 | No clear effect |
| Replace CE with SL | ResNet · CIFAR100 | accuracy: 78.29 \pm 0.18 -> 78.03 \pm 0.04 | Hurt |
| Replace HD with SL | MobileNet · CIFAR100 | accuracy: 73.00 \pm 0.30 -> 74.78 \pm 0.23 | Helped |
| Replace CE with SL | Tiny · MNIST | accuracy: 99.08 \pm 0.06 -> 99.03 \pm 0.04 | No clear effect |
Surprising findings
- Pearson chi-squared performs worst in almost all image scenarios, and on CIFAR100 VGG the GAN objective drops well below CE.
Limitations
- Gains over cross-entropy are small (often under 1-2 points) and results are single-setting image benchmarks with small networks.
- Theoretical convergence guarantee (Lemma) holds only in the nonparametric limit with global optimum assumed.
Experimental setup
Datasets: MNIST Fashion MNIST CIFAR10 CIFAR100
Benchmarks: MNIST Fashion MNIST CIFAR10 CIFAR100 4-PAM symbol error rate
Metrics: classification accuracy symbol error rate
Baselines: cross-entropy reverse KL squared Hellinger GAN divergence Pearson chi-squared maxL MAP Genie
Builds on: f-divergence variational representation GAN divergence objective proper losses
Closest papers by meaning
- When Optimizing $f$-Divergence is Robust with Label Noise (ICLR 2021 · similarity 0.87)
- Loss Functions and Operators Generated by f-Divergences (ICML 2025 · similarity 0.87)
- On $f$-Divergence Principled Domain Adaptation: An Improved Framework (NeurIPS 2024 · similarity 0.85)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.84)
- Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.83)
- Bounds on $L_p$ Errors in Density Ratio Estimation via $f$-Divergence Loss Functions (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.