Research map / Label-efficient and robust learning / Recognition and long-tail

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Learning from unlabeled and positive-unlabeled data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Generative Cross-Entropy (GenCE), a drop-in classification loss that rewrites the class-conditional likelihood via Bayes' rule and normalizes each sample's softmax score against in-batch predictions, and proves it is strictly proper under a completeness condition.

Problem

Cross-entropy's sample efficiency is limited when labels are scarce; existing remedies act on data rather than on the training objective. The paper asks whether a discriminative network can incorporate the generative learning principle through the loss alone.

Approach

Apply Bayes' theorem to the class-conditional likelihood, approximate the marginal p(y) by a mini-batch average, yielding a non-local loss that normalizes each sample's class score by the sum of predicted class-y probabilities over the batch; extend proper-scoring-rule theory to non-local rules and prove strict properness under a completeness condition.

Key results

• On balanced small-data subsets, GenCE beats CE in all 12 configurations, e.g. +9.5 points on CIFAR-100 VGG at N=2000 (24.40 vs 14.92).
• On class-imbalanced CIFAR-100 with rho=100 (ResNet-50), GenCE reaches 40.31% vs 38.42% for CE.
• GenCE attains the lowest ECE in five of six small-data configurations, e.g. 13.24% vs 23.00% CE on Mini-ImageNet N=5000; Far-OOD entropy AUROC 94.87 vs 88.40 for CE.

Surprising findings

• Calibration gains are attributed to the batch normalization dividing out uniform confidence inflation, rather than an explicit regularizer.
• Label smoothing applied on top of GenCE has only a marginal effect.

Limitations

• Accuracy depends on how well each mini-batch reflects the class marginal; rare-class regimes may need class-stratified sampling.
• Strict-properness guarantee is population-level only; no finite-sample analysis is given.
• Evaluated only on image classification with CNN backbones trained from scratch.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Mini-ImageNet SVHN

Benchmarks: CIFAR-10 CIFAR-100 Mini-ImageNet SVHN

Metrics: top-1 accuracy ECE AUROC

Baselines: cross-entropy GCE focal loss MAE Brier loss SupCon label smoothing

Builds on: cross-entropy proper scoring rules focal loss generalized cross-entropy mixup

Closest papers by meaning

• Generative Classifiers Avoid Shortcut Solutions (ICLR 2025 · similarity 0.88)
• Provably Reliable Classifier Guidance via Cross-Entropy Control (NeurIPS 2026 · similarity 0.87)
• Cross-Entropy Is All You Need To Invert the Data Generating Process (ICLR 2025 · similarity 0.86)
• The Entropy Enigma: Success and Failure of Entropy Minimization (ICML 2024 · similarity 0.86)
• A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.86)
• Cut Your Losses in Large-Vocabulary Language Models (ICLR 2025 · similarity 0.85)