Research map / Label-efficient and robust learning / Recognition and long-tail
Set Learning for Accurate and Calibrated Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes odd-k-out (OKO) set-learning, training a classifier to identify the pair class among sets of examples, which improves accuracy and calibration without post-hoc scaling.
Problem
Standard cross-entropy training on single examples yields overconfident, poorly calibrated classifiers, especially with few training examples and heavy-tailed class distributions, and calibration is usually fixed post hoc.
Approach
Replace single-example classification with an odd-one-out style set objective: sets of k+2 examples contain a pair from the same class plus k odd-class examples, and the network's logits are summed over the set during training, so at inference it classifies single examples with no architecture change.
Key results
- For 10-shot MNIST, OKO achieves an average test accuracy of 87.62%, improving the previously reported best by 8.59%.
- Averaged over training settings, OKO reaches 93.62% on uniform MNIST and 85.67% on heavy-tailed MNIST, versus 92.34% and 78.21% for vanilla cross-entropy.
- OKO attains lowest ECE across most training settings, with batch-balancing plus label smoothing slightly lower only on CIFAR-100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · MNIST, FashionMNIST, CIFAR-10, and CIFAR-100; uniform and heavy-tailed training | accuracy | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · MNIST uniform | accuracy: 92.34% -> 93.62% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · MNIST heavy-tailed | accuracy: 81.42% -> 85.67% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · FashionMNIST uniform | accuracy: 81.12% -> 81.49% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · FashionMNIST heavy-tailed | accuracy: 71.45% -> 74.02% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · CIFAR-10 uniform | accuracy: 55.87% -> 57.63% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · CIFAR-10 heavy-tailed | accuracy: 44.72% -> 44.95% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · CIFAR-100 uniform | accuracy: 32.72% -> 35.11% | Helped |
| Train with OKO set learning instead of single-example training | CNN and ResNet classifiers · CIFAR-100 heavy-tailed | accuracy: 13.96% -> 14.13% | Helped |
| Train with OKO instead of the compared calibration methods | CNN and ResNet classifiers · Balanced training settings | MAE between entropies and cross-entropies | Mixed |
| Train with OKO instead of the compared calibration methods | CNN and ResNet classifiers · Heavy-tailed training settings | MAE between entropies and cross-entropies | Mixed |
| Use the hard OKO loss instead of the soft OKO loss (ablation) | OKO · Ablation experiments | performance | Helped |
Surprising findings
- OKO yields well-calibrated predictions and smoothed logits even when trained with hard labels, without any temperature scaling or label smoothing.
- Hard OKO loss consistently outperformed the soft loss in ablations.
Limitations
- Classes are treated as semantically equally distant, as in standard cross-entropy.
- Developed only for supervised learning with labeled data.
- Theory relies on simplifying assumptions such as memorized, class-identical outputs for the loss landscape analysis.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST FashionMNIST CIFAR-10 CIFAR-100
Metrics: test accuracy expected calibration error mean absolute error between entropy and cross-entropy reliability diagrams
Baselines: Vanilla cross-entropy Label smoothing Focal Loss Weighted CE Batch-balancing BB + LS BB + temperature scaling
Builds on: label smoothing batch-balancing temperature scaling focal loss
Closest papers by meaning
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.84)
- MaxEnt Loss: Constrained Maximum Entropy for Calibration under Out-of-Distribution Shift (AAAI 2024 · similarity 0.84)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.84)
- The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.83)
- The Entropy Enigma: Success and Failure of Entropy Minimization (ICML 2024 · similarity 0.83)
- Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.