Research map / Label-efficient and robust learning / Recognition and long-tail
Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Calibration-aware Semantic Mixing (CSM), which uses class-conditional diffusion (EDM) to generate semantically mixed images with soft labels, reannotates mixing ratios via CLIP-prototype projection, and trains with an L2 loss for balanced confidence calibration.
Problem
Standard calibration methods rely on one-hot labels that assume full certainty, and mixup-based soft-label augmentation produces low-fidelity images and misaligned mixing coefficients, hurting calibration.
Approach
Generate high-fidelity semantic mixtures by conditioning a diffusion model on a shared noise latent with interpolated class embeddings, reannotate soft labels by projecting CLIP features onto class prototypes, and use an L2 loss that the paper argues is confidence-balanced for soft labels.
Key results
- On CIFAR-100 with ResNet-50, CSM (L2) reaches ECE 1.29 and AECE 1.63 versus 12.94 for plain CE, with ACC 78.84.
- On CIFAR-10 with ResNet-50, CSM achieves ACC 95.79 and ECE 0.54, outperforming RankMixup (ECE 2.59) and FCL (ECE 0.76).
- On ImageNet with ResNet-50, CSM achieves ECE 1.32 versus 3.93 for RankMixup and 1.46 for CALS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CSM instead of the strongest listed ECE baseline, FCL | ResNet-50 · CIFAR-10 | ECE: 0.76(0.76) -> 0.54(0.54) | Helped |
| use CSM instead of the strongest listed ECE baseline, FL | ResNet-50 · CIFAR-100 | ECE: 2.83(1.66) -> 1.29(1.29) | Helped |
| use CSM instead of RankMixup | ResNet-50 · ImageNet | ECE: 3.93 -> 1.32 | Helped |
| use CSM instead of CALS | Swin-Transformer-V2 · ImageNet | ECE: 1.61 -> 1.49 | Helped |
| remove calibrated reannotation (ablation) | CSM (L2), ResNet-50 · CIFAR-100 | ECE: 1.29 -> 5.20 | Hurt |
| replace calibrated reannotation with uncalibrated CLIP labels (ablation) | CSM (L2), ResNet-50 · CIFAR-100 | ECE: 1.29 -> 52.78 | Hurt |
| replace soft augmented labels with one-hot labels (ablation) | CSM (L2), ResNet-50 · CIFAR-100 | ECE: 1.29 -> 10.84 | Hurt |
| replace L2 loss with CE loss (ablation) | CSM, ResNet-50 · CIFAR-100 | ECE: 1.29 -> 2.47 | Hurt |
| replace L2 loss with focal loss (ablation) | CSM, ResNet-50 · CIFAR-100 | ECE: 1.29 -> 2.43 | Hurt |
| replace semantic-mixing augmentation with pixel-space Mixup and use CE loss (ablation) | calibration training, ResNet-50 · CIFAR-100 | ECE: 1.29 -> 8.68 | Hurt |
| replace semantic-mixing augmentation with pixel-space Mixup and use focal loss (ablation) | calibration training, ResNet-50 · CIFAR-100 | ECE: 1.29 -> 2.45 | Hurt |
| replace semantic-mixing augmentation with pixel-space Mixup and use L2 loss (ablation) | calibration training, ResNet-50 · CIFAR-100 | ECE: 1.29 -> 2.49 | Hurt |
Surprising findings
- Using CLIP labels directly for the augmented samples without the reannotation step leads to very poor calibration (ECE 52.78 on CIFAR-100).
- Mixup with Focal loss or L2 loss attains ECE around 2.5 on CIFAR-100 while CSM with CE or FL loss gives different balance, indicating the choice of loss matters substantially for soft-labeled augmentations.
Limitations
- Augmented sample generation and training require substantial additional time relative to plain training, with the number of augmented samples per batch being the main cost factor.
- The theoretical assumption of an optimal encoder is relaxed in practice, and the simplification to a single scaling factor s is a heuristic tuned on validation sets.
- Some results are mixed: CSM is not best on CIFAR-100 OOD detection against Tiny-ImageNet and comparable rather than superior on some settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet OOD AUROC on CIFAR-10/CIFAR-100/Tiny-ImageNet
Metrics: ACC ECE AECE post-temperature ECE AUROC
Baselines: CE MMCE ECP LS MbLS FL FLSD CPC FCL Mixup RegMixup AugMix
Builds on: Mixup RankMixup EDM CLIP classifier-free guidance
Closest papers by meaning
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.88)
- Divide et Calibra: Multiclass Local Calibration via Vector Quantization (NeurIPS 2026 · similarity 0.87)
- Tailoring Mixup to Data for Calibration (ICLR 2025 · similarity 0.87)
- LaSCal: Label-Shift Calibration without target labels (NeurIPS 2024 · similarity 0.87)
- Provable Uncertainty Decomposition via Higher-Order Calibration (ICLR 2025 · similarity 0.86)
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.