Research map / Label-efficient and robust learning / Recognition and long-tail

Towards Calibrated Multi-label Deep Neural Networks

CVPR 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Confidence calibration objectives · 18 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that asymmetric multi-label losses (focal, ASY) are not strictly proper and are poorly calibrated, and proposes the Strictly Proper Asymmetric (SPA) loss plus a Label Pair Regularizer (LPR) that improve calibration without hurting multi-label accuracy.

Problem

Multi-label DNNs trained with popular asymmetric losses (focal, ASY) produce poorly calibrated class probabilities, which harms downstream uses such as multi-label retrieval that combine several probability estimates.

Approach

Theoretically shows focal and ASY losses lack strict properness, then designs a composite loss with a custom inverse link that is both asymmetric and strictly proper, plus a pairwise label-pair BCE regularizer that adds calibration constraints per example.

Key results

• SPA+LPR reduces ACE on MS-COCO from 30.6 (ASY) to 4.2 (ECA-ResNet50-T) while keeping mAP@y at 77.7 vs 77.5 for ASY.
• On WIDER-A, SPA reduces ACE of BCE by about 4x (11.5 to 5.3 with ECA-ResNet50-T).
• In multi-label retrieval, SPA-trained networks outperform BCE and ASY on all four datasets, with the gap to ASY widening as query length grows.

Measured effects reported in the paper

Surprising findings

• Asymmetric losses that give higher multi-label accuracy (ASY) are substantially less calibrated than plain BCE, and temperature-like mapping phi only partly fixes this.
• BCE, though strictly proper, has poor mAP due to label imbalance.

Limitations

• Strict properness only guarantees calibration asymptotically; finite-data calibration needs the LPR regularizer.
• Theory assumes label independence; joint class probability estimation is left for future work.
• SPA hyperparameters are fixed across datasets without dataset-specific tuning, and TWL required temperature scaling since it is not a CPE loss.

Experimental setup

Datasets: MS-COCO PASCAL VOC 2012 WIDER Attribute MPI-I VISPR VQA v2.0

Benchmarks: MS-COCO VOC2012 WIDER-A VISPR VQA v2.0

Metrics: mAP@y mAP@x ACE MCE multi-label retrieval mAP

Baselines: BCE Focal loss ASY loss TWL Focal + phi ASY + phi

Builds on: asymmetric loss (ASY) focal loss binary cross-entropy temperature scaling two-way loss

Closest papers by meaning

• The Devil Is in the Margin: Margin-Based Label Smoothing for Network Calibration (CVPR 2022 · similarity 0.88)
• Noisy Label Calibration for Multi-View Classification (AAAI 2025 · similarity 0.86)
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels (AAAI 2024 · similarity 0.86)
• Training Deep Neural Networks with Virtual Smoothing Classes (AAAI 2025 · similarity 0.85)
• Batch Selection for Multi-Label Classification Guided by Uncertainty and Dynamic Label Correlations (AAAI 2025 · similarity 0.85)
• LaSCal: Label-Shift Calibration without target labels (NeurIPS 2024 · similarity 0.84)