Research map / Label-efficient and robust learning / Recognition and long-tail

Improving Accuracy and Calibration via Differentiated Deep Mutual Learning

CVPR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Confidence calibration objectives · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diff-DML, a deep mutual learning variant that keeps auxiliary models diverse (fixed learning rate and no cross-entropy for the auxiliary) to improve both accuracy and calibration of a single inference model.

Problem

Modern DNNs trained with cross-entropy are overconfident; existing calibration methods often trade accuracy for calibration, and deep ensembles are costly at inference. Standard deep mutual learning loses prediction diversity, which hurts calibration.

Approach

Diff-DML differentiates the primary and auxiliary models (auxiliary uses a fixed higher learning rate and drops the cross-entropy term, learning only from a KL to the primary model) so predictions stay diverse; only the primary model is used at inference, and a theory shows DML acts as an implicit deep ensemble.

Key results

• On CIFAR-100, Diff-DML with ResNet50 reaches 79.05% accuracy and 0.86% ECE versus 77.85%/10.39% for the baseline and 77.54%/10.04% for CE.
• Compared with the previous state-of-the-art FL+MDCA, Diff-DML improves ResNet34/ResNet50 accuracy by 1.3/3.1 points and reduces ECE by 49.6%/43.8% relative on CIFAR-100.
• Diff-DML-2 (two auxiliary models) reaches 79.14% accuracy with ResNet34, comparable to DE-3 (79.34%) at lower inference cost.

Measured effects reported in the paper

Surprising findings

• Auxiliary models trained without cross-entropy still yield a stronger primary model, improving over CE on accuracy as well as calibration.

Limitations

• Requires the auxiliary model to have performance comparable to the primary model, complicating scaling to large models.
• Training cost rises with the number of auxiliary models (1.4x to 1.9x).
• Evaluated only on small image classification benchmarks with ResNet backbones.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet SVHN

Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet SVHN

Metrics: accuracy ECE classwise-ECE FPR-95% D-error AUROC AUPR

Baselines: CE DML DE-3 MDCA (FL+MDCA) MMCE DCA FL FL+DTDP Temperature Scaling Vector Scaling Dirichlet Scaling

Builds on: Deep Mutual Learning Deep Ensembles MDCA Focal loss Temperature Scaling

Closest papers by meaning

• Improving model calibration with accuracy versus uncertainty optimization (NeurIPS 2020 · similarity 0.86)
• Better Uncertainty Calibration via Proper Scores for Classification and Beyond (NeurIPS 2022 · similarity 0.86)
• A Benchmark Study on Calibration (ICLR 2024 · similarity 0.85)
• Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration (CVPR 2025 · similarity 0.84)
• SURE: SUrvey REcipes for building reliable and robust deep networks (CVPR 2024 · similarity 0.84)
• Uncertainty Weighted Gradients for Model Calibration (CVPR 2025 · similarity 0.84)