Research map / Label-efficient and robust learning / Noisy labels
Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes supervision signals in confidence-difference (ConfDiff) classification and proposes CRCR, which splits pairs by confidence-difference magnitude and adds a consistency regularizer for small-difference pairs, with error bounds and noisy-CD experiments.
Problem
Pairwise unlabeled data with confidence differences can train binary classifiers (ConfDiff), but pairs with small confidence differences give unreliable supervision that pushes same-class instances apart, hurting accuracy especially under noise.
Approach
Partition pairs by |confidence difference| threshold: keep the original ConfDiff risk for large-difference pairs and add a consistency regularization term (weighted by 1/log(|c|+eps)) that encourages similar outputs for small-difference pairs; also derives an estimation error bound.
Key results
- With balanced classes (pi=0.5) and no added noise, CRCR-ABS achieves 0.962 on MNIST and 0.848 on K-MNIST versus 0.944 and 0.825 for ConfDiff-ABS.
- Under 100% noise with pi=0.5, CRCR-ABS reaches 0.960 on MNIST and 0.856 on K-MNIST, showing robustness to artificial confidence-difference noise.
- CRCR-ABS outperforms nearly all baselines in balanced settings, with the exception of Pcomp-Unbiased on Pendigits without noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CRCR-ABS instead of the strongest listed baseline, ConfDiff-ABS | ConfDiff classification · MNIST (π=0.5, m=0) | accuracy: 0.944±0.003 -> 0.962±0.006 | Helped |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · K-MNIST (π=0.5, m=0) | accuracy: 0.825±0.011 -> 0.848±0.013 | Helped |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · F-MNIST (π=0.5, m=0) | accuracy: 0.952±0.004 -> 0.955±0.002 | Helped |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · CIFAR-10 (π=0.5, m=0) | accuracy: 0.848±0.014 -> 0.869±0.009 | Helped |
| Use CRCR-ABS instead of Pcomp-Unbiased | ConfDiff classification · Pendigits (π=0.5, m=0) | accuracy: 0.775±0.018 -> 0.753±0.009 | Hurt |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · Optdigits (π=0.5, m=0) | accuracy: 0.799±0.023 -> 0.823±0.023 | Helped |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · MNIST (π=0.2, m=0) | accuracy: 0.910±0.015 -> 0.916±0.022 | No clear effect |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · K-MNIST (π=0.2, m=0) | accuracy: 0.841±0.014 -> 0.856±0.006 | Helped |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · F-MNIST (π=0.2, m=0) | accuracy: 0.940±0.010 -> 0.922±0.007 | Hurt |
| Use CRCR-ABS instead of Pcomp-ABS | ConfDiff classification · CIFAR-10 (π=0.2, m=0) | accuracy: 0.833±0.004 -> 0.812±0.017 | Hurt |
| Use CRCR-ABS instead of Pcomp-Unbiased | ConfDiff classification · Pendigits (π=0.2, m=0) | accuracy: 0.820±0.025 -> 0.784±0.024 | Hurt |
| Use CRCR-ABS instead of ConfDiff-ABS | ConfDiff classification · Optdigits (π=0.2, m=0) | accuracy: 0.830±0.016 -> 0.825±0.005 | No clear effect |
Surprising findings
- Pcomp-Unbiased outperforms CRCR-ABS on Pendigits without added noise, which the authors attribute to Pcomp using only ordering information.
- ReLU-corrected ConfDiff and Pcomp variants tend to collapse to near-random (0.800 accuracy) on imbalanced noisy data.
Limitations
- Threshold theta and weight alpha require tuning; recommended values depend on dataset and class prior.
- Noisy confidence differences are synthetically generated from a logistic regression classifier rather than real human annotations.
- Unbiased CRCR variant is unstable on complex or noisy datasets.
Experimental setup
Datasets: MNIST Kuzushiji-MNIST Fashion-MNIST CIFAR-10 Optdigits Pendigits
Benchmarks: MNIST Kuzushiji-MNIST Fashion-MNIST CIFAR-10 Optdigits Pendigits
Metrics: classification accuracy (mean±std over 5 runs)
Baselines: Pcomp-Unbiased Pcomp-ReLU Pcomp-ABS Pcomp-Teacher ConfDiff-Unbiased ConfDiff-ReLU ConfDiff-ABS
Builds on: ConfDiff classification Pcomp classification unbiased PU risk estimation
Closest papers by meaning
- Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget (ICML 2024 · similarity 0.85)
- Label Noise: Ignorance Is Bliss (NeurIPS 2024 · similarity 0.83)
- Retraining with Predicted Hard Labels Provably Increases Model Accuracy (ICML 2025 · similarity 0.83)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.83)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.83)
- Mitigating Label Bias in Machine Learning: Fairness through Confident Learning (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.