Research map / Label-efficient and robust learning / Noisy labels

An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration

NeurIPS 2026 · Label-efficient and robust learning › Noisy labels · approach: Noisy cross-modal hashing and multi-label · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Controlled audit of soft-label learning on re-annotated MNIST and Ambiguous-MNIST (Mukhoti) using forced-choice Yes/No/Unsure human annotations, separating label mode shifts from soft-label supervision and measuring calibration (KLD) and dataset-cartography alignment with human uncertainty.

Problem

Prior studies report accuracy gains from human soft-labels but conflate them with corrections of mislabeled data (mode shifts), obscuring the true effect of soft-labels on calibration, training dynamics, and alignment with human uncertainty.

Approach

Re-annotate subsets of MNIST and Ambiguous-MNIST with per-class forced-choice Yes/No/Unsure judgments, build soft-labels, and add a majority-vote one-hot control to isolate soft-label supervision from mode shift; evaluate with KLD against human distributions and dataset cartography correlations with human unsure proportion.

Key results

• On Mukhoti NoHLV items, training on majority-mode relabels adds +9.0 to +14.8 points, while soft-label supervision adds a further +4.0 to +7.1 points across architectures and groups.
• On HLV samples, soft-label training (soft_w) reduces KL divergence to human distributions substantially (e.g., Mukhoti LeNet HLV 0.479 vs 0.905 for majority one-hot and 1.612 for synthetic labels).
• Soft-label training yields the strongest Spearman correlation with human unsure proportion (e.g., LeNet on Mukhoti confidence -0.68 vs -0.44 for majority-vote one-hot and -0.09 for synthetic).

Surprising findings

• Models trained on synthetic Mukhoti labels score highest on the original synthetic test labels but drop by roughly 23-32 points when evaluated against human re-annotations.
• On MNIST, the benefit from relabeling modes is near zero or slightly negative, with nearly all gains coming from soft-label information.

Limitations

• Restricted to simple digit datasets and small models (FFN, LeNet), so results may not transfer to natural images, medical imaging, or language.
• Synthetic target findings are specific to the Mukhoti generation method and do not rule out better synthetic approximations.
• Aggregation abstracts away individual annotator differences in unsure thresholds; the Mukhoti subset is small (about 2,100 training samples).

Experimental setup

Datasets: MNIST Ambiguous-MNIST (Mukhoti)

Benchmarks: MNIST re-annotated test set Ambiguous-MNIST re-annotated test set

Metrics: accuracy KL divergence Brier score Spearman correlation

Baselines: original one-hot labels synthetic Ambiguous-MNIST labels majority-vote one-hot labels LeNet SimpleFFN DeeperFFN

Builds on: dataset cartography Ambiguous-MNIST LeNet

Closest papers by meaning

• Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies Between Model Predictions and Human Responses in VQA (AAAI 2025 · similarity 0.86)
• Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025 · similarity 0.85)
• Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot (NeurIPS 2026 · similarity 0.85)
• Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.84)
• Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability (ICML 2025 · similarity 0.84)
• Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.84)