Research map / Label-efficient and robust learning / Recognition and long-tail

Human and AI Perceptual Differences in Image Classification Errors

AAAI 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Post-hoc explanation analysis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares error distributions of noisy human annotators (CIFAR-N) and thirteen robust-learning classifiers on CIFAR-10, finds machines err similarly to each other but differently from humans, and shows that thresholding machine confidence to fall back on a human label beats machine-machine teaming.

Problem

Whether AI image classifiers trained with noisy labels make mistakes like humans do, beyond what overall accuracy captures, and whether these perceptual differences can be exploited for better human-AI teaming.

Approach

Analyzes confusion matrices of human and machine errors, partitions test images by machine confidence/agreement and human time/agreement difficulty, and uses a simple confidence-threshold rule to replace low-confidence machine predictions with a human label.

Key results

• Machine classifiers reach 84.5-90.0% accuracy on CIFAR-10 when trained on CIFAR-N noisy labels, versus 81.9-82.8% for individual noisy human annotators.
• Regression slope of accuracy vs. machine confidence is 1.00 for machines but only 0.11 for humans, indicating humans' accuracy is largely unaffected by machine difficulty.
• Thresholding machine confidence (eta up to 0.6) to swap in a human label gives significant accuracy gains and beats machine-machine teaming; oracle teaming of CE with a human reaches 95.82%.

Measured effects reported in the paper

Surprising findings

• Humans outperform machines on the subset where machines have low confidence or low agreement, despite lower overall accuracy.
• Low-accuracy human annotators provide larger teaming gains than higher-accuracy machine classifiers.

Limitations

• Only CIFAR-10 is studied, so findings may not generalize to other datasets.
• Only a limited set of machine classification methods is examined.
• The realistic collaboration rule is simple and its gains are well below the oracle upper bound.

Experimental setup

Datasets: CIFAR-10 CIFAR-N CIFAR-H

Benchmarks: CIFAR-10

Metrics: accuracy matching percentage on balanced set confusion matrix element comparison paired t-test linear regression slope

Baselines: CORES PLS F-Div GCE FW PeerLoss NLS CAL Co-teaching+ JoCoR Co-teaching

Builds on: co-teaching CIFAR-N CIFAR-H

Closest papers by meaning

• Partial success in closing the gap between human and machine vision (NeurIPS 2021 · similarity 0.90)
• Human Expertise in Algorithmic Prediction (NeurIPS 2024 · similarity 0.85)
• Evaluating alignment between humans and neural network representations in image-based learning tasks (NeurIPS 2024 · similarity 0.85)
• When does perceptual alignment benefit vision representations? (NeurIPS 2024 · similarity 0.85)
• Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.85)
• Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision (NeurIPS 2026 · similarity 0.85)