Research map / Learning theory and algorithms / Prediction and decision losses
Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds bootstrap confidence intervals and a copy-probability generative model for error consistency (EC), then uses them to re-analyze Model-vs-Human and Brain-Score rankings and find many model differences statistically insignificant.
Problem
Error consistency (EC), the standard behavioral-similarity metric between classifiers and humans, is reported as a noisy point estimate without confidence intervals or significance tests, and it depends on accuracy mismatch, so benchmark rankings (Model-vs-Human, Brain-Score) may be unreliable.
Approach
Obtain confidence intervals and significance tests for EC via bootstrapping over trials, and model EC as a copy process where EC equals copy probability times an accuracy-mismatch factor, enabling simulation-based power planning.
Key results
- Bootstrapped 95% CIs for Model-vs-Human EC show the best model (CLIP ViT, EC 0.279) overlaps with more than 70% of models (ranks 8 to 45), with only the top-vs-second difference significant.
- Average Kendall's tau between the original model ranking and bootstrap rankings is 0.88 on Model-vs-Human.
- Simulations suggest CIs still cover about 0.2 units of kappa at N=400 trials, and the authors recommend at least 1000 trials per classifier.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP-trained ViT instead of BiTM-ResnetV2-101x1 | Model-vs-Human corrupted natural images · Model-vs-Human corrupted natural images | EC to humans: 0.251 -> 0.279; difference 0.028 (p < \num{1e-10}) | Helped |
| Bootstrap the full calculation to quantify uncertainty in model EC values | Model-vs-Human benchmark · Model-vs-Human corrupted natural images | significance of neighboring-model EC differences | Hurt |
| Collect more trials per classifier | Error-consistency experiments · Simulated classifiers with equal accuracy | 95% confidence-interval width: covering 0.2 units of $\kappa$ even at $N=400$ trials | Helped |
| Use 1000 trials per classifier | Error-consistency experiments · Behavioral-alignment benchmarking | EC confidence-interval precision | Helped |
| Avoid floor- and ceiling-level classifier accuracy | Error-consistency experiments · Behavioral-alignment benchmarking | EC confidence-interval precision | Helped |
| Compare models using the available Brain-Score EC data | Brain-Score · Brain-Score top-30 models | rank-order resolution | Hurt |
| Use the copy model to estimate EC uncertainty | Synthetic classifier comparisons · Synthetic data; sufficient trials or accuracy away from floor and ceiling | agreement with bootstrap uncertainty estimates | Helped |
| Use the copy model with low trial count and near-ceiling accuracy | Synthetic classifier comparisons · Synthetic data; low trials and accuracy near 100% | agreement with bootstrap uncertainty estimates | Hurt |
Surprising findings
- Many reported differences between deep vision models on Model-vs-Human and Brain-Score are statistically insignificant, though the human-machine gap holds.
- EC is biased downward at low trial counts when accuracies are near floor or ceiling, and is zero whenever one classifier is perfectly accurate.
Limitations
- Relies on bootstrapping rather than an analytic confidence interval, limiting p-value resolution to about 1/N.
- Brain-Score CIs are estimated from assumed copy-model simulations because raw model trials are not stored.
- Copy model diverges from bootstrap results at low trial counts with accuracy near ceiling.
Experimental setup
Datasets: Model-vs-Human human and model responses on corrupted ImageNet-style images Brain-Score behavioral data
Benchmarks: Model-vs-Human Brain-Score
Metrics: error consistency 95% bootstrap confidence intervals p-values Kendall's tau
Baselines: Model-vs-Human EC analysis without confidence intervals Brain-Score behavioral EC ranking
Builds on: error consistency Cohen's kappa Model-vs-Human benchmark Brain-Score
Closest papers by meaning
- Measuring Error Alignment for Decision-Making Systems (AAAI 2025 · similarity 0.88)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.88)
- Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability (ICML 2025 · similarity 0.86)
- Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.86)
- Improved Techniques for Training Consistency Models (ICLR 2024 · similarity 0.86)
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.