Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Bayesian Inference for Correlated Human Experts and Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hierarchical Bayesian model (logistic-normal with LKJ prior and temperature scaling) that jointly models correlated human experts and a pre-trained classifier to sequentially query as few experts as possible while predicting the expert consensus.
Problem
Predicting the class labels that a group of human experts would produce (their consensus) while minimizing the number of costly expert queries, using classifier probabilities and partially observed expert votes.
Approach
Model latent expert and classifier confidence logits jointly as a multivariate normal with LKJ prior and a temperature-scaled categorical vote model, use MCMC posterior samples to simulate unobserved expert votes, and choose the next expert to query by expected entropy reduction of the consensus; experts are treated as identifiable and correlated, unlike prior exchangeable approaches.
Key results
- On ChestX-ray, the proposed method reaches 0% error with 2.55 queries on average versus 3.16 for INFEXP.
- On Chaoyang, the method reaches 0% error with 1.58 queries versus 1.82 for INFEXP.
- Expected calibration error of the method is generally below 1% across datasets at error thresholds 0.05, 0.025, and 0.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Bayesian method instead of INFEXP + ε-greedy querying | Consensus prediction on ChestX-Ray · ChestX-Ray | average expert queries at 0% error: 3.16 -> 2.55 | Helped |
| Use the Bayesian method instead of INFEXP + ε-greedy querying | Consensus prediction on Chaoyang · Chaoyang | average expert queries at 0% error: 1.82 -> 1.58 | Helped |
| Use the Bayesian method instead of confusion + calibration | Consensus prediction · ChestX-Ray and Chaoyang | error rate | Helped |
| Use the Bayesian method instead of INFEXP and confusion + calibration | Consensus prediction · CIFAR-10H and ImageNet-16H | error rate | Mixed |
| Reduce the error threshold from 0.05 to 0.01 | Bayesian method · ChestX-Ray, Chaoyang, CIFAR-10H, and ImageNet-16H | ECE: 0.009 -> <0.001; 0.008 -> <0.001; 0.007 -> <0.001; 0.009 -> <0.001 | Helped |
| Move from the first 50 to the last 50 examples | Bayesian method with e=0.01 · ChestX-Ray, Chaoyang, CIFAR-10H, and ImageNet-16H | average experts queried: 2.56 -> 1.57; 2.10 -> 1.61; 1.98 -> 1.10; 2.07 -> 1.32 | Helped |
| Use the sliding-window approach after a distribution shift | Bayesian method · ImageNet-16H with low-noise then high-noise examples | error rate: 0% error | Helped |
| Predict whether any expert votes positive instead of majority vote | Bayesian method · ChestX-Ray | error rate versus query cost | Mixed |
| Predict unanimous positive votes instead of majority vote | Bayesian method · ChestX-Ray | average expert queries | Hurt |
Surprising findings
- The confusion matrices + calibration baseline never reaches 0% error even when all experts are queried, because it does not model the consensus directly.
Limitations
- Evaluation limited to image classification with small numbers of classes and agents; covariance grows quickly with K, M, H.
- Several datasets use synthetic experts built by combining non-identifiable annotators.
- Baseline INFEXP and confusion methods are modified versions rather than originals.
Experimental setup
Datasets: ChestX-ray Chaoyang CIFAR-10H ImageNet-16H
Benchmarks: ChestX-ray Chaoyang CIFAR-10H ImageNet-16H
Metrics: error rate relative to expert consensus average number of expert queries expected calibration error
Baselines: INFEXP + epsilon-greedy querying confusion matrices + calibration
Builds on: INFEXP logistic normal model LKJ prior temperature scaling
Closest papers by meaning
- Principled Bayesian Optimization in Collaboration with Human Experts (NeurIPS 2024 · similarity 0.85)
- Human Expertise in Algorithmic Prediction (NeurIPS 2024 · similarity 0.84)
- A Square Peg in a Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning (ICML 2025 · similarity 0.83)
- idSCD: Identifying Training Datasets through Semantic Correlation Descriptors (NeurIPS 2026 · similarity 0.83)
- Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation (ICLR 2024 · similarity 0.83)
- SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.