Research map / Large language models / Language, safety and interpretability
When is Multicalibration Post-Processing Necessary?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large empirical study of when multicalibration post-processing (HKRR, HJZ) helps versus ERM and simple calibration (Platt, isotonic, temperature scaling) across tabular, image, and language datasets.
Problem
Whether multicalibration is a natural consequence of ERM in practice, and when multicalibration post-processing is needed and effective, given finite data split between training and post-processing over finite subgroup collections.
Approach
Systematic large-scale empirical evaluation comparing ERM, classical recalibration, and multicalibration post-processing (HKRR and HJZ) across model families and calibration-set fractions, measuring max group-wise smooth ECE on held-out test data.
Key results
- On tabular data, ERM-trained MLPs, random forests, and logistic regression are already nearly multicalibrated; HKRR/HJZ show no statistically significant max smECE improvement for these models.
- SVMs, decision trees, and naive Bayes improve by 0.05 or more in max smECE with multicalibration post-processing, and isotonic regression often matches these gains.
- On image and language data, HKRR improved worst group smECE over ERM in 5 of 6 cases, with reductions such as 0.07 to 0.02 for ViT on Camelyon17.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply HKRR multicalibration post-processing | DistilBERT · Civil Comments | worst-group smECE: 0.06 to 0.043 | Helped |
| apply HKRR multicalibration post-processing | ViT · Camelyon17 | worst-group smECE: 0.07 to 0.02 | Helped |
| apply HKRR multicalibration post-processing | ResNet-56 · Amazon Polarity | worst-group smECE: 0.09 to 0.05 | Helped |
| apply multicalibration post-processing to neural-network ERM | MLP · tabular datasets | worst-group calibration error | No clear effect |
| apply HKRR or HJZ post-processing | SVM, decision tree, or naive Bayes · tabular datasets | max smECE: 0.05 or more | Helped |
| apply multicalibration post-processing | MLP, random forest, or logistic regression · HMDA | worst-group calibration error: 0.03-0.07 | Helped |
| use isotonic regression instead of no post-processing | naive Bayes · tabular datasets | max smECE: up to 0.01 error | Helped |
| use multicalibration post-processing at a calibration fraction that lowers worst-group error | MLP · HMDA | accuracy: about 2% | Hurt |
| select the post-processing method using worst-group ECE rather than worst-group smECE | decision tree · MEPS | worst-group calibration error: 0.156 | Mixed |
| use HKRR rather than HJZ | image and language models · image and language datasets | worst-group calibration error | Helped |
| reuse model training data for multicalibration post-processing instead of holding out calibration data | multicalibration post-processing · tabular datasets | worst-group smECE: 0.025; 0.05-0.1 | Mixed |
| use isotonic regression rather than ERM | large image and language models · image and language datasets | worst-group calibration error: 4/6 | Helped |
Surprising findings
- Calibrated-out-of-the-box models such as MLPs are already nearly multicalibrated, leaving no room for post-processing on tabular data.
- Reusing training data for multicalibration post-processing can significantly hurt worst-group calibration, dropping performance by 0.05-0.1 on Credit Default.
Limitations
- Restricted to binary classification.
- Limited hyperparameter search for base models on image and language data; calibration fractions limited to {0.0, 0.2, 0.4}.
- Groups are feature-defined conjunctions with at least 0.5% of data; results may not transfer to other group designs.
Experimental setup
Datasets: ACS Income UCI Bank Marketing UCI Credit Default HMDA MEPS CelebA Camelyon17 CivilComments Amazon Polarity
Benchmarks: ACS Income UCI Bank Marketing UCI Credit Default HMDA MEPS CelebA Camelyon17 CivilComments Amazon Polarity
Metrics: max group-wise smooth ECE max group-wise binned ECE test accuracy
Baselines: ERM Platt scaling isotonic regression temperature scaling HKRR HJZ
Builds on: HKRR multicalibration algorithm HJZ multicalibration algorithm family Platt scaling isotonic regression temperature scaling
Closest papers by meaning
- Who’s the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration (AAAI 2025 · similarity 0.86)
- Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift (NeurIPS 2024 · similarity 0.85)
- Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.85)
- Unified Approach for Weakly Supervised Multicalibration (NeurIPS 2026 · similarity 0.84)
- Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks (ICML 2024 · similarity 0.84)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.