Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Robust Uncertainty Quantification Using Conformalised Monte Carlo Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MC-CP, combining an adaptive MC dropout that stops early on variance convergence with conformal prediction to produce prediction sets/intervals for classification and quantile regression.
Problem
Uncertainty quantification in deep learning is computationally expensive with MC dropout and BNNs, while conformal prediction methods produce overly conservative prediction sets/intervals.
Approach
An adaptive MC dropout that monitors per-class/quantile variance convergence and terminates forward passes early (threshold delta and patience P), whose mean predictions feed into conformal prediction (RAPS for classification, CQR for regression).
Key results
- On CIFAR-10 MC-CP reduces test error to 1.47% vs 5.07% for Naive CP, 3.29% for RAPS and 35.54% for MC dropout, with prediction size 4.11.
- On Boston Housing MC-CP reaches 98.46% empirical coverage vs 95.97% for CQR, with MAE 0.35 vs 0.31.
- Adaptive MC dropout averaged about 500 of 1000 forward passes on CIFAR-10, cutting inference time from 35.52s to 17.99s (about 50% faster).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RAPS with MC-CP | CNN classification · CIFAR-10 | test error: 3.29 $\pm$ 0.30 -> 1.47 $\pm$ 0.15 | Helped |
| Replace RAPS with MC-CP | CNN classification · CIFAR-100 | test error: 4.83 $\pm$ 0.25 -> 3.54 $\pm$ 0.20 | Helped |
| Replace RAPS with MC-CP | CNN classification · MNIST | test error: 1.10 $\pm$ 0.04 -> 0.32 $\pm$ 0.01 | Helped |
| Replace RAPS with MC-CP | CNN classification · Fashion-MNIST | test error: 1.12 $\pm$ 0.07 -> 0.82 $\pm$ 0.06 | Helped |
| Replace RAPS with MC-CP | CNN classification · Tiny ImageNet | test error: 4.57 $\pm$ 0.09 -> 3.99 $\pm$ 0.41 | Helped |
| Replace CQR with MC-CP | Deep quantile regression · Boston Housing | empirical coverage: 95.97$\pm$ 5.10 -> 98.46 $\pm$ 4.83 | Helped |
| Replace CQR with MC-CP | Deep quantile regression · Boston Housing | MAE: 0.31 $\pm$ 0.61 -> 0.35 $\pm$ 0.20 | Hurt |
| Replace CQR with MC-CP | Deep quantile regression · Abalone | empirical coverage: 92.94 $\pm$ 2.36 -> 95.98 $\pm$ 3.07 | Helped |
| Replace CQR with MC-CP | Deep quantile regression · Blog Feedback | empirical coverage: 90.73 $\pm$ 0.34 -> 95.73 $\pm$ 0.34 | Helped |
| Replace CQR with MC-CP | Deep quantile regression · Concrete | empirical coverage: 90.34 $\pm$ 3.69 -> 93.36 $\pm$ 2.49 | Helped |
| Replace CQR with MC-CP | Deep quantile regression · Protein | empirical coverage: 94.79 $\pm$ 0.01 -> 96.06 $\pm$ 0.73 | Helped |
| Use δ=0.1 and P=1 rather than δ=0.00001 and P=100 (ablation) | Adaptive MC dropout on CIFAR-10 · CIFAR-10 | test error: 1.50 $\pm$ 0.48 -> 6.25 $\pm$ 0.43 | Hurt |
Surprising findings
- MC dropout alone gives essentially no accuracy gain over the baseline CNN on CIFAR-10 (35.54% vs 38.05% error is a gain, but prediction sizes stay at 1 for MC and baseline), showing conformal calibration drives most of the improvement.
- Naive CP has very high variance in prediction set size on CIFAR-100 (sizes from 1 to 86), indicating poor scalability to many classes.
Limitations
- Excludes Deep Ensembles and Bayesian neural networks from comparisons, so baselines are limited.
- Evaluated only with small CNN and VGG models and small-scale benchmarks; no object detection or segmentation.
- On regression MC-CP has slightly higher MAE than CQR on several datasets.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST Tiny ImageNet Boston Housing Abalone Blog Feedback Concrete Compressive Strength Physicochemical Properties of Protein Tertiary Structure
Benchmarks: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST Tiny ImageNet Boston Housing Abalone Blog Feedback Concrete Protein
Metrics: test error prediction set size mean absolute error empirical coverage forward passes runtime
Baselines: Baseline CNN MC dropout Naive CP RAPS CQR Deep quantile regression
Builds on: MC dropout conformal prediction RAPS CQR Platt scaling
Closest papers by meaning
- Reliable Uncertainty Quantification in Machine Learning via Conformal Prediction (AAAI 2025 · similarity 0.90)
- Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows (NeurIPS 2025 · similarity 0.87)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.87)
- Robust Conformal Prediction with a Single Binary Certificate (ICLR 2025 · similarity 0.85)
- Epistemic Uncertainty Quantification For Pre-Trained Neural Networks (CVPR 2024 · similarity 0.85)
- Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.