Research map / Continual learning / Online and replay-based

Model Uncertainty Quantification by Conformal Prediction in Continual Learning

ICML 2025 · Continual learning › Online and replay-based · approach: Unified and unsupervised continual frameworks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CPCL, a conformal prediction method for regression-based uncertainty quantification in continual learning that replays calibration samples from past tasks, reconstructs sequentially dependent nonconformity scores, and fits a quantile regression forest, with asymptotic coverage proofs.

Problem

Conformal prediction assumes exchangeable data, but continual learning tasks arrive sequentially and violate exchangeability, while access to previous-task data for calibration is limited, so reliable uncertainty quantification under continual learning is unclear.

Approach

Replay a buffer of past-task samples to form a calibration set, compute sigmoid-transformed nonconformity scores, reconstruct a dataset of sequential score vectors, and train a quantile regression forest on it to estimate conditional quantiles for prediction intervals; also links interval length growth to forgetting.

Key results

• On simulated data over 100 runs, CPCL coverage stays above the target 1-α across SI, EWC, MAS and fine-tuning for α from 0.05 to 0.3 (e.g., above 0.95 with EWC at α=0.05).
• On Tiny ImageNet with 5 sequential tasks, average prediction interval length tends to increase as tasks accumulate; with EWC at α=0.1 it reaches about 9 after task 5 versus below 8 after task 2.
• Theory proves pointwise convergence of the estimated conditional CDF and quantile of CPCL as calibration size grows, giving asymptotic conditional coverage.

Measured effects reported in the paper

Surprising findings

• Interval length growth is interpreted as a signal of catastrophic forgetting, linking calibration-based uncertainty to forgetting.

Limitations

• Theoretical guarantees rely on strong assumptions (Lipschitz conditional CDF, strictly increasing CDF, tree-construction conditions) and asymptotic regime as calibration size grows.
• Experiments are limited to regression with a small number of tasks (2 simulated, 5 Tiny ImageNet) and a pretrained AlexNet used as the data-generating mechanism rather than a realistic predictor.
• Replay buffer of samples from previous tasks is assumed; alternative replay techniques are not developed.

Experimental setup

Datasets: Simulated Gaussian linear regression data Tiny ImageNet

Benchmarks: Simulated linear regression data Tiny ImageNet (regression, 5 tasks)

Metrics: marginal coverage average prediction interval length

Baselines: SI EWC MAS DGR Fine-tuning

Builds on: Conformal prediction (Vovk et al., 2005) Quantile regression forests (Meinshausen, 2006) Xu & Xie (2023) conformal time series prediction Replay-based continual learning

Closest papers by meaning

• Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.88)
• An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.87)
• Temporal-Difference Variational Continual Learning (NeurIPS 2025 · similarity 0.87)
• BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.87)
• Reliable Uncertainty Quantification in Machine Learning via Conformal Prediction (AAAI 2025 · similarity 0.87)
• Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.86)