Research map / Large language models / Reasoning and chain of thought
Beyond Reactivity: Proactive Adaptive Conformal Inference for Online LLM Factuality
Xinyu Liu, Jun Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a proactive adaptive conformal inference method for online LLM factuality that handles non-exchangeable query streams, providing guarantees on output correctness.
Problem
LLM hallucinations limit reliability; existing conformal prediction approaches for LLM factuality assume exchangeability, which fails in online settings where user queries and interests shift over time.
Approach
Shifts from reactive adaptive conformal inference to a proactive approach for online LLM factuality under non-exchangeable query streams, providing correctness guarantees on LLM outputs. Exact mechanism not stated in the available abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace reactive DtACI with PACE (proactive shift signal plus reactive error) | DtACI · Synthetic abrupt shift | recovery time (steps): 120.66 -> 46.06 | Improved |
| replace DtACI with PACE | DtACI · Synthetic abrupt shift | tracking RMSE: 0.0461 -> 0.0349 | Improved |
| replace DtACI with PACE | DtACI · Synthetic smooth shift | tracking RMSE: 0.0325 -> 0.0238 | Improved |
| replace DtACI with PACE | DtACI · MMLU stream | tracking RMSE: 0.0175 -> 0.0068 | Improved |
| replace DtACI with PACE | DtACI · MMLU stream | recovery time (steps): 24.6 -> 3.0 | Improved |
| replace fixed-step ACI (gamma=0.2) with PACE | ACI gamma=0.2 · WikiData stream | claims retained (%): 72.17 -> 72.27 | Improved |
| replace DtACI with PACE | DtACI · WikiData stream | tracking RMSE: 0.1656 -> 0.1670 | No clear effect |
| replace static SCP with PACE | SCP · MMLU stream | factuality (%): 88.07 -> 89.85 | Improved |
| use sliding window n=25 instead of n=5 (ablation) | PACE · Synthetic abrupt shift | tracking RMSE: 0.0467 -> 0.0401 | Improved |
| use sliding window n=25 instead of n=100 (ablation) | PACE · Synthetic abrupt shift | tracking RMSE: 0.0496 -> 0.0401 | Improved |
| use sliding window n=5 instead of n=100 (ablation) | PACE · Synthetic abrupt shift | recovery time (steps): 80.27 -> 38.20 | Improved |
| increase weight of proactive signal (WikiData) (ablation) | PACE · WikiData stream | claims retained (%): 71.4 -> 72.3 | Improved |
Experimental setup
Builds on: conformal prediction adaptive conformal inference
Closest papers by meaning
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.89)
- Language Models with Conformal Factuality Guarantees (ICML 2024 · similarity 0.88)
- TEST-TIME SCALING IN DIFFUSION LLMS VIA HIDDEN SEMI-AUTOREGRESSIVE EXPERTS (ICLR 2026 · similarity 0.87)
- ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.87)
- Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.87)
- Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
