Research map / Large language models / Language, safety and interpretability
The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
Nishant Subramani, Palash Goyal, Yiwen Song, Mani Malek, Yuan Xue, Tomas Pfister, Hamid Palangi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace histogram/kernel recalibration of raw confidence with ACUTE activation-based random forest estimators | LLM confidence estimation · MMLU (6 LLMs avg) | AUC-EURO (all): 0.79 -> 0.83 (NWKR vs ACUTE late act) | Improved |
| replace NWKR with ACUTE pca20 on tool-calling | LLM confidence estimation · APIGen (6 LLMs avg) | AUC-EURO (all): 0.78 -> 0.88 | Improved |
| HRE/NWKR recalibration of raw confidences | raw LLM confidence · MMLU (6 LLMs avg) | smECE: 0.17 -> 0.07 | Improved |
| ACUTE vs NWKR/HRE on calibration error | ACUTE · APIGen / MMLU | smECE: 0.02 (NWKR) vs 0.06 (ACUTE pca20) on APIGen | No clear effect |
| add ACUTE on SCITLDR (hard task) | ACUTE · SCITLDR dev (6 LLMs avg) | AUC-EURO (all): 0.77 -> 0.78 | Improved |
| use ACUTE on larger MoE model Qwen3-30B-A3B | ACUTE · APIGen | AUC-EURO (all): 0.797 -> 0.872 (NWKR vs ACUTE late act) | Improved |
| use cosine similarity features instead of raw activations (ablation) | ACUTE · APIGen (6 LLMs avg) | AUC-EURO (all): 0.87 -> 0.83 | Hurt |
| use middle/late layers instead of early layers for activations (ablation) | ACUTE · MMLU (6 LLMs avg) | AUC-EURO (all): 0.79 -> 0.83 | Improved |
| increase PCA components from 10 to 20 (ablation) | ACUTE pca · APIGen (6 LLMs avg) | AUC-EURO (all): 0.87 -> 0.88 | Improved |
| ACUTE with only 25 training examples vs baselines with 1000 (ablation) | ACUTE · APIGen (gemma3-12b-it) | AUC-EURO | Improved |
| add posthoc calibration (Platt, isotonic, beta) on top of ACUTE (ablation) | ACUTE · SCITLDR (7 LLMs avg) | smECE and AUC-EURO | Improved |
| add posthoc calibration on ACUTE cosine (ablation) | ACUTE cosine · APIGen (7 LLMs avg) | AUC-EURO | No clear effect |
Closest papers by meaning
- Calibrating LLMs with Semantic-level Reward (NeurIPS 2026 · similarity 0.90)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.90)
- SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.90)
- QA-Calibration of Language Model Confidence Scores (ICLR 2025 · similarity 0.89)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.89)
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
