Research map / Large language models / Language, safety and interpretability
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
Mohammad Anas Jawad, Cornelia Caragea
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CaliDist (As.) distraction-based scaling to vanilla logit confidence | Llama-3.1-8B · 7-dataset average | ECE: 15.18 -> 4.37 | Improved |
| apply CaliDist (As.) to vanilla confidence | Qwen3-8B · 7-dataset average | ECE: 26.26 -> 3.77 | Improved |
| replace temperature scaling with CaliDist (Pr.) | Llama-3.1-8B · 7-dataset average | ECE: 7.42 -> 4.08 | Improved |
| replace self-consistency (15 passes) with CaliDist (As.) | Llama-3.1-8B · 7-dataset average | ECE: 18.65 -> 4.37 | Improved |
| apply CaliDist to verbalized confidence | Llama-3.1-8B · 5-dataset average | ECE: 18.00 -> 9.50 | Improved |
| apply CaliDist to verbalized confidence | Gemini-2.0-Flash · 3-dataset average | ECE: 12.69 -> 13.00 | No clear effect |
| apply CaliDist to log-prob confidence | GPT-4o-Mini · 3-dataset average | ECE: 20.60 -> 2.82 | Improved |
| tune sigmoid alpha and beta by grid search instead of default alpha=0, beta=1 (ablation) | CaliDist · Llama-3.1 and Qwen-3 | ECE | Improved |
| use CaliDist instead of SPUQ | Llama-3.1-8B · CSQA | ECE: 12.68 -> 4.73 | Improved |
| use CaliDist instead of Vector Scaling with 200 validation samples | Llama-3.1-8B · MNLI | ECE: 3.47 -> 2.80 | Improved |
| use CaliDist instead of FSD consistency | Llama-3.1 ContractNLI · ContractNLI | ECE: 19.16 -> 4.04 | Improved |
| use CaliDist (As.) instead of vanilla confidence | Qwen3-8B · CSQA | AUROC: 70.01 -> 77.62 | Improved |
Closest papers by meaning
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.92)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.91)
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.90)
- Thermometer: Towards Universal Calibration for Large Language Models (ICML 2024 · similarity 0.90)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.90)
- Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
