Research map / Large language models / Language, safety and interpretability
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
Gaojie Jin, Yong Tao, Lijia Yu, Tianjin Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive margin to confidence estimator training (ablation) | Learning Confidence (Vanilla), Mistral-7B · AlpacaEval | ranking loss RK: 0.3865 -> 0.3393 | Improved |
| add adaptive margin to confidence estimator training (ablation) | Learning Confidence (Vanilla), Mistral-7B · AlpacaEval | AUROC: 0.6179 -> 0.6672 | Improved |
| add adaptive margin (ablation) | Learning Confidence (Vanilla), Llama3-70B · AlpacaEval | ranking loss RK: 0.3236 -> 0.2776 | Improved |
| add adaptive margin (ablation) | Learning Confidence (Vanilla), Qwen2.5-72B · AlpacaEval | ranking loss RK: 0.3370 -> 0.2707 | Improved |
| add adaptive margin (ablation) | Learning Confidence (Vanilla), Mistral-7B · HH-RLHF | ranking loss RK: 0.3841 -> 0.3286 | Improved |
| add adaptive margin (ablation) | CSE + Vanilla, L->Q->O · AlpacaEval | guarantee success rate (%): 93.4 -> 94.8 | Improved |
| add adaptive margin (ablation) | CSE + Vanilla, L->Q->O · HH-RLHF | guarantee success rate (%): 37.9 -> 51.6 | Improved |
| add adaptive margin (ablation) | CSE + Vanilla, M->L->O · AlpacaEval | guarantee success rate (%): 75.7 -> 82.1 | Improved |
| add adaptive margin (ablation) | CSE + Vanilla, L->Q->O · TL;DR | guarantee success rate (%): 75.4 -> 79.2 | Improved |
| replace Simulated Annotators confidence with learned confidence | CSE, L->Q->O · Chatbot Arena | guarantee success rate (%): 90.2 -> 92.7 | Improved |
| replace perfect-calibration GPT-4 heuristic with CSE using learned confidence | Heuristic Selection, L->Q->O · AlpacaEval | guarantee success rate (%): 0.0 -> 94.8 | Improved |
| replace Heuristic Selection with CSE + Ours | Heuristic Selection · Chatbot Arena coverage (L->Q->O) | coverage (%): 88.4 -> 56.4 | Hurt |
Closest papers by meaning
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.91)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.90)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.90)
- SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.89)
- ConfTuner: Training Large Language Models to Express Their Confidence Verbally (NeurIPS 2025 · similarity 0.89)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
