Research map / Large language models / Language, safety and interpretability

Aligning Language Models with Selective Prediction

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLSR, a GRPO-based alignment method that uses a lifted, batch-ranked AURC reward on verbalized confidence to improve LLM selective prediction, outperforming RLVR and RLCR on HotPotQA and BigMath.

Problem

LLMs answer regardless of confidence, causing confident errors; existing alignment (RLVR for accuracy, RLCR for calibration) does not optimize selective prediction, which is measured by AURC.

Approach

Replace the correctness reward with a lifted AURC reward (+/- rank-based weights computed over a pooled batch of rollouts), so correct and incorrect predictions are pushed apart by confidence ranking, enabling mini-batch GRPO training for selective prediction.

Key results

• On HotPotQA-trained Qwen2.5-7B, RLSR reaches in-domain AURC 0.44 vs 0.56 for RLVR and 0.51 for RLCR, and 66.0% accuracy at 10% coverage vs 45.5% and 53.5%.
• RLCR achieves the best ECE (0.05 in-domain on HotPotQA) but lags RLSR on AURC, showing calibration and selective prediction are different objectives.
• In the MedQA risk-controlled deployment at 75% target accuracy, RLSR attains 78.9% test accuracy at 3.0% test coverage, while RLCR drops from 81.0% validation to 72.2% test accuracy.

Surprising findings

• RLVR consistently degrades out-of-domain selective prediction relative to the unaligned base model.
• Post-hoc temperature scaling on RLVR does not change selective prediction performance because it preserves confidence ranking.

Limitations

• Optimizes the holistic AURC rather than risk-constrained coverage, the more practical error-control setting.
• Evaluated with verbalized confidence only; the method is confidence-scorer agnostic but other scorers are only briefly explored.
• Batch-based ranking approximates the population AURC, with performance dependent on batch size.

Experimental setup

Datasets: HotPotQA-Modified BigMath HotPotQA SimpleQA TriviaQA GPQA Math500 GSM8K CommonsenseQA MedQA

Benchmarks: HotPotQA BigMath Digits SimpleQA TriviaQA GPQA Math500 GSM8K CommonsenseQA MedQA

Metrics: AURC ECE Accuracy@10% Accuracy@25% Accuracy@50%

Baselines: BASE RLVR RLCR RLVR-Temp

Builds on: GRPO RLVR RLCR

Closest papers by meaning

• Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference (NeurIPS 2025 · similarity 0.89)
• Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.89)
• EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.88)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.88)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.88)
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.88)