Research map / Large language models / Language, safety and interpretability
Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CROQ, which uses conformal prediction sets to prune distractor options in multiple-choice questions before re-prompting the LLM, and CP-OPT, a differentiable score-learning method that shrinks prediction sets while keeping coverage.
Problem
LLMs are overconfident and unreliable on multiple-choice decision tasks (MCQ QA and tool selection), and existing conformal prediction scores (logits, self-consistency) often yield large prediction sets that limit how much answer pruning can help.
Approach
Use conformal prediction sets to remove distractor options and re-ask the LLM the revised, shorter MCQ (CROQ), and learn a conformal score via a differentiable surrogate objective (CP-OPT) that minimizes expected set size under a coverage constraint, using LLM penultimate-layer features and logits.
Key results
- CP-OPT scores reduced average set size vs logits in 17 of 27 settings at 95% coverage (significant), with reductions most pronounced at 15 options.
- CROQ with logits improved accuracy in 19 of 27 settings (up to 6.43%); with CP-OPT scores, accuracy improved in 24 settings (up to 7.24%).
- CROQ with CP-OPT beat CROQ with logits in 22 of 27 settings (up to 4.56%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CROQ using logit scores instead of standard inference | Llama-3 · MMLU, 10 options | accuracy: 54.82 -> 56.29 (+1.47*) | Helped |
| Apply CROQ using logit scores instead of standard inference | Llama-3 · MMLU, 4 options | accuracy: 64.02 -> 63.83 (-0.19) | Hurt |
| Apply CROQ using logit scores instead of standard inference | Llama-3 · ToolAlpaca, 15 options | accuracy: 81.43 -> 87.85 (+6.43*) | Helped |
| Apply CROQ using logit scores instead of standard inference | Llama-3 · TruthfulQA, 15 options | accuracy: 37.22 -> 37.22 (0.00) | No clear effect |
| Apply CROQ using CP-OPT scores instead of standard inference | Llama-3 · MMLU, 15 options | accuracy: 51.99 -> 54.77 (+2.78*) | Helped |
| Apply CROQ using CP-OPT scores instead of standard inference | Llama-3 · ToolAlpaca, 15 options | accuracy: 81.43 -> 88.67 (+7.24*) | Helped |
| Apply CROQ using CP-OPT scores instead of standard inference | Phi-3 · TruthfulQA, 10 options | accuracy: 55.70 -> 56.20 (+0.51) | No clear effect |
| Use CP-OPT scores rather than logit scores for CROQ | Llama-3 · MMLU, 10 options | accuracy: 56.29 -> 57.11 (+0.82*) | Helped |
| Use CP-OPT scores rather than logit scores for CROQ | Phi-3 · MMLU, 10 options | accuracy: 61.57 -> 61.05 (-0.52) | Hurt |
| Use CP-OPT scores rather than logit scores for conformal prediction | Llama-3 · MMLU, 10 options | average set size: 5.53 -> 4.90* | Helped |
| Use CP-OPT scores rather than logit scores for conformal prediction | Gemma-2 · ToolAlpaca, 15 options | average set size: 3.54 -> 1.77* | Helped |
| Use CP-OPT scores rather than logit scores for conformal prediction | Llama-3 · ToolAlpaca, 4 options | average set size: 1.17 -> 1.18 | Hurt |
Surprising findings
- Accuracy on MCQs increases nearly monotonically as distractor options are removed, even when pruning is random.
- Logit-based conformal sets tend to over-cover relative to the 95% target.
Limitations
- Gains are modest and often not statistically significant in many settings; some settings show small accuracy drops.
- Coverage guarantee is marginal and multi-round CROQ requires further calibration work not addressed.
- Evaluated only on small-to-medium open-weight models and specific MCQ datasets.
Experimental setup
Datasets: MMLU TruthfulQA ToolAlpaca MMLU-Pro BIRD
Benchmarks: MMLU TruthfulQA ToolAlpaca MMLU-Pro BIRD (NL2SQL)
Metrics: average prediction set size empirical coverage accuracy before/after CROQ accuracy gain statistical significance (paired t-test)
Baselines: LLM logits (softmax) conformal scores standard MCQ inference CROQ with logits
Builds on: split conformal prediction conformal training
Closest papers by meaning
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.90)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.89)
- Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models (NeurIPS 2025 · similarity 0.88)
- Length Optimization in Conformal Prediction (NeurIPS 2024 · similarity 0.88)
- Conformal Information Pursuit for Interactively Guiding Large Language Models (NeurIPS 2025 · similarity 0.88)
- Large Language Models Are Not Robust Multiple Choice Selectors (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.