Research map / Large language models / Language, safety and interpretability
Large Language Models Are Not Robust Multiple Choice Selectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLMs have a selection bias toward certain option IDs in MCQs, attributes it mainly to token bias, and proposes PriDe, a label-free inference-time debiasing method that estimates the option-ID prior from a small sample.
Problem
LLMs are not robust to option position changes in multiple-choice questions because they prefer specific option IDs (e.g., "A"), which distorts MCQ-based evaluation.
Approach
Decompose the observed option-ID distribution into a prior over option IDs and a debiased distribution over option contents; estimate the prior by permutation-based debiasing on a small subset of test samples and reuse the averaged prior to debias the rest, without labels.
Key results
- PriDe with 5% estimation samples achieves average RStd reduction of -7.6 and accuracy gain of +1.2 on MMLU (0-shot) across 20 LLMs at about 1.15x cost, versus Cyclic Perm's -8.7 RStd at 4x cost.
- Removing option IDs reduces selection bias (RStd) notably, showing token bias is a primary cause, while shuffling IDs or replacing ID symbols gives no remarkable reduction.
- Moving golden answers to position D lowers gpt-3.5-turbo MMLU accuracy by 6.3 points (67.2 to 60.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove option IDs (ask model to pick option content via likelihood) (ablation) | 20 LLMs · MMLU/ARC/CSQA 0-shot | RStd (selection bias): -6.4 MMLU, -5.1 ARC, -6.2 CSQA (avg RStd change) | Helped |
| Remove option IDs (ablation) | 20 LLMs · MMLU 0-shot | Acc: -2.1 avg | Hurt |
| Shuffle default option IDs (A/B/C/D) (ablation) | gpt-3.5-turbo · MMLU 0-shot | RStd: 5.5 to 5.1 | No clear effect |
| Replace IDs with a/b/c/d, 1/2/3/4, or (A)/(B)/(C)/(D) (ablation) | gpt-3.5-turbo · MMLU 0-shot | RStd: 6.8, 3.8, 8.1 vs 5.5 default | No clear effect |
| Explicit debiasing instruction in system message (ablation) | gpt-3.5-turbo · MMLU 0-shot | RStd: 6.1 vs 5.5 default | No clear effect |
| Chain-of-thought prompting (ablation) | gpt-3.5-turbo · MMLU 0-shot | RStd: 4.5 vs 5.5 default; Acc 66.8 vs 67.2 | No clear effect |
| PriDe debiasing with prior estimated on 5% of test samples | 20 LLMs · MMLU 0-shot | RStd: -7.6 avg at x1.15 cost | Helped |
| PriDe debiasing with prior estimated on 5% of test samples | 20 LLMs · MMLU 0-shot | Acc: +1.2 avg | Helped |
| Cyclic permutation debiasing (x n cost) | 20 LLMs · MMLU 0-shot | RStd: -8.7 avg at x4 cost | Helped |
| Larger prior-estimation budget (80%) in PriDe | 20 LLMs · MMLU 0-shot | Acc: +4.1 avg at x3.4 cost | Helped |
| PriDe with 5% estimation budget on MMLU, Cyclic Perm, and 5-shot setting | 20 LLMs · MMLU 5-shot | RStd: -5.3 avg (PriDe 5%) vs -5.8 (Cyclic Perm) | Helped |
| Cross-domain transfer of prior (estimated on source domain, applied to target) | PriDe · MMLU and ARC cross-domain 0-shot | debiasing (RStd) | Helped |
Surprising findings
- Position bias is less regular and model-dependent than token bias; token bias is the more salient cause of selection bias.
- Debiasing methods such as PriDe and Cyclic Permutation usually improve accuracy even though this is not their goal.
Limitations
- Removing option IDs usually degrades model performance, so it is not a practical debiasing method.
- Estimated priors correlate with empirical selection bias only moderately for some models and scales may differ from recall frequencies.
- Cross-domain transfer shows slight performance degradation when domain gap is large; 5-shot transferability is weaker.
Experimental setup
Datasets: MMLU ARC-Challenge CommonsenseQA
Benchmarks: MMLU ARC-Challenge CommonsenseQA
Metrics: accuracy standard deviation of recalls (RStd) recall balance
Baselines: Cyclic Permutation Full Permutation Removing IDs Debiasing Instruct Chain-of-Thought
Builds on: cyclic permutation debiasing chain-of-thought prompting
Closest papers by meaning
- B-score: Detecting biases in large language models using response history (ICML 2025 · similarity 0.89)
- Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction (ICML 2025 · similarity 0.88)
- Listening to the Wise Few: Query–Key Alignment Unlocks Latent Correct Answers in Large Language Models (NeurIPS 2026 · similarity 0.88)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.88)
- Set-LLM: A Permutation-Invariant LLM (NeurIPS 2025 · similarity 0.87)
- Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.