Research map / Large language models / Language, safety and interpretability
SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the SELF-[IN]CORRECT framework comparing an LLM's discrimination of its own sampled answers against its generation accuracy, finding LLMs are not reliably better at discrimination than generation across GSM8K, TriviaQA, MT-Bench and TruthfulQA.
Problem
Whether LLMs can self-improve by judging their own previous outputs depends on whether they discriminate their self-generated candidates better than they generate them; this has not been tested on a unified basis across tasks.
Approach
A unified framework that shares the same self-sampled candidates between a random-choice generation phase and a multiple-choice discrimination phase by the same model, quantified as DG-Diff = S_disc - S_gen, with McNemar/Wilcoxon tests of the hypothesis that discrimination is not better than generation.
Key results
- In 54 of 56 model-task experiments, the one-sided test fails to reject the SELF-[IN]CORRECT hypothesis (p > 0.05), with average DG-Diff negative on GSM8K (-0.76) and TriviaQA (-0.99).
- LLaMA-2 13B Chat on GSM8K drops from 28.3 generation score to 22.8 discrimination score.
- Flan-T5-XXL and Flan-UL2 show positive DG-Diff on most tasks (e.g., TriviaQA +5.8 and +4.2), and the hypothesis is rejected for TriviaQA and TruthfulQA, suggesting a link to autoregressive pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Discriminate among self-generated answers instead of using a randomly selected generation | LLMs evaluated in the main experiments · GSM8K, TriviaQA, MT-Bench, TruthfulQA | DG-Diff: Task Avg.: -0.76; -0.99; -0.06; 0.06 | Mixed |
| Use non-autoregressive Flan models rather than autoregressively pretrained models | Flan-T5 XXL and Flan-UL2 · GSM8K, TriviaQA, MT-Bench, TruthfulQA | DG-Diff: Task Avg.: 0.9; 5.0; 0.05; 5.4 | Helped |
| Double the discrimination-phase in-context examples (ablation) | LLaMA-2 Chat models · GSM8K, TriviaQA, MT-Bench | DG-Diff: -2.8 -> -1.4; -5.5 -> -5.9; -5.9 -> -5.8; -0.1 -> -0.4; 0.0 -> 0.1; -1.6 -> -1.7; -0.13 -> -0.09; -0.51 -> -0.53; -0.17 | Mixed |
| Triple the discrimination-phase in-context examples (ablation) | LLaMA-2 Chat models · GSM8K, TriviaQA | DG-Diff: -2.8 -> 0.1; -5.5 -> -6.8; -5.9 -> -3.9; -0.1 -> 0.2; 0.0 -> 0.1; -1.6 -> -0.5 | Mixed |
| Add chain-of-thought rationales to discrimination examples (ablation) | LLaMA-2 Chat models · TriviaQA, MT-Bench | DG-Diff: -0.1 -> -0.3; 0.0 -> -0.3; -1.6 -> -1.8; -0.13 -> -0.06; -0.51 -> -0.41; -0.17 -> -0.18 | No clear effect |
| Replace incorrect candidates with simpler distractors (ablation) | Models in the simplified discrimination experiment · TriviaQA and GSM8K | DG-Diff | Helped |
| Ask for the chosen answer verbatim rather than its option number (ablation) | Models with invalid discrimination responses · GSM8K and TriviaQA | invalid responses: 14.6 -> 21.5; 13.1 -> 14.2; 13.3 -> 15.3; 25.8 -> 28.0 | Hurt |
| Change discrimination prompt wording (ablation) | LLaMA-2 13B Chat · GSM8K | discrimination accuracy: 22.8; 23.3; 21.9 | No clear effect |
| Score each candidate separately instead of choosing from the candidate list (ablation) | LLaMA-2 Chat models · GSM8K and MT-Bench | accuracy / GPT-Score | No clear effect |
Surprising findings
- Discrimination is often worse than generation even though humans typically find discrimination easier than generation.
- Adding more in-context examples or chain-of-thought rationales to the discrimination prompt does not consistently close the gap.
Limitations
- Discrimination prompts are longer than generation prompts, which may confound the comparison.
- Limited set of models and tasks; the non-autoregressive explanation is not controlled given differing pretraining data.
- Discrimination uses option-selection prompting; other methods such as likelihood ranking were not explored.
Experimental setup
Datasets: GSM8K TriviaQA MT-Bench TruthfulQA
Benchmarks: GSM8K TriviaQA MT-Bench TruthfulQA
Metrics: accuracy GPT-4 score GPT-judge truthful and informative rate DG-Diff McNemar test p-value Wilcoxon signed-rank p-value
Baselines: GPT-3.5-turbo GPT-4-turbo LLaMA-2 Chat
Builds on: self-refine self-rewarding language models self-consistency
Closest papers by meaning
- LLM Evaluators Recognize and Favor Their Own Generations (NeurIPS 2024 · similarity 0.89)
- Tell me about yourself: LLMs are aware of their learned behaviors (ICLR 2025 · similarity 0.87)
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.86)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.86)
- FEEDBACK FRICTION: LLMs Struggle to Fully Incorporate External Feedback (NeurIPS 2025 · similarity 0.86)
- Looking Inward: Language Models Can Learn About Themselves by Introspection (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.