Research map / Large language models / Alignment and preferences
Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes aligning LLMs by having the model induce harmful responses, write mistake analyses of them, and fine-tune on those analyses (unguided) alongside helpful/harmless data, without external models or human annotation.
Problem
Existing alignment methods (SFT, RLHF) rely on human-annotated error-free instruction-response pairs and discard harmful data; the paper asks how LLMs can learn from their own mistakes safely to improve harmlessness.
Approach
Exploit the model's stronger discrimination ability: guided mistake induction produces harmful self-responses, the model writes mistake analyses of them, and these unguided analysis samples are added to SFT so the model learns what to avoid; guided harmless prompting is used at inference.
Key results
- On Alpaca-7B, mistake analysis with self-induced mistakes and GPT-3.5 analysis raises Harmless Rate from 52.5% to 74.1%, versus 64.1% for RLHF and 63.0% for SFT.
- With ChatGLM-6B as base, Goal Hijacking Rate improves from 77.2% (SFT) to 85.3% for the full method.
- Harmfulness discrimination accuracy of Alpaca-7B rises from 54.5% to 72.6% after mistake-analysis training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mistake analysis for alignment | Alpaca-7B · PKU-SafeRLHF test | Harmless Rate: 52.5 -> 72.4 | Helped |
| use GPT-3.5 instead of Alpaca as the analysis source | Ours · PKU-SafeRLHF test | Harmless Rate: 72.4 -> 74.1 | Helped |
| add mistake analysis to ChatGLM-6B alignment | ChatGLM-6B · Goal Hijacking | Goal Hijacking Rate (%): 68.4 -> 85.3 | Helped |
| use unguided instead of guided SFT instruction (ablation) | mistake-analysis alignment · PKU-SafeRLHF test | Harmless Rate: 67.4 -> 71.0 | Helped |
| use Alpaca-induced mistakes instead of original training-set mistakes (ablation) | mistake-analysis alignment · PKU-SafeRLHF test | Harmless Rate: 71.0 -> 72.4 | Helped |
| add guided analysis rather than use unguided analysis (ablation) | mistake-analysis alignment · PKU-SafeRLHF test | Harmless Rate: 63.3 -> 72.4 | Helped |
| double mistake-analysis quantity by adding analyses of original-dataset mistakes (ablation) | mistake-analysis alignment · PKU-SafeRLHF test | Harmless Rate: 72.4 -> 71.2 | Hurt |
| place hint keywords at positions #1, #2, and #3 rather than no hint (ablation) | Alpaca guided mistake induction · Mistake induction | Harmful Rate (%): 47.2 -> 72.2 | Helped |
| include hint keywords at position #2 in addition to position #3 (ablation) | Alpaca guided mistake induction · Mistake induction | Harmful Rate (%): 67.1 -> 69.5 | Helped |
| use unguided rather than guided SFT instruction (ablation) | ChatGLM-6B mistake-analysis defense · Goal Hijacking | Goal Hijacking Score: 7.64 -> 8.02 | Helped |
| use mistake analysis for binary harmfulness classification | Alpaca-7B · PKU-SafeRLHF test | Accuracy (%): 54.5 -> 72.6 | Helped |
Surprising findings
- Unaligned models like Alpaca can identify harmful content in their own responses better than they avoid generating it (discrimination easier than generation).
- Guided analysis during SFT training performed worse than unguided analysis, attributed to shortcut learning.
Limitations
- Evaluation relies mainly on GPT-4 single-response grading, with human verification only described briefly.
- Adding multiple analyses for the same instruction reduced effectiveness.
- Compute hardware and training costs are not reported.
Experimental setup
Datasets: PKU-SafeRLHF AlpacaFarm SafetyPrompts MOSS
Benchmarks: PKU-SafeRLHF test set AlpacaFarm test set SafetyPrompts test set MOSS helpfulness set
Metrics: Helpful Score Harmless Score Harmless Rate Goal Hijacking score and rate binary harmfulness discrimination accuracy
Baselines: Alpaca (vanilla) SFT RLHF (PPO-Lag) CoH Critique-Revise ChatGLM
Builds on: Chain of Hindsight Critique-Revise PPO-Lag LoRA Alpaca
Closest papers by meaning
- Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.89)
- Large Language Model Unlearning (NeurIPS 2024 · similarity 0.89)
- Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.89)
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.89)
- Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only (ICLR 2025 · similarity 0.89)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.