Research map / Large language models / Language, safety and interpretability
A Theoretical Understanding of Self-Correction through In-context Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically shows that transformers can perform gradient descent on Bradley-Terry/Plackett-Luce alignment objectives in-context from self-correction triplets, and proposes Checking-as-Context (CaC) for intrinsic self-correction to reduce social bias and jailbreak success.
Problem
Little is known about how LLM self-correction arises; prior in-context learning theory covers only linear regression with linear attention and does not explain alignment from preference-style (query, response, reward) triplets.
Approach
Formulates self-correction as in-context alignment: a stacked multi-head softmax transformer with FFN can implement gradient descent on BT (N=2) and Plackett-Luce (N>2) objectives using (x, y, r) triplets, with critics acting as supervision. Proposes Checking-as-Context, a generation-critique-regeneration procedure with no external feedback.
Key results
- On AdvBench, CaC-self-instruct reduces Vicuna-7b ASR from 95% to 1% on GCG-individual and from 90% to 0% on GCG-transfer.
- CaC reduces Llama2-7b-chat ASR on GCG-individual from 38% to 0% and on AutoDAN from 12% to 0%.
- Ablations on synthetic data show softmax attention, at least 3 heads, FFN, and depth (about 15+ layers) are needed for in-context alignment; linear attention and no-FFN models fail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Checking-as-Context self-correction | Vicuna · AdvBench, GCG-individual | ASR: 95% -> 1% | Helped |
| Add Checking-as-Context self-correction | Vicuna · AdvBench, GCG-transfer | ASR: 90% -> 0% | Helped |
| Add Checking-as-Context self-correction | Vicuna · AdvBench, AutoDAN | ASR: 91% -> 29% | Helped |
| Add Checking-as-Context self-correction | Llama2 · AdvBench, AutoDAN | ASR: 12% -> 0% | Helped |
| Replace multi-round checking with diverse checking context (ablation) | CaC on Vicuna · AdvBench, AutoDAN | ASR: 88% -> 80% | Helped |
| Add self-instruct refinement instead of repeating the query (ablation) | CaC on Vicuna · AdvBench, AutoDAN | ASR: 88% -> 29% | Helped |
| Add diverse checking context (ablation) | CaC on Vicuna · AdvBench, GCG-individual | ASR: 2% -> 2% | No clear effect |
| Use 2 checking rounds instead of 1 (ablation) | CaC-self on Vicuna · AdvBench, GCG-individual | ASR: 4% -> 2% | Helped |
| Use 3 checking rounds instead of 2 (ablation) | CaC-self on Vicuna · AdvBench, GCG-individual | ASR: 2% -> 2% | No clear effect |
| Use 2 checking rounds instead of 1 (ablation) | CaC-diverse on Vicuna · AdvBench, GCG-individual | ASR: 6% -> 3% | Helped |
| Use 3 checking rounds instead of 2 (ablation) | CaC-diverse on Vicuna · AdvBench, GCG-individual | ASR: 3% -> 2% | Helped |
| Use more accurate critics (ablation) | CaC · BBQ | accuracy | Helped |
Surprising findings
- Linear attention can hardly solve the in-context alignment task, while softmax attention performs much better.
- On AutoDAN, CaC-self-instruct remains at 29% ASR on Vicuna, showing persuasive instructions can still partly defeat self-correction.
Limitations
- Theory relies on a simplified linear regression alignment setup with constructed transformer weights rather than realistic language tasks.
- Self-correction gains depend on critic accuracy; persuasive jailbreak prompts such as AutoDAN still partially succeed on Vicuna.
- Some ASR judgments rely on suffix detection and GPT-4 with human check, which may be imperfect.
Experimental setup
Datasets: BBQ AdvBench GLUE
Benchmarks: BBQ AdvBench GLUE
Metrics: normalized MSE accuracy attack success rate
Baselines: Self-reminder RAIN ICD gradient descent on PL model
Builds on: in-context learning theory for linear regression Bradley-Terry model Plackett-Luce model Self-reminder ICD
Closest papers by meaning
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.90)
- Small Language Model Can Self-Correct (AAAI 2024 · similarity 0.89)
- Explorations of Self-Repair in Language Models (ICML 2024 · similarity 0.89)
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.89)
- Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.