Research map / Large language models / Alignment and preferences
RAIN: Your Language Models Can Align Themselves without Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAIN, a training-free inference method that lets frozen LLMs self-evaluate partial generations and rewind a token-set search tree (PUCT-style) to produce more harmless and truthful outputs.
Problem
Aligning large language models with human preferences usually requires finetuning (RLHF, RLAIF, DPO) and human or AI-labeled data; the paper asks whether frozen LLMs can align themselves at inference time without any data or parameter updates.
Approach
Integrate self-evaluation with a rewindable tree search over token sets during auto-regressive decoding: forward PUCT-guided exploration, LLM self-scoring of leaf texts, value/visit-count backpropagation with sibling similarity updates, and rewinding, so the final next-token-set distribution is reweighted toward self-judged harmless text.
Key results
- On HH, RAIN raises LLaMA 30B harmlessness from 82.2% (vanilla) to 97.0% per GPT-4 evaluation while keeping helpfulness.
- Under GCG white-box attacks, RAIN lowers Vicuna 33B attack success rate from 94% to 19%.
- On TruthfulQA, RAIN raises LLaMA-2-chat 13B True+Info from 68.5% to 72.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RAIN inference | Vicuna · AdvBench, GCG white-box and transferred attacks | attack success rate: 86% -> 72%; 80% -> 55% (7B); 83% -> 38%; 79% -> 32% (13B); 94% -> 19%; 34% -> 10% (33B) | Helped |
| Add RAIN inference | LLaMA-2-chat 13B · TruthfulQA | truthfulness: 69.2% -> 74.1% | Helped |
| Add RAIN inference | LLaMA-2-chat 13B · TruthfulQA | informativeness: 98.8% -> 98.6% | No clear effect |
| Add RAIN inference | LLaMA 7B · IMDB | positive sentiment rate: 62.1% -> 82.1% | Helped |
| Add RAIN inference | Alpaca 7B and Vicuna 7B · IMDB | positive sentiment rate: 72.5% -> 94.4% (Alpaca 7B); 64.4% -> 89.1% (Vicuna 7B) | Helped |
| Add RAIN inference | LLaMA 7B, LLaMA-2 7B, and LLaMA 30B · HH | harmlessness rate: 64.4% -> 69.3% (LLaMA 7B); 81.2% -> 86.6% (LLaMA-2 7B); 82.2% -> 97.0% (LLaMA 30B) | Helped |
| Use 500 repeated vanilla samples and select with the LLM instead of one-time sampling | LLaMA 7B · HH | harmlessness rate: 64.4% -> 64.4% | No clear effect |
| Use 500 repeated vanilla samples and select with the LLM instead of one-time sampling | LLaMA 30B · HH | harmlessness rate: 82.2% -> 88.1% | Helped |
| Remove similarity update (ablation) | RAIN on LLaMA-2-chat 13B · AdvBench, GCG white-box attack | attack success rate: 19% -> 22% | Hurt |
| Remove similarity update and dynamic node addition (ablation) | RAIN on LLaMA-2-chat 13B · AdvBench, GCG white-box attack | attack success rate: 19% -> 25% | Hurt |
| Remove similarity update, dynamic node addition, and exploration encouragement (ablation) | RAIN on LLaMA-2-chat 13B · AdvBench, GCG white-box attack | attack success rate: 19% -> 27% | Hurt |
Surprising findings
- Repeated cherry-pick sampling (500 trials, self-selected) barely improves harmlessness for LLaMA 7B (64.4% unchanged) and only modestly for 30B, versus RAIN's large gains.
- Self-evaluation accuracy for smaller models is near chance (e.g., 52% for LLaMA v1 7B), yet RAIN still improves harmlessness.
Limitations
- Inference is about 4x slower than vanilla decoding on LLaMA models (time ratio 3.78x to 4.36x).
- Robustness claims are only against static GCG attacks, not adaptive attacks.
- Self-evaluation is less accurate for small models, so gains are smaller at small scale.
Experimental setup
Datasets: Anthropic HH AdvBench TruthfulQA IMDB
Benchmarks: HH AdvBench TruthfulQA IMDB
Metrics: harmlessness rate helpfulness rate attack success rate truthful and informative rate positive sentiment proportion time ratio self-evaluation accuracy
Baselines: Vanilla auto-regressive inference RLHF RLAIF Repeated cherry-pick sampling
Builds on: RLAIF PUCT superficial alignment hypothesis Sentence-BERT
Closest papers by meaning
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.90)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.89)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.89)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.89)
- Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.