Research map / Large language models / Alignment and preferences
Aligner: Efficient Alignment by Learning to Correct
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Aligner, a small model-agnostic seq2seq module that takes a query and an upstream answer and outputs a corrected answer, trained on preference-derived correction data to plug into any LLM without RL.
Problem
Existing alignment methods (SFT, RLHF, DPO) are resource-heavy and require access to model parameters, which is ill-suited to rapidly changing alignment requirements and API-based models.
Approach
Learn correctional residuals between original and corrected answers with a small conditional seq2seq model that is stacked on top of any upstream LLM's outputs, so alignment is handled by a plug-and-play module rather than by modifying the upstream model.
Key results
- Aligner-7B averages +21.9% helpfulness (abstract says 68.9%) and +23.8% harmlessness across 11 upstream models in the 3H evaluation.
- Aligner-7B boosts GPT-4 helpfulness by 17.5% and harmlessness by 26.9%.
- Stacking Aligner-2B on GPT-4 Turbo raises Alpaca-Eval LC Win Rate from 55.0% to 58.3%, above the reported 57.5% of GPT-4 Omni.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Aligner-7B to upstream models | 11 upstream LLMs · E-Dialogue | Empathy improvement: 82.5% | Helped |
| Apply Aligner-7B to upstream models | 11 upstream LLMs · BeaverTails and HarmfulQA | Helpfulness and harmlessness improvement: 27.6%; 21.2%; 16.1%; 26.4% | Mixed |
| Stack Aligner-2B on GPT-4 Turbo | GPT-4 Turbo · Alpaca-Eval | LC Win Rate: 55.0% to 58.3% | Helped |
| Use identity-mapping warm-up before Q-A-C training (ablation) | Aligner · BeaverTails and HarmfulQA | Performance | Helped |
| Vary warm-up proportion; performance peaks at 10k to 50k (ablation) | Aligner warm-up · BeaverTails and HarmfulQA | Performance: 10k to 50k | Mixed |
| Use Aligner-7B instead of CAI without training | GPT-4 · Helpfulness | Helpfulness improvement: +20.01% vs +17.47% | Hurt |
| Use Aligner-7B instead of Self-Critique | GPT-4 · Helpfulness | Helpfulness improvement: +26.56% vs +17.47% | Hurt |
| Use Aligner-7B instead of CAI without training | GPT-4 · Harmlessness | Harmlessness improvement: +9.65% vs +26.88% | Helped |
| Use Aligner-7B instead of Self-Critique | GPT-4 · Harmlessness | Harmlessness improvement: +15.30% vs +26.88% | Helped |
| Train Aligner using weak-to-strong corrected labels with SFT | Alpaca2-13B · BeaverTails and HarmfulQA | Average helpfulness improvement: +28.4% | Helped |
| Train with weak-to-strong corrected labels using RLHF or DPO rather than SFT | Alpaca-7B · BeaverTails and HarmfulQA | Average helpfulness improvement: +14.0%; -38.9%; -51.3% | Hurt |
Surprising findings
- Llama2-7B-Chat plus Aligner-7B outperforms the larger Llama2-13B-Chat on BeaverTails and HarmfulQA helpfulness/harmlessness GSB comparisons.
- Aligner-7B gives a helpfulness gain on the Alpaca2 SFT setting, while RLHF and DPO fine-tuned weak-to-strong models often lose helpfulness versus the original strong model.
Limitations
- Adds an extra model at inference, increasing inference cost and sequential latency.
- Evaluation relies heavily on GPT-4 and crowdsourced judgments with modest annotator consistency (80-85%).
- Corrections are principle-based and the paper focuses on single-turn dialogue; multi-turn and domain-specific control are left to future work.
Experimental setup
Datasets: HH-RLHF PKU-SafeRLHF Ultra-Feedback E-Dialogue DialogSum BeaverTails HarmfulQA TruthfulQA
Benchmarks: E-Dialogue DialogSum BeaverTails HarmfulQA TruthfulQA Alpaca-Eval
Metrics: helpfulness win-rate improvement harmlessness win-rate improvement truthfulness reliability improvement LC Win Rate
Baselines: GPT-4 base Constitutional AI Self-Critique SFT RLHF DPO
Builds on: SFT RLHF DPO ResNet residual learning Constitutional AI weak-to-strong generalization
Closest papers by meaning
- Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction (AAAI 2025 · similarity 0.93)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.92)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.91)
- Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.91)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.91)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.