Research map / Large language models / Alignment and preferences
Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-Adapter, which customizes an RLHF-pretrained LLM to new human preferences by learning a residual Q-function via an adapter (LoRA) under the Bradley-Terry loss, preserving original capabilities.
Problem
Customizing a pre-trained, RLHF-aligned LLM to new human preferences without forgetting its original capabilities, when the original reward function and training data are unavailable.
Approach
Cast LLM customization as maximizing the sum of an unknown pre-training reward r1 and a new preference reward r2; use residual Q-learning so the customized policy is computed from the frozen pre-trained LLM plus a learned residual Q-function, and learn that Q-function directly from preference pairs via the Bradley-Terry loss (no explicit reward model), implemented as a LoRA adapter.
Key results
- On DSP-Academy, Q-Adapter keeps MMLU at 66.40 vs 67.68 for the base model, while SFT drops to 63.49.
- Length-controlled win rate against SFT on DSP-Entertainment reaches 63.77 for Q-Adapter vs 46.66 for Replay under GPT-4o judging.
- Q-Adapter achieves the best r2 (new preference) win rate in all four DSP domains, e.g., 63.77 in Entertainment, and the trend holds under Claude-3.5-Sonnet and DeepSeek-chat judges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-Adapter instead of SFT, Replay, or policy regularization for domain customization | Llama-3.1-8B-Instruct · DSP domains | general benchmark scores | Helped |
| Use Q-Adapter instead of SFT, Replay, or policy regularization for domain customization | Llama-3.1-8B-Instruct · DSP domains | win rate against SFT | Helped |
| Customize the base model with any reported DSP method | Llama-3.1-8B-Instruct · DSP domains | general benchmark scores | Hurt |
| Use Q-Adapter instead of DPO or Replay during training | Llama-3.1-8B-Instruct · DSP Academy checkpoints | MMLU accuracy | Helped |
| Use Q-Adapter rather than PR (DPO) when customizing helpfulness to harmlessness | Llama-3.1-8B-Instruct trained on HH-RLHF helpful data · HH-RLHF helpful and harmless | win rate against SFT | Helped |
| Increase alpha_0 within the reported range 0.03 to 0.001 (ablation) | Q-Adapter · HH-RLHF | win rate | Mixed |
| Use a larger alpha_0 such as 0.1 (ablation) | Q-Adapter · HH-RLHF | training stability | Hurt |
| Add L2 regularization to constrain the estimated reward range (ablation) | Q-Adapter · MMLU | MMLU score | Helped |
| Vary beta from 0.01 to 1.0 (ablation) | Q-Adapter · MMLU | MMLU score | No clear effect |
Surprising findings
- PR (PPO) collapses on DSP-Business (GSM8k 23.06, IFEval 10.49), indicating unstable PPO training.
- Reported Llama-3.1 base-model scores differ from the official Llama 3 paper due to a different evaluation configuration.
Limitations
- Assumes the base LLM was pre-trained with RLHF; not applicable to SFT-only models as stated by authors.
- Only one round of customization was performed; continual customization is left to future work.
- Inference requires two forward passes (base and adapter), increasing cost relative to a single model.
Experimental setup
Datasets: Domain-Specific Preference (DSP) HH-RLHF Alpaca 52K
Benchmarks: MMLU MMLU Pro GSM8k BBH IFEval AlpacaEval
Metrics: accuracy normalized accuracy length-controlled win rate
Baselines: SFT Replay PR (DPO) PR (PPO)
Builds on: residual Q-learning IPL DPO maximum entropy RL LoRA
Closest papers by meaning
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.88)
- Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs (ICLR 2025 · similarity 0.88)
- Reverse Engineering Human Preferences with Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.