Research map / Large language models / Alignment and preferences
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates LLM alignment as Bayesian inverse RL and proposes AVA, an approximated variational objective (AVRIL-based) that jointly learns per-token rewards and policy from preference or demonstration data.
Problem
Existing RLHF and DPO alignment only models reward differences between chosen and rejected responses, not the true reward of each demonstration, and assumes reward only at sentence end, leading to underused training signal and possible reward hacking.
Approach
Treat the reward as a posterior conditioned on demonstrations and optimize an approximated variational reward imitation learning (AVRIL) objective over token-level states and actions, using a TQR architecture (Q-value head plus Gaussian reward head with attention-based reward weights), with a contrastive variant for preference data.
Key results
- AVA-p reaches 72.30 (GPT-2 vs BART) / 74.84 reward accuracy on Helpful with BART, above Bradley-Terry (69.39 / 67.56) and Pref-Trans (71.37 / 72.37).
- PPO with AVA-p reward beats PPO with Bradley-Terry on Summary with 18.5% win vs 15.2% lose (win 18.5 vs lose 15.2), and PPO w/ Pref-Trans with 33.9 win vs 31.5 lose.
- AVA-p direct optimization beats DPO on Harmless 13.7 win vs 12.5 lose and on Summary 44.9 win vs 44.1 lose, with small margins.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AVA-p rather than Pref-Trans for reward modeling | Reward modeling · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | reward accuracy: 70.27 vs 70.26; 72.30 vs 71.32; 72.37 vs 71.37; 74.84 vs 72.37; 61.79 vs 59.31; 64.31 vs 56.91 | Helped |
| Use AVA-p reward model instead of Pref-Trans reward model for PPO | PPO fine-tuning · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | win rate: 9.0; 2.6; 33.9 | Mixed |
| Use AVA-p instead of DPO for direct preference optimization | Direct optimization · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | win rate: 13.7; 14.4; 44.9 | Mixed |
| Use AVA-d instead of AfD-based alignment | Direct optimization from demonstrations · Alpaca-GPT-4 and Math-GPT-4o | win rate: 57.2; 56.5; 44.3; 45.4 | Mixed |
| Add reward weighting to the TQR architecture (ablation) | AVA-p · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | reward accuracy: 70.06 -> 70.27; 70.73 -> 72.30; 69.81 -> 72.37; 69.32 -> 74.84; 60.55 -> 61.79; 58.89 -> 64.31 | Helped |
| Include negative demonstrations in the AVA-p objective (ablation) | AVA-p · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | reward accuracy: 70.54 -> 70.27; 70.36 -> 72.30; 69.75 -> 72.37; 69.15 -> 74.84; 62.06 -> 61.79; 58.65 -> 64.31 | Mixed |
| Include the inverse-reinforcement-learning TD-error constraint and reward prior (ablation) | AVA-p · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | reward accuracy: 69.48 -> 70.27; 67.46 -> 72.30; 68.87 -> 72.37; 65.38 -> 74.84; 58.96 -> 61.79; 58.46 -> 64.31 | Helped |
| Add the CER auxiliary objective (ablation) | AVA-p · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | reward accuracy: 70.06 -> 70.27; 70.73 -> 72.30; 69.81 -> 72.37; 69.32 -> 74.84; 60.55 -> 61.79; 58.58 -> 64.31 | Mixed |
| Initialize the Q-value head from the pretrained policy rather than from scratch (ablation) | AVA-p · Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary | reward accuracy: 68.67 -> 70.27; 68.69 -> 72.30; 68.51 -> 72.37; 67.60 -> 74.84; 61.25 -> 61.79; 57.76 -> 64.31 | Helped |
Surprising findings
- AVA-d trained only on chosen half of preference data achieves the best reward accuracy on 2 of 6 tasks.
- Gains over DPO in direct optimization are small (e.g., Summary 44.9 win vs 44.1 lose).
Limitations
- Gains over baselines are often small and no significance tests are reported.
- Reward modeling pipeline requires RL fine-tuning which is costly.
- Reward-model experiments use small GPT-2 and BART backbones rather than LLMs.
Experimental setup
Datasets: Anthropic-Harmless Anthropic-Helpful OpenAI-Summary Alpaca-GPT-4 Math-GPT-4o
Benchmarks: Anthropic-Harmless Anthropic-Helpful OpenAI-Summary Alpaca-GPT-4 Math-GPT-4o
Metrics: reward accuracy win rate win/tie/lose rate
Baselines: Bradley-Terry Preference Transformer DPO AfD SFT PPO
Builds on: AVRIL Preference Transformer DPO Bradley-Terry AfD
Closest papers by meaning
- Imitating Language via Scalable Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)
- SALMON: Self-Alignment with Instructable Reward Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.