Research map / Reinforcement learning / RL from feedback
Alignment of Large Language Models with Constrained Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAID, an iterative dual-based (multi-shot) alignment method that alternates Lagrangian LLM policy maximization (via DPO) with dual subgradient descent, and proves primal-dual and optimality gap bounds up to an LLM parametrization gap, validated on PKU-SafeRLHF and HH-RLHF.
Problem
Constrained LLM alignment in parameter space: maximize a primary reward (helpfulness) subject to constraints on secondary utilities (safety, humor, etc.), where primal-dual policy gradient methods often fail to converge and one-shot dual methods are not optimal in LLM parameter space.
Approach
Iterate between Lagrangian maximization of the LLM policy (via DPO on pseudo-preferences from a composite reward) and dual descent on the multipliers, warm-started from a one-shot solution; theory bounds the primal-dual gap and reward/constraint optimality gaps by the LLM parametrization gap.
Key results
- Multi-shot CAID tracks safety constraint thresholds {3,...,9} more closely than one-shot, which falls short for small b and overshoots for large b.
- Multi-shot achieves a higher empirical helpfulness-safety Pareto curve than the one-shot method and DPO_s/DPO_h baselines.
- In the multi-constraint setting (harmlessness and humor, 14 combinations), both constraints are generally satisfied close to the reference lines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iterative multi-shot dual updates instead of one-shot alignment | Safety alignment of Alpaca-7b-reproduced · PKU-SafeRLHF-30K test | constraint satisfaction across safety thresholds | Helped |
| Use iterative multi-shot dual updates instead of one-shot alignment | Safety alignment of Alpaca-7b-reproduced · PKU-SafeRLHF-30K test | helpfulness-safety trade-off | Helped |
| Use iterative multi-shot dual updates instead of one-shot alignment | Safety alignment of Alpaca-7b-reproduced · PKU-SafeRLHF-30K test | safety score distribution | Helped |
| Train with safety as the sole objective | DPO safety-only · PKU-SafeRLHF-30K test | helpfulness-safety trade-off | Mixed |
| Train with helpfulness as the sole objective | DPO helpfulness-only · PKU-SafeRLHF-30K test | helpfulness-safety trade-off | Mixed |
| Apply iterative dual-based alignment to two constraints | LLaMA2-7B alignment · Anthropic HH-RLHF test | harmlessness and humor constraint satisfaction | Helped |
| Increase harmlessness and humor constraint levels | LLaMA2-7B alignment · Anthropic HH-RLHF test | helpfulness improvement | Hurt |
Limitations
- Experiments limited to 7B models due to compute and datasets.
- Theory focuses on optimality; sample complexity and robustness not analyzed.
- Results depend on a parametrization gap assumption and strict feasibility/strong convexity assumptions.
Experimental setup
Datasets: PKU-SafeRLHF-30K Anthropic HH-RLHF
Benchmarks: PKU-SafeRLHF-30K test split Anthropic HH-RLHF test split (2000 prompts)
Metrics: average helpfulness score average safety score constraint satisfaction (actual vs. threshold b) helpfulness-safety Pareto trade-off harmlessness and humor improvements
Baselines: one-shot dual-based method DPO_s DPO_h
Builds on: one-shot safety alignment DPO stepwise alignment safe RLHF
Closest papers by meaning
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.92)
- L3Ms — Lagrange Large Language Models (ICLR 2025 · similarity 0.91)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
- One-Shot Safety Alignment for Large Language Models via Optimal Dualization (NeurIPS 2024 · similarity 0.90)
- Oracle-Robust Online Alignment for Large Language Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.