Research map / Learning theory and algorithms / Combinatorial and search optimization

Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF

ICML 2025 · Learning theory and algorithms › Combinatorial and search optimization · approach: Multi-objective optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MOPO, a projection-based (Blackwell-approachability) framework that turns non-linear p-norm multi-objective RLHF into a sequence of linear-aggregation subproblems, extended to multi-group consensus and malfare objectives, with offline/online regret guarantees and a nearly training-free practical variant.

Problem

Multi-objective RLHF with non-linear (p-norm, p≤1) aggregation of per-objective rewards is computationally expensive (reward-based, retraining per aggregation), and multi-group preferences with group-specific weights lack a principled formulation.

Approach

Recast maximizing a non-linear p-norm aggregate as minimizing the distance from the expected reward vector to a convex target set; iteratively project along directions so each step reduces to a linear-aggregation RLHF subproblem (solvable by MOD/DPO-style methods), and extend to multi-group consensus and malfare objectives.

Key results

• On Harmless/Helpful with p=0.5, MOPO achieves distance 0.015 at alpha=(0.5,0.5) versus 0.078 for RS, 0.103 for MOD and 1.314 for AR.
• On Harmless/Humor (p=0.5, c=1.3), MOPO is not uniformly best; MOD wins at alpha=(0.3,0.7) (0.572 vs 0.578) and RS/MOD are close at several weights.
• In the multi-group setting, the variant max-min RLHF beats MOPO at alpha=(0.5,0.5) and (0.9,0.1) on Harmless/Humor, so MOPO is not uniformly better.

Measured effects reported in the paper

Surprising findings

• Naive non-linear reward aggregation (AR) performs poorly, attributed to vanishing gradients from clipping negative rewards.
• The proposed method does not dominate baselines in all settings; a max-min RLHF variant outperforms it at some weights in the multi-group experiments.

Limitations

• Theoretical guarantees rely on linear reward parameterization with known features and a BT model; practical experiments use neural reward models without these guarantees.
• Experiments are small-scale (three objectives, a single LLaMA2-7B reference, 7 iterations) and report distance scores rather than standard alignment benchmarks.
• Multi-objective weights and p are assumed or estimated via a softmax-selection model whose validity is not empirically tested.

Experimental setup

Datasets: Anthropic-HH

Benchmarks: Harmless/Helpful/Humor reward models (distance to target set)

Metrics: distance between reward vector and target set (lower is bett minimum objective value (max-min comparison)

Baselines: MORLHF RS MOD AR (direct non-linear reward aggregation) max-min RLHF

Builds on: DPO MOD max-min RLHF Blackwell approachability Rewarded soups

Closest papers by meaning

• Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.90)
• Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.90)
• Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.89)
• MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.89)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.89)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.89)