Research map / Optimization / Nonconvex and smooth optimization
Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates preference-based selection within the Pareto set as a smooth optimization over the simplex via the Pareto manifold, and proposes Pareto majorization-minimization with an O(ε0^-2) iteration bound to approximate preference stationarity.
Problem
Selecting a single preferred solution from the Pareto set of multiple strongly convex objectives, i.e., optimizing a preference function constrained to the (implicit, nonconvex, nonsmooth) Pareto set, with convergence guarantees for local methods.
Approach
Reparametrizes the Pareto set as a diffeomorphic image of the simplex (the Pareto manifold), turning the Pareto-constrained problem into a smooth problem over the simplex; defines preference stationarity and approximate versions, and runs majorization-minimization using implicit-gradient surrogates with approximate gradients from local Hessian and Jacobian information.
Key results
- PMM attains an eps0-approximate preference stationary point within O(eps0^-2) iterations under strong convexity and Lipschitz smoothness assumptions.
- Shows PNG stationarity is not necessary for preference optimality via a 2D quadratic counterexample where PNG stationary points converge to e1 while the optimum is the origin.
- Proves preference stationarity is necessary and that first-order-only stationarity conditions are trivial (cannot be necessary and non-trivial).
Surprising findings
- Any necessary first-order stationarity condition based only on local gradients is trivial, so second-order information is needed.
- The PNG stationarity notion from prior work can avoid preference optimal points.
Limitations
- Analysis requires strong convexity and Lipschitz smoothness of objectives and preference function.
- Guarantees are for local stationarity, not global preference optimality.
- Experiments are absent; theory only.
Experimental setup
Metrics: convergence rate iteration complexity
Baselines: Pareto navigating gradient descent
Builds on: Pareto navigating gradient descent majorization-minimization implicit function theorem
Closest papers by meaning
- An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.89)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.89)
- FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learning (NeurIPS 2024 · similarity 0.89)
- Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization (ICML 2025 · similarity 0.88)
- Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.