Research map / Large language models / Alignment and preferences
ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models Using Pareto High-Quality Data
Haoran Gu, Handing Wang, Yi Mei, Mengjie Zhang, Yaochu Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| represent preferences as directions with Pareto high-quality data instead of linear scalarization | ParetoHqD · Helpful Assistant (HH-RLHF, 2 rewards) | hypervolume: 0.6325 (RiC) -> 0.7526 | Improved |
| use ParetoHqD instead of linear-scalarization baselines | multiobjective alignment · Helpful Assistant (HH-RLHF, 2 rewards) | collapse rate (%): 41.88 (RiC) -> 7.03 | Improved |
| small Pareto high-quality training sets and data augmentation from only M+1 models | ParetoHqD · Helpful Assistant (HH-RLHF, 2 rewards) | training GPU hours: 132.47 (RiC) -> 55.87 | Improved |
| add stage 2 data augmentation and further fine-tuning (ablation) | ParetoHqD · Helpful Assistant (HH-RLHF, 2 rewards) | Pareto front quality | Improved |
| replace linear scalarization selection with Pareto filtering and preference directions in SFT data selection (ablation) | ParetoHqD stage 1 · Helpful Assistant (HH-RLHF), concave-front sub-dataset | Pareto front coverage | Improved |
| replace linear scalarization with preference directions in stage 1 data selection (ablation) | ParetoHqD stage 1 · Helpful Assistant (HH-RLHF) | collapse rate (%): 6.74 -> 6.11 | No clear effect |
| train on a small high-quality top-k set instead of the whole scalarized dataset (ablation) | SFT · Helpful Assistant (HH-RLHF) | collapse rate (%): 35.74 -> 6.74 | Improved |
| scalarized SFT on the full set versus the base model (ablation) | Llama-2 7B · Helpful Assistant (HH-RLHF) | collapse rate (%): 18.63 -> 35.74 | Hurt |
Closest papers by meaning
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.92)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.91)
- MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.90)
- PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model (ICML 2025 · similarity 0.89)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.89)
- Robust Multi-Objective Preference Alignment with Online DPO (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
