Research map / Large language models / Alignment and preferences
Group Preference Optimization: Few-Shot Alignment of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Group Preference Optimization (GPO), a meta-learned in-context transformer preference module attached to a frozen LLM that predicts group preferences from a few examples, and evaluates it on OpinionQA and GlobalOpinionQA alignment.
Problem
Aligning LLMs to the subjective preferences of many distinct groups when per-group preference data is scarce and per-group fine-tuning or prompting is expensive or ineffective.
Approach
Augment a base LLM with an independent in-context transformer, meta-trained across groups, that maps LLM embeddings of (prompt, response) pairs plus a few context preference labels to predicted group preferences; the predictor then drives re-ranking or policy optimization without updating the base LLM.
Key results
- GPO averages a 7.1% alignment-score gain over the In-context Finetune baseline on 22 US demographic groups in OpinionQA.
- GPO averages an 8.4% improvement over the second-best baseline across 14 countries in GlobalOpinionQA.
- GPO used about 4.7x less training time than In-context Finetune on an RTX A6000 for GlobalOpinionQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GPO's architecture with a standard autoregressive transformer using a causal mask and positional encoding (ablation) | GPO · GlobalOpinionQA | Alignment Score: 0.798 ± 0.007 -> 0.780 ± 0.009; 0.820 ± 0.004 -> 0.782 ± 0.004; 0.799 ± 0.015 -> 0.772 ± 0.006 | Hurt |
| use only the last-token embedding instead of averaging token embeddings (ablation) | GPO with Alpaca-7B embeddings · OpinionQA, meta train split of 80% | Alignment Score: 0.946 ± 0.007 -> 0.903 ± 0.014 | Hurt |
| concatenate last-token and average-token embeddings instead of using average-token embeddings (ablation) | GPO with Alpaca-7B embeddings · OpinionQA, meta train split of 80% | Alignment Score: 0.946 ± 0.007 -> 0.942 ± 0.009 | Hurt |
| add group meta-data embeddings (ablation) | GPO with Alpaca-7B · OpinionQA, meta train on 40% groups | Alignment Score: 0.920 ± 0.003 -> 0.900 ± 0.003 | Hurt |
| add group meta-data embeddings (ablation) | GPO with Alpaca-7B · OpinionQA, meta train on 60% groups | Alignment Score: 0.926 ± 0.013 -> 0.916 ± 0.017 | Hurt |
| add group meta-data embeddings (ablation) | GPO with Alpaca-7B · OpinionQA, meta train on 80% groups | Alignment Score: 0.946 ± 0.007 -> 0.926 ± 0.006 | Hurt |
| model target preference dependencies rather than assuming conditional independence (ablation) | GPO · preliminary experiments | Alignment performance | No clear effect |
| use GPO instead of In-Context Finetune | Alpaca-7B and Llama2-13B-chat · OpinionQA | Alignment Score: 7.1% | Helped |
| use GPO instead of the second-best baseline | Alpaca-7B and Llama2-13B-chat · GlobalOpinionQA | Alignment Score: 8.4% | Helped |
| use GPO instead of In-Context Finetune | GPO and In-Context Finetune · GlobalOpinionQA | training time: approximately 4.7 times more training time | Helped |
Surprising findings
- Few-shot prompting with group preference examples reduced alignment relative to the base LM, which the authors hypothesize is due to prompt format being out of distribution.
- Adding group meta-data embeddings to GPO lowered performance relative to GPO without them.
Limitations
- Evaluated only on multiple-choice survey data in English; long-form generation not validated.
- Survey data may not represent group populations; possible amplification of harmful preferences.
- Transformer is trained from scratch without pretrained backbone initialization.
Experimental setup
Datasets: OpinionQA GlobalOpinionQA
Benchmarks: OpinionQA GlobalOpinionQA
Metrics: alignment score (1-Wasserstein for OpinionQA, 1-Jensen-Shann individual alignment accuracy
Baselines: LM Base LM Steered Few-shot Prompt SFT per group Reward Model In-Context Finetune Uniform Distribution
Builds on: Neural processes / in-context meta-learning over datasets LoRA PPO DPO
Closest papers by meaning
- No Preference Left Behind: Group Distributional Preference Optimization (ICLR 2025 · similarity 0.92)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.91)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.91)
- Group Robust Preference Optimization in Reward-free RLHF (NeurIPS 2024 · similarity 0.91)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.91)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.