Research map / Large language models / Alignment and preferences

MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models

NeurIPS 2024 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MetaAligner, a policy-agnostic, SFT-trained weak-to-strong corrector that performs zero-shot multi-objective alignment on unseen objectives by editing policy outputs via text objective descriptions.

Problem

Existing multi-objective preference alignment methods depend on policy model parameters, requiring costly re-alignment for each new policy model, and cannot generalize to objectives unseen during training.

Approach

A stacked conditional seq-to-seq corrector that takes a frozen policy model's output plus text descriptions of target objectives and rewrites it toward higher preference; trained on dynamically reformulated multi-objective data with warm-up, equal-preference, and contrastive-preference SFT stages, enabling plug-and-play use on any policy (including closed models) and prompt-based zero-shot objecti

Key results

• MetaAligner-13B averages 82.67% win rate on HH-RLHF and 61.75% on UltraFeedback over LLaMA2-Chat-7B, outperforming MORLHF, MODPO, SFT and Aligner-7B.
• MetaAligner-1.1B reaches comparable average win rates to MORLHF/MODPO/SFT at 6.37%-48.71% of their GPU training hours (up to 93.63% savings).
• On six unseen objectives, MetaAligner-13B reaches an average 61.25% win rate, versus 48.5% for MetaAligner-7B.

Measured effects reported in the paper

Surprising findings

• Self-refinement by the policy model itself is competitive with some multi-objective baselines, but MetaAligner-7B still beats it on 6 of 7 objectives for LLaMA2-Chat-70B at about 1/10 inference cost.
• Aligning on additional objectives such as Fair can reduce win rates on Readable and Factual, indicating an alignment tax.

Limitations

• Stacking the MetaAligner module on policy models increases inference computation and deployment burden.
• Generalization to unseen objectives tested on only six objectives (up to 10 total aligned).
• Smaller MetaAligner models show limited gains against large policy models, with win-rate advantage decreasing as policy size grows.

Experimental setup

Datasets: HH-RLHF UltraFeedback IMHI

Benchmarks: HH-RLHF UltraFeedback IMHI

Metrics: win rate vs ground-truth response GPU training hours GPT-4 score 0-10

Baselines: MORLHF MODPO SFT (RiC) Aligner-7B Self-Refinement CDPO

Builds on: Aligner RiC MODPO MORLHF CDPO

Closest papers by meaning

• SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.91)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.91)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
• Meta-Learning Preferences for Multilingual LLM Alignment (NeurIPS 2026 · similarity 0.91)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.91)
• ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)