Research map / Large language models / Alignment and preferences
PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PARM, a single preference-conditioned autoregressive reward model trained with bilinear low-rank adaptation (PBLoRA) that guides frozen LLMs at test time for multi-objective alignment, cutting inference cost versus GenARM's k separate ARMs.
Problem
Multi-objective test-time alignment of frozen LLMs to multi-dimensional user preference vectors; existing GenARM trains one ARM per preference dimension, causing inference overhead and misalignment between guidance and preferences.
Approach
A single autoregressive reward model is trained jointly across all preference dimensions, with a preference-conditioned bilinear low-rank adapter (B W(p) A, W depending on the preference vector) so one ARM traces the Pareto set and guides frozen LLMs via token-level rewards.
Key results
- On safety alignment with Alpaca-7B, PARM reaches HV 113.38 and MIP 2.59 versus GenARM's 99.34 and 0.80.
- With a 1.1B PARM guiding LLaMA-2-7B-Chat on helpful assistant, PARM gets HV 82.12 vs GenARM 44.38, with inference time 38.96s vs 48.39s and about 1.53M vs 4.59M reward-model parameters.
- Weak-to-strong: PARM-7B guiding Alpaca-65B yields HV 121.73 vs GenARM 114.76 and MOD-w2s 96.57.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GenARM with PARM | Alpaca-7B safety alignment · PKU-SafeRLHF-10K test | HV: 99.34 -> 113.38 | Helped |
| replace GenARM with PARM | Alpaca-7B safety alignment · PKU-SafeRLHF-10K test | MIP: 0.80 -> 2.59 | Helped |
| use PARM-7B rather than GenARM to guide the frozen 65B model | Alpaca-65B safety alignment · PKU-SafeRLHF-10K test | HV: 114.76 -> 121.73 | Helped |
| use PARM-7B rather than GenARM to guide the frozen 65B model | Alpaca-65B safety alignment · PKU-SafeRLHF-10K test | MIP: 1.81 -> 3.46 | Helped |
| replace GenARM with PARM | LLaMA-2-7B-Chat helpful assistant · HH-RLHF test | HV: 44.38 -> 82.12 | Helped |
| replace GenARM with PARM | LLaMA-2-7B-Chat helpful assistant · HH-RLHF test | MIP: 0.93 -> 1.42 | Helped |
| use a single PARM instead of three GenARM reward models | LLaMA-2-7B-Chat helpful assistant · HH-RLHF test; generating 512 tokens | inference time (seconds): 48.39 -> 38.96 | Helped |
| use PBLoRA instead of three separate LoRA ARMs | helpful assistant reward models · HH-RLHF test | learnable parameters (× 10^6): 4.59 -> 1.53 | Helped |
| replace default PBLoRA with SVD-LoRA (rank 8) (ablation) | PARM · Safety alignment ablation | HV: 113.38 -> 101.81 | Hurt |
| replace default PBLoRA with SVD-LoRA (rank 8) (ablation) | PARM · Safety alignment ablation | MIP: 2.59 -> 1.62 | Hurt |
| remove the preference-agnostic component (ablation) | PBLoRA · Safety alignment ablation | HV: 113.38 -> 104.42 | Hurt |
| remove the preference-agnostic component (ablation) | PBLoRA · Safety alignment ablation | MIP: 2.59 -> 2.38 | Hurt |
Limitations
- Evaluated only on two-dimension safety and three-dimension helpfulness/harmlessness/humor settings with oracle reward models.
- Results depend on preference-vector sampling and oracle reward models rather than human judgments.
Experimental setup
Datasets: PKU-SafeRLHF-10K HH-RLHF
Benchmarks: PKU-SafeRLHF-10K test split HH-RLHF test split
Metrics: Hypervolume (HV) Mean Inner Product (MIP) inference time
Baselines: Rewarded Soups MOD GenARM MOD-w2s SVD-LoRA
Builds on: GenARM LoRA SVD-LoRA DPO
Closest papers by meaning
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.91)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.91)
- MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.91)
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.91)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.