Research map / Large language models / Alignment and preferences
MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MATO, a training-free multi-objective personalized alignment method that recovers per-preference rewards from the backbone LLM via prompt contrast, closed-form weight reoptimization, and FTRL-based online token-distribution optimization at test time.
Problem
Aligning LLMs with multiple, possibly conflicting, natural-language user preferences without training or external per-preference reward models, with steerable trade-offs among objectives.
Approach
Derives token-level preference rewards from prompt-conditioned log-ratios of the backbone LLM, adapts objective weights in closed form toward under-served objectives (KL-regularized to the user prior), and refines the token distribution with online FTRL-style optimization.
Key results
- On the Multifaceted dataset with GPT-4o judge, MATO reaches AMR 0.75, APS 4.30, Worst 3.42 on Qwen3-8B, beating OPAD (0.74, 4.28, 3.39).
- On Llama-3.2-1B, MATO outperforms OPAD by +0.37 on Worst and +0.24 on APS.
- Ablation on Qwen3-0.6B: removing both modules drops AMR from 0.27 to 0.21 and APS from 2.99 to 2.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add weight optimization (dynamic closed-form reweighting of objective weights) (ablation) | MATO on Qwen3-0.6B · Multifaceted subset | APS: 2.88 -> 2.99 | Helped |
| add online optimization (FTRL token-distribution refinement) (ablation) | MATO on Qwen3-0.6B · Multifaceted subset | APS: 2.83 -> 2.99 | Helped |
| add both weight optimization and online optimization (ablation) | MATO on Qwen3-0.6B · Multifaceted subset | Worst: 1.62 -> 1.87 | Helped |
| use MATO test-time optimization instead of prompt-based baselines | Multifaceted, four backbone LLMs · Multifaceted (GPT-4o judge) | AMR: e.g. Llama-3.2-1B 0.39 vs OPAD 0.36 | Helped |
| add test-time optimization decoding (MATO) over OPAD on Llama-3.2-1B | Llama-3.2-1B · Multifaceted (GPT-4o judge) | Worst: 2.25 vs 1.88 (+0.37) | Helped |
| combine preferences into a single prompt (Linear Alignment, CoS, OPAD style methods) vs. Preference Prompt | Qwen3-8B · Multifaceted (GPT-4o judge) | APS: 4.19 for Preference Prompt and Linear Alignment | No clear effect |
| decoding-based distribution steering (Linear Alignment, CoS, OPAD) relative to Preference Prompt on small model | Llama-3.2-1B · Multifaceted (GPT-4o judge) | APS: 2.78 vs 3.07 for Linear Alignment vs Preference Prompt | Hurt |
| single-objective online learning (Amulet) applied to multi-objective setting | Multifaceted, all backbones · Multifaceted (GPT-4o judge) | APS: Amulet 2.92 on Llama-3.1-8B vs Preference Prompt 3.97 | Hurt |
| MATO test-time optimization vs training-based baselines in Pareto steerability | HH-RLHF helpfulness-humor trade-off · HH-RLHF steerability | Pareto front quality | Helped |
| MATO decoding latency vs. Amulet and OPAD | MATO · Efficiency | inference seconds/token: 0.16 vs 0.10 (OPAD), 0.15 (Amulet) | Hurt |
Surprising findings
- Linear Alignment, CoS, and OPAD perform worse than the simple Preference Prompt baseline on the smallest model (Llama-3.2-1B), suggesting the combined signal misguides optimization.
- Training-based methods (MORLHF, RiC, MOD) give smoother Pareto fronts but lower overall alignment than training-free methods, attributed to train-test shift and reward hacking.
Limitations
- Per-token latency is higher than Amulet/OPAD (0.16 s/token vs 0.10-0.15), with extra cost from T online optimization steps.
- Training-based baselines could not be applied to the Multifaceted dataset due to the large number of preferences, so comparisons are limited to the HH-RLHF steerability setting.
- Reward discovery relies on prompt conditioning as an approximation of the single-objective optimal policy, so approximation errors are possible.
Experimental setup
Datasets: Multifaceted HH-RLHF
Benchmarks: Multifaceted HH-RLHF
Metrics: all preference match rate average preference score worst preference score Pareto front
Baselines: Base LLM Preference Prompt Linear Alignment CoS Amulet OPAD MORLHF RiC MOD
Builds on: KL-regularized RL closed form DPO Linear Alignment Amulet OPAD
Closest papers by meaning
- PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model (ICML 2025 · similarity 0.91)
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.91)
- Robust Multi-Objective Preference Alignment with Online DPO (AAAI 2025 · similarity 0.90)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.90)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.90)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.