Research map / Large language models / Alignment and preferences

Rethinking Personalized Generation: Test-time Alignment via Factorized Ranking Models

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows personalized generation has large best-of-N headroom that generalist reward models fail to exploit, and proposes a ~2.8M-parameter MLP ranker on frozen Qwen2.5-7B hidden states, trained pointwise on ROUGE-L of on-policy candidates, that beats 8B reward models on nine LaMP, LaMP-QA and XRec datasets.

Problem

Personalized generation for LLMs: standard alignment optimizes monolithic preferences, and reward-model-based best-of-N selection fails to exploit the personalization headroom in candidate pools while being expensive to run on large pools.

Approach

Decouple selection from understanding: train a tiny MLP over the generator's own final-layer last-token hidden states of query, user profile, and candidate, using pointwise regression on within-pool standardized ROUGE-L against user-authored references, and optionally use its gradient to steer decoding at uncertain tokens.

Key results

• On Best-of-64, the 2.8M ranker outperforms the best of four 8B generalist reward models on all nine datasets, by 0.010 (Tweet) to 0.048 (Amazon) ROUGE-L.
• The finetuned 8B Skywork reward model leads the 30M ranker by +0.019 ROUGE-L on Scholarly; the ranker recovers 45% to 94% of its gain over a single sample.
• Scoring 64 candidates takes 0.098 ms with the 2.8M ranker versus 2.4 s mean for 8B reward models (four orders of magnitude lower latency).

Surprising findings

• Generalist 8B reward models stay flat or decline as the candidate pool grows, and on long-form QA never beat their N=1 score.
• Pairwise and listwise ranking objectives perform comparably to pointwise MSE, so the loss is not the source of gains.

Limitations

• Evaluation relies on lexical overlap (ROUGE-L/BLEU) with user-authored references; no human study of style consistency.
• Ranker is trained on ROUGE-L targets and the gain under BLEU is partly transfer; within-pool agreement with the metric is only moderate.
• Guided decoding recovers only a modest part of the selection gain; most of the headroom requires scoring a pool.

Experimental setup

Datasets: LaMP News LaMP Scholarly LaMP Tweet LaMP-QA Art LaMP-QA Lifestyle LaMP-QA Society XRec Amazon XRec Yelp XRec Google Personalized RewardBench

Benchmarks: LaMP LaMP-QA XRec

Metrics: ROUGE-L ROUGE-1 BLEU BERTScore

Baselines: Skywork-Reward-V2-Llama-3.1-8B internlm2-7b-reward URM-LLaMa-3.1-8B ArmoRM-Llama3-8B-v0.1 PAL VPL PReF LoRe GPO SynthesizeMe

Builds on: Best-of-N sampling Bradley-Terry reward modeling FUDGE LaMP XRec

Closest papers by meaning

• Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures (NeurIPS 2026 · similarity 0.89)
• MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.89)
• Inference-Time Personalized Alignment with a Few User Preference Queries (NeurIPS 2025 · similarity 0.89)
• Rethinking Personalization in Large Language Models at the Token Level (ICML 2026 · similarity 0.89)
• GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.89)
• PAD: Personalized Alignment of LLMs at Decoding-time (ICLR 2025 · similarity 0.88)