Research map / Image generation / Text-to-image diffusion
Inference-Time Personalized Alignment with a Few User Preference Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UserAlign, an inference-time personalized alignment method that elicits a user's preferences with a few pairwise comparisons over a fixed candidate pool, using logistic best-arm identification with a consistent (noise-free) version-space refinement.
Problem
Aligning generative model outputs to an individual user's preferences on the fly when only a small number of pairwise preference queries are affordable.
Approach
Combines a loss-based logistic-bandit confidence set with version-space elimination via intersecting halfspaces from observed pairwise duels, treating user feedback as consistent and noise-free, so the confidence set shrinks much faster and the method selects the best response from a pre-generated pool with few queries.
Key results
- On food64d, UserAlign (epsilon=0.0) reaches 98.67% win-rate at about 19.1 interaction steps, versus 90.0% for iidBest at 20 steps and 84.67% for the theoretical variant at 199 steps.
- In a human study with 960 MTurk participants, UserAlign at t=10 reached 89.2% win-rate on food64d with persona versus 78.3% for iidBest at t=10.
- On visual512d with human users, UserAlign reached 90.0% win-rate at t=20 with persona versus 81.5% for iidBest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use active informative pair selection with human pairwise feedback instead of random-pair IIDBest querying | UserAlign · food64d and visual512d, human users, with- and without-persona | win-rate (%): UserAlign exceeds IIDBest in all reported conditions | Helped |
| Add consistent-feedback halfspace constraints to the loss-based confidence set (ablation) | UserAlign · food64d, GPT-simulated users, epsilon=0.0 | win-rate (%): 84.67~(2.94) -> 98.67~(0.94) | Helped |
| Add consistent-feedback halfspace constraints to the loss-based confidence set (ablation) | UserAlign · travel64d, GPT-simulated users, epsilon=0.0 | win-rate (%): 76.67~(3.45) -> 100.00~(0.00) | Helped |
| Add consistent-feedback halfspace constraints to the loss-based confidence set (ablation) | UserAlign · visual512d, GPT-simulated users, epsilon=0.0 | win-rate (%): 59.33~(4.01) -> 94.67~(1.83) | Helped |
| Add consistent-feedback halfspace constraints to the loss-based confidence set (ablation) | UserAlign · dsp64d, GPT-simulated users, epsilon=0.0 | win-rate (%): 72.47~(1.15) -> 96.27~(0.49) | Helped |
Surprising findings
- The theoretically grounded loss-based confidence set (without consistency refinement) needs close to the full 199-step cap on most domains, showing that the practical noise-free refinement is what makes elicitation fast.
Limitations
- Assumes consistent, noise-free pairwise feedback, which may not hold for less stable user preferences.
- Relies on a fixed pre-generated response pool rather than adaptive generation.
- Treats each user question independently without leveraging historical preference data.
Experimental setup
Datasets: Domain Specific Preference (DSP) dataset GPT-generated personas and candidate pools
Benchmarks: food2d food64d travel64d visual512d dsp64d
Metrics: win-rate against zero-temperature baseline interaction cost (number of queries)
Baselines: Oracle Random iidBest UserAlign (loss-based confidence set, theoretical variant)
Builds on: Bradley-Terry-Luce preference model logistic bandits best-arm identification version space elimination
Closest papers by meaning
- PAD: Personalized Alignment of LLMs at Decoding-time (ICLR 2025 · similarity 0.90)
- MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Rethinking Personalized Generation: Test-time Alignment via Factorized Ranking Models (NeurIPS 2026 · similarity 0.89)
- PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model (ICML 2025 · similarity 0.88)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.88)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.