Research map / Large language models / Alignment and preferences
Learning to summarize user information for personalized reinforcement learning from human feedback
HyunJi Nam, Yanming Wan, Mickel Liu, Peter F. Ahnn, Jianxun Lian, Natasha Jaques
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the summarizer jointly with reward model (vs fixed summarizer) (ablation) | PLUS · UltraFeedback-P2 (Qwen-0.5B) | reward model accuracy (%): 59.90 -> 69.40 | Improved |
| train the summarizer jointly with reward model (vs fixed summarizer) (ablation) | PLUS · UltraFeedback-P4 (Qwen-0.5B) | reward model accuracy (%): 56.89 -> 62.10 | Improved |
| train the summarizer jointly (vs fixed summarizer) (ablation) | PLUS · Pets OOD | reward model accuracy (%): 85.83 -> 93.67 | Improved |
| train the summarizer jointly (vs fixed summarizer) (ablation) | PLUS · Pets | reward model accuracy (%): 96.20 -> 99.80 | No clear effect |
| train the reward model (vs static reward model) with trained 3B summarizer (ablation) | PLUS · UltraFeedback-P2 | reward model accuracy (%): 53.45 -> 69.40 | Improved |
| replace GPT-4o summarizer with co-trained 3B summarizer (ablation) | PLUS · UltraFeedback-P2 | reward model accuracy (%): 59.30 -> 69.85 | Improved |
| add PLUS summaries instead of preference-only ICL | reward model · UltraFeedback-P2 (conversation data) | reward model accuracy (%): 58.55 -> 68.3 | Improved |
| add user guideline to the prompt (ablation) | PLUS · UltraFeedback-P2 | reward model accuracy (%): 69.40 -> 70.25 | Improved |
| add PLUS summary to GPT-4o judge | GPT-4o · PRISM controversial | preference accuracy (%): 52.2 -> 73 | Improved |
| add PLUS summary to GPT-4.1 judge on new users | GPT-4.1 · PRISM new users | preference accuracy (%): 74.35 -> 74.35 | No clear effect |
| add PLUS summarization (3B) in place of 8B ICL reward model | PLUS · UltraFeedback-P2 | reward model accuracy (%): 61.4 -> 66.9 | Improved |
| use PLUS instead of BTL on unseen PRISM users | reward model · PRISM held-out users | reward model accuracy (%): 62.9 -> 59.7 | Hurt |
Closest papers by meaning
- RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs (AAAI 2025 · similarity 0.92)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.91)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.89)
- Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.89)
- PersonalLLM: Tailoring LLMs to Individual Preferences (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
