Research map / Large language models / Language, safety and interpretability
RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an LLM user summarizer with RL using a frozen LLM's future-activity prediction accuracy as reward (plus a length reward), yielding concise summaries that transfer to unseen tasks and datasets.
Problem
Long, noisy user activity histories are costly and hard for LLMs to use in personalization; existing summaries lack task-relevant context and no reference user summaries exist for supervised training.
Approach
Use the downstream prediction correctness of a frozen LLM (predicting the user's next activity, in multiple-choice form) as the reward for RL training of a summarizer, with a length reward and KL regularization, removing the need for reference summaries or a trained reward model.
Key results
- RLPF improves future activity prediction over Gemini Nano-2 zero-shot by about 13.4% on average and outperforms RLAIF by about 22% on average, while compressing context by 73.8% versus all activities.
- Across 19 unseen task/dataset cases, RLPF beats zero-shot in 16 cases, e.g. +44.33% on Amazon CDs future activity prediction.
- Auto Rater win rates vs zero-shot reach up to 84.59% on abstractiveness (Google Local Review).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train summaries with RLPF prediction feedback instead of zero-shot summarization | Gemini 1.0 Nano-2 summaries · Amazon Books Future Activity | accuracy: 0.664 -> 0.709 | Helped |
| Train summaries with RLPF prediction feedback instead of zero-shot summarization | Gemini 1.0 Nano-2 summaries · Google Local Review Future Activity | accuracy: 0.483 -> 0.560 | Helped |
| Train summaries with RLPF prediction feedback instead of zero-shot summarization | Gemini 1.0 Nano-2 summaries · MovieLens 2015 Future Activity | accuracy: 0.607 -> 0.719 | Helped |
| Train summaries with RLPF prediction feedback instead of zero-shot summarization | Gemini 1.0 Nano-2 summaries · Amazon Books Fav Category | recall@3: 0.594 -> 0.605 | Helped |
| Train summaries with RLPF prediction feedback instead of zero-shot summarization | Gemini 1.0 Nano-2 summaries · MovieLens 2015 Fav Category | recall@3: 0.774 -> 0.818 | Helped |
| Train summaries with RLPF prediction feedback instead of zero-shot summarization | Gemini 1.0 Nano-2 summaries · Google Local Review Fav Category | recall@3: 0.487 -> 0.559 | Helped |
| Train summaries with RLPF rather than RLAIF | Gemini 1.0 Nano-2 summaries · Amazon Books Fav Category | recall@3: 0.613 -> 0.605 | Hurt |
| Train summaries with RLPF prediction feedback | Gemini 1.0 Nano-2 summaries · MovieLens 2015 Future Activity using PaLM-2 XS evaluator | accuracy: 0.638 -> 0.741 | Helped |
| Use PaLM-2 XS rather than Gemini 1.0 Nano-2 as the policy model (ablation) | RLPF · MovieLens 2015 Fav Category | recall@3: 0.860 -> 0.849 | Hurt |
| Train on Future Activity Prediction rather than Favorite Genre Prediction as the reward task (ablation) | RLPF · MovieLens 2015 Future Activity | accuracy: +18.46% | Helped |
| Train on Favorite Genre Prediction rather than Future Activity Prediction as the reward task (ablation) | RLPF · MovieLens 2015 Favorite Genre Prediction | recall@3: +24.55% | Helped |
| Train on Rating Prediction rather than Future Activity Prediction as the reward task (ablation) | RLPF · MovieLens 2015 Rating Prediction | accuracy: +8.00% | Helped |
Surprising findings
- A supervised fine-tuned Gemini 1.0 Pro reached 94% on the training task but scored zero on other tasks, indicating overfitting while RLPF transferred.
- Optimizing for future activity prediction also improved intrinsic factuality, abstractiveness and readability without explicit criteria rewards.
Limitations
- Reward depends on a multiple-choice formulation with four options rather than exact-match prediction.
- Readability by Flesch-Kincaid was mixed after RLPF, and LLM auto-rater is the main intrinsic metric.
- Privacy and data leakage risks of using user data are acknowledged but not analyzed.
Experimental setup
Datasets: Amazon Review Books Google Local Review MovieLens 2015 MovieLens 2003 Amazon Movies and TV Amazon CDs and Vinyl Amazon Toys and Games Amazon Video Games
Benchmarks: Future Activity Prediction Favorite Genre/Category prediction Rating prediction Common City retrieval Review generation User age/gender/occupancy prediction
Metrics: accuracy recall@3 ROUGE-Lsum win rate (Auto Rater) token length
Baselines: Gemini 1.0 Nano-2 zero-shot Gemini 1.0 Pro zero-shot Gemini 1.0 Nano-2 few-shot Gemini 1.0 Nano-2 crafted prompt Direct RLAIF First X / Random X / Last X activities All activities
Builds on: REINFORCE RLAIF Adafactor
Closest papers by meaning
- HYDRA: Model Factorization Framework for Black-Box LLM Personalization (NeurIPS 2024 · similarity 0.87)
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.87)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.87)
- Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-Horizon Agents (NeurIPS 2026 · similarity 0.87)
- Rethinking Personalization in Large Language Models at the Token Level (ICML 2026 · similarity 0.87)
- Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.