Research map / Large language models / Language, safety and interpretability
PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PersonalSum, a manually annotated Norwegian news dataset of personalized summaries from crowd users with profiles and source sentences, and benchmarks GPT-3.5, Gemini 1.0 Pro and NorwAI-Mixtral with few-shot prompts using entity, plot and position factors.
Problem
Existing summarization datasets use generic or pseudo-personalized references from journalists or controllable settings, so there is little human-annotated data on whether LLM summaries meet ordinary readers' individual interests.
Approach
Crowdsourced Norwegian news annotations where each ordinary user writes a summary of articles they choose to read, with source sentences and user profiles, plus few-shot prompting that injects each user's history and entity/plot/position factors to test personalization.
Key results
- On PersonalSum 2-shot, GPT-3.5 Turbo's best ROUGE-1 was 38.43 with all factors versus 37.90 for generic summaries, a small gain.
- On Topic-centric PersonalSum 5-shot, all personalized prompts beat generic summaries for GPT-3.5 Turbo, with Plot reaching ROUGE-1 40.22 versus 37.61 generic.
- Human evaluation on 50 samples found Fleiss' kappa of 0.66 to 0.98 across models and settings, indicating substantial to almost perfect agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use personalized user history and factor-guided prompting instead of generic prompting | LLMs generating summaries on PersonalSum · PersonalSum | summarization metrics | Mixed |
| Use five user-history examples rather than two | LLMs generating summaries on Topic-centric PersonalSum · Topic-centric PersonalSum | summarization metrics | Helped |
| Increase user-history examples from five to ten | LLMs generating summaries on Topic-centric PersonalSum · Topic-centric PersonalSum | summarization metrics | Hurt |
| Add plot information to the prompt (ablation) | GPT-3.5 Turbo · Topic-centric PersonalSum | Rouge-1: 37.61 -> 40.22 | Helped |
| Add entity and article-position information to the prompt (ablation) | Gemini 1.0 Pro · PersonalSum | Rouge-1: 35.21 -> 35.91 | Helped |
| Use all personalized factors rather than generic prompting (ablation) | GPT-3.5 Turbo · PersonalSum | Rouge-1: 37.90 -> 38.43 | Helped |
| Add plot and article-position information to the prompt (ablation) | NorwAI-Mixtral-8x7B-instruct · Topic-centric PersonalSum | Rouge-1: 33.40 -> 35.75 | Helped |
| Use user history to guide summaries rather than generic prompting | LLMs generating summaries on Topic-centric PersonalSum · Topic-centric PersonalSum | summarization metrics | Helped |
| Increase the number of user-history examples in the prompt | LLMs generating summaries on PersonalSum · PersonalSum | summarization metrics | Hurt |
| Use prompts that explicitly include entities, plots, or article-structure preferences | LLMs generating summaries · PersonalSum human evaluation | alignment with personalized content needs | Helped |
Surprising findings
- Adding more user history (10-shot) generally performed worse than 2-shot on PersonalSum, suggesting history introduces noise.
- Llama3-70B-Instruct produced much lower scores on Topic-centric PersonalSum (ROUGE-1 about 17-18 for personalized prompts versus 35.98 generic), often generating English summaries.
Limitations
- Small dataset (1,099 personalized summaries from 441 articles; 276 on topic-centric subset) limits training and generalization.
- Annotators could not choose the articles they annotated themselves.
- BERTScore struggled to distinguish between prompting conditions.
Experimental setup
Datasets: PersonalSum Topic-centric PersonalSum
Benchmarks: PersonalSum Topic-centric PersonalSum
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BERTScore entailment score Fleiss' kappa human consistency and coherence ratings
Baselines: Generic summary prompting Direct history prompting
Builds on: PENS Controllable text summarization
Closest papers by meaning
- Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.88)
- SummPilot: Bridging Efficiency and Customization for Interactive Summarization System (AAAI 2025 · similarity 0.88)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.87)
- Learning from Failure: Improving Meeting Summarization without Good Samples (AAAI 2024 · similarity 0.86)
- Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.85)
- Context Steering: Controllable Personalization at Inference Time (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.