Research map / Large language models / Language, safety and interpretability
Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces constrained timeline summarization (CTLS), a CREST benchmark of 47 entities with 5 constraints each, and REACTS, a training-free LLM pipeline with per-article constrained summarization, LLM self-reflection filtering, embedding-based event clustering, and cluster selection.
Problem
Standard timeline summarization produces one generic timeline, but readers want timelines restricted to events meeting a user-specified constraint; prior LLM context-based approaches also cannot scale to large news streams.
Approach
Summarize each article under a constraint, use the same LLM to self-reflect and discard summaries that violate the constraint, then cluster surviving event summaries by embedding similarity plus LLM and date checks to build the timeline, instead of concatenating articles into a prompt.
Key results
- REACTS with Llama-3.1 70B reaches AR-1 F1 0.1316 and Date F1 0.2925 on filtered events, versus 0.0793 and 0.1751 for the baseline.
- Self-reflection improves Llama-3.1 70B date F1 by 6.38% on all events and AR-1 F1 by 2.82%.
- With an oracle retriever, REACTS with Llama-3.1 70B reaches Date F1 0.4622 versus 0.3324 for the baseline on filtered events.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 8B · CREST all events | AR-1 F1: 0.0561 -> 0.0768 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 8B · CREST all events | AR-2 F1: 0.0263 -> 0.0351 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 8B · CREST all events | Date F1: 0.1395 -> 0.1758 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 70B · CREST all events | AR-1 F1: 0.0809 -> 0.1091 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 70B · CREST all events | AR-2 F1: 0.0393 -> 0.0501 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 70B · CREST all events | Date F1: 0.1773 -> 0.2411 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 8B · CREST filtered events | AR-1 F1: 0.0729 -> 0.0977 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 8B · CREST filtered events | AR-2 F1: 0.0338 -> 0.0459 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 8B · CREST filtered events | Date F1: 0.1746 -> 0.2238 | Helped |
| Add self-reflection to REACTS (ablation) | REACTS with Llama 3.1 70B · CREST filtered events | AR-1 F1: 0.1040 -> 0.1316 | Helped |
| Use REACTS instead of direct-prompt Baseline | Llama 3.1 8B · CREST filtered events | AR-1 F1: 0.0246 -> 0.0977 | Helped |
| Use REACTS instead of direct-prompt Baseline | Llama 3.1 70B · CREST filtered events | Date F1: 0.1751 -> 0.2925 | Helped |
Surprising findings
- LLMs often produce an event summary even when it violates the specified constraint, which self-reflection is designed to catch.
- The baseline with Llama-3.1 8B often fails to stop generation and sometimes produces nonsensical outputs.
Limitations
- Dataset limited to British and American news sources from The Guardian.
- Constraints were generated by GPT-4o and then curated by humans, and GPT-4o was used in dataset construction, which may give it an advantage.
- Evaluation of only 10% of the test set with GPT-4o due to cost.
Experimental setup
Datasets: ENTITIES CREST
Benchmarks: CREST
Metrics: AR-1 F1 AR-2 F1 Date F1 precision recall
Baselines: Baseline (concatenate articles into prompt) REACTS w/o SR
Builds on: LLM-TLS TextRank GTE
Closest papers by meaning
- EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents (AAAI 2025 · similarity 0.83)
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.82)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.81)
- Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.81)
- Logic Induced High-Order Reasoning Network for Event-Event Relation Extraction (AAAI 2025 · similarity 0.81)
- Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.