Research map / Large language models / Retrieval and knowledge

EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents

AAAI 2025 · Large language models › Retrieval and knowledge · approach: Domain-specific QA datasets · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EventSum, a large-scale Chinese event-centric multi-document summarization dataset built from Baidu Baike, with a manually annotated test set and four NLI-based event recall metrics, and benchmarks 10 long-context LLMs on it.

Problem

Existing multi-document summarization datasets are news-focused, not organized around dynamic events in chronological order, are often automatically built with leakage risk, and are evaluated with metrics (ROUGE, BERTScore) that poorly capture event completeness.

Approach

Event-centric multi-document summarization dataset for Chinese (5,100 events, ~11.4 docs each) with a human-written test set, plus NLI-based recall metrics (Event, Argument, Causal, Temporal Recall) trained on event-extraction and automatically built data to score coverage of structured event elements.

Key results

• Best open-source model glm-4-9b-chat-1M reaches ROUGE-1 49.3 and BERTScore 77.0; best commercial GPT-4o ROUGE-1 47.5.
• Claude-3-Opus achieves the best event-centric recalls on most metrics (Argument Recall 50.3, Causal Recall 67.3, Temporal Recall 56.8), while Event Recall is lowest across models (best 27.4 by Qwen2-7b-Instruct).
• NLI-based metric predictions agree with human judgment with consistency of 90.9% to 97.0% across the four recall metrics.

Measured effects reported in the paper

Surprising findings

• Open-source models outperform commercial models on common metrics (ROUGE, BERTScore), while commercial models like Claude-3-Opus lead on the designed event-centric recall metrics.
• Few-shot demonstrations generally slightly decreased performance across most metrics.

Limitations

• Training data for summaries is automatically constructed from Baidu Baike and retrieved news, with auto-data quality pass rate of 81%.
• Only the test set is manually annotated; temporal relations are annotated by an LLM (glm-4-9b).
• Only Chinese; the paper notes plans to extend to English and longer time span events.

Experimental setup

Datasets: EventSum (Baidu Baike-derived) CMNEE MAVEN-ERE (translated to Chinese) Bing News Search retrieved articles

Benchmarks: EventSum

Metrics: ROUGE-1 ROUGE-2 ROUGE-L BERTScore Event Recall Argument Recall Causal Recall Temporal Recall

Baselines: Baichuan2-13b-chat Llama3-chinese-8B-Instruct Yi-1.5-9b-chat-16k Qwen2-7b-Instruct glm-4-9b-chat InternLM2.5-7B-Chat-1M glm-4-9b-chat-1M MoonShot Claude-3-Opus GPT-4o

Builds on: Multi-News MAVEN-ERE CMNEE ROUGE BERTScore

Closest papers by meaning

• Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities (AAAI 2024 · similarity 0.84)
• Is a Large Language Model a Good Annotator for Event Extraction? (AAAI 2024 · similarity 0.84)
• Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance (AAAI 2025 · similarity 0.83)
• MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval (CVPR 2025 · similarity 0.83)
• Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.82)
• AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval (ICLR 2024 · similarity 0.82)