Research map / Large language models / Language, safety and interpretability
A Semantic QA-Based Approach for Text Summarization Evaluation
Ping Chen, Fei Wu, Tong Wang, Wei Ding
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply topic-word question filtering (ablation) | QA-based evaluation · DUC 2007, question length 20, confidence 0.8 | Pearson correlation: 0.6120 -> 0.7725 | Improved |
| Apply topic-word question filtering (ablation) | QA-based evaluation · DUC 2007, question length 20, confidence 0.9 | Pearson correlation: 0.6379 -> 0.8363 | Improved |
| Apply topic-word question filtering (ablation) | QA-based evaluation · DUC 2007, question length 30, confidence 0.8 | Pearson correlation: 0.6373 -> 0.6310 | Hurt |
| Apply topic-word question filtering (ablation) | QA-based evaluation · DUC 2007, question length 30, confidence 0.9 | Pearson correlation: 0.5092 -> 0.5462 | Improved |
| Increase question length limit from 20 to 30 words (ablation) | QA-based evaluation · DUC 2007, confidence 0.8, no filtering | Pearson correlation: 0.6120 -> 0.6373 | Improved |
| Increase question length limit from 20 to 30 words (ablation) | QA-based evaluation · DUC 2007, confidence 0.9, no filtering | Pearson correlation: 0.6379 -> 0.5092 | Hurt |
| Increase question length limit from 20 to 30 words (ablation) | QA-based evaluation · DUC 2007, confidence 0.8, with filtering | Pearson correlation: 0.7725 -> 0.6310 | Hurt |
| Increase question length limit from 20 to 30 words (ablation) | QA-based evaluation · DUC 2007, confidence 0.9, with filtering | Pearson correlation: 0.8363 -> 0.5462 | Hurt |
| Raise answer confidence threshold from 0.8 to 0.9 (ablation) | QA-based evaluation · DUC 2007, question length 20, no filtering | Pearson correlation: 0.6120 -> 0.6379 | Improved |
| Raise answer confidence threshold from 0.8 to 0.9 (ablation) | QA-based evaluation · DUC 2007, question length 20, with filtering | Pearson correlation: 0.7725 -> 0.8363 | Improved |
| Raise answer confidence threshold from 0.8 to 0.9 (ablation) | QA-based evaluation · DUC 2007, question length 30, no filtering | Pearson correlation: 0.6373 -> 0.5092 | Hurt |
| Raise answer confidence threshold from 0.8 to 0.9 (ablation) | QA-based evaluation · DUC 2007, question length 30, with filtering | Pearson correlation: 0.6310 -> 0.5462 | Hurt |
Closest papers by meaning
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.85)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.84)
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.83)
- Learning from Failure: Improving Meeting Summarization without Good Samples (AAAI 2024 · similarity 0.83)
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.83)
- Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
