Research map / Large language models / Language, safety and interpretability
Document Summarization with Conformal Importance Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies conformal prediction (calibrated thresholds on sentence-level importance scores) to extractive document summarization to guarantee, with user-set error and recall levels, that summaries retain a target fraction of important sentences, and adds a hybrid extractive-then-abstractive pipeline.
Problem
LLM-based summarizers offer no reliable guarantee that critical content is retained, which matters in high-stakes domains such as healthcare, law, and finance.
Approach
Adapts conformal factuality to extractive summarization: a per-document conformal score is the largest importance threshold that still retains a fraction beta of important sentences, and a calibrated quantile over calibration documents sets a filter threshold giving distribution-free probability at least 1-alpha that the summary achieves recall at least beta.
Key results
- Empirical coverage over 400 random splits lies within the theoretical bounds [1-alpha, 1-alpha+1/(n+1)] across datasets and scoring functions (e.g., ECTSum with Gemini 2.5 Flash).
- With alpha=0.2 and beta=0.8, Gemini 2.5 Flash removes 37% of ECTSum sentences and 49% of CSDS sentences, while Gemini 2.0 Flash-Lite achieves the highest AUPRC on CNN/DM, TLDR-AIC and TLDR-Full.
- Direct GPT-4o mini abstractive prompting achieved coverage on only 55-70% of ECTSum summaries, while the hybrid extractive-then-abstractive pipeline greatly improved recall and coverage at similar conciseness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini 2.5 Flash instead of other tested importance scorers | Conformal Importance Summarization · ECTSum, CSDS | AUPRC: 0.43 (ECTSum); 0.69 (CSDS) | Helped |
| Use Gemini 2.0 Flash-Lite instead of other tested importance scorers | Conformal Importance Summarization · CNN/DM, TLDR-AIC, TLDR-Full, MTS-Dialog | AUPRC: 0.42; 0.39; 0.25; 0.95 | Helped |
| Use Gemini 2.5 Flash scoring | Conformal Importance Summarization · ECTSum, CSDS, TLDR-AIC, TLDR-Full | proportion of sentences removed: 0.37; 0.49; 0.41; 0.47 | Helped |
| Raise the target error rate α | Conformal Importance Summarization · CSDS | empirical recall | Hurt |
| Lower the target recall β | Conformal Importance Summarization · CSDS | empirical recall | Hurt |
| Raise the target recall β | Conformal Importance Summarization · CSDS and other evaluated datasets | proportion of sentences removed | Hurt |
| Constrain GPT-4o mini importance scores to binary values rather than floats (ablation) | Sentence-importance scoring · All evaluated datasets | AUPRC | Hurt |
| Use ROUGE-1, ROUGE-2, or ROUGE-L instead of SBERT cosine similarity to generate labels (ablation) | Conformal Importance Summarization on CNN/DM and TLDR-AIC · CNN/DM, TLDR-AIC | AUPRC and proportion of sentences removed | No clear effect |
| Reduce calibration set size from n=100 to n=50 (ablation) | Conformal Importance Summarization on ECTSum · ECTSum | proportion of sentences removed: 0.33 -> 0.31 | No clear effect |
| Reduce calibration set size from n=100 to n=25 (ablation) | Conformal Importance Summarization on ECTSum · ECTSum | proportion of sentences removed: 0.33 -> 0.28 | Hurt |
| Prompt an LLM to directly summarize while retaining a target fraction of important information, rather than use conformal extraction | Direct abstractive summarization · ECTSum | coverage: 55-70% | Hurt |
| Add abstractive synthesis after conformal extractive selection | Hybrid extractive-abstractive summarization · ECTSum | recall and coverage | Helped |
Surprising findings
- Constraining GPT-4o mini to binary importance scores substantially degraded AUPRC relative to float scores.
- Direct abstractive prompting for a target recall fails to reach the target on many summaries, with coverage of only 55-70% on ECTSum.
Limitations
- Only two datasets have explicit sentence-level extractive labels; the other three required label refinement via SBERT or GPT-4o mini, which may not match domain expert preferences.
- Experiments are limited in document length (SciTLDR-Full averages 216 sentences), far below EHR-scale documents.
- The hybrid abstractive step carries no coverage guarantee and was not extensively prompt-tuned.
Experimental setup
Datasets: ECTSum CSDS CNN/DM SciTLDR (TLDR-AIC, TLDR-Full) MTS-Dialog
Benchmarks: ECTSum CSDS CNN/DM TLDR-AIC TLDR-Full MTS-Dialog
Metrics: empirical coverage empirical recall proportion of sentences removed AUPRC
Baselines: GPT-4o mini direct labeling Cosine Similarity Centrality Sentence Centrality GUSUM LexRank Llama3-8B Qwen3-8B Gemini 2.0 Flash-Lite
Builds on: conformal factuality conformal prediction TextRank BERTSum
Closest papers by meaning
- Preserve Context Information for Extract-Generate Long-Input Summarization Framework (AAAI 2023 · similarity 0.85)
- Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.85)
- Text summarization via global structure awareness (ICLR 2026 · similarity 0.85)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.84)
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.84)
- Unsupervised Extractive Summarization with Learnable Length Control Strategies (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.