Research map / Large language models / Language, safety and interpretability
SummPilot: Bridging Efficiency and Customization for Interactive Summarization System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents SummPilot, a GPT-4o-based interactive summarization system with semantic graphs, entity clustering, and FactScore-style explainable evaluation, validated by small user studies.
Problem
Automatic summarization often fails to meet individual user requirements, and existing interactive summarization systems struggle to represent content relationships across documents and to show how user inputs affect the summary.
Approach
Combines an LLM-driven relation-triple semantic graph with user checkboxes for including or excluding triples, entity coreference clustering, and an evaluation module that scores compression, coverage, and factual consistency via atomic-fact verification and flags unverified facts as possible errors.
Key results
- SUS average score of 86.3 across six participants, classified as excellent.
- Advanced Mode scored 4.3 on USE usefulness and 4.8 on adjusting the summary, versus 2.9 on ease of use for Basic Mode.
- In problem-solving tasks both modes reached 90% correct, but Advanced Mode took longer (5:02 vs 3:33).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Advanced Mode with interactive summarization features instead of Basic Mode | SummPilot · Multi-News problem-solving tasks | correct rate: 90% -> 90% | No clear effect |
| Use Advanced Mode with interactive summarization features instead of Basic Mode | SummPilot · Multi-News problem-solving tasks | time spent: 3:33 -> 5:02 | Hurt |
| Use Advanced Mode rather than Basic Mode | SummPilot · Multi-News comparative user study | understanding the article | Helped |
| Use Advanced Mode rather than Basic Mode | SummPilot · Multi-News comparative user study | ease of use | Hurt |
Surprising findings
- Advanced Mode took longer to complete problem-solving tasks than Basic Mode (5:02 vs 3:33) despite the paper's claim of faster task completion.
- Both modes achieved identical 90% correct rates in problem-solving.
Limitations
- Evaluation relies on small user studies with six participants per study.
- Relies on proprietary GPT-4o, so reproducibility depends on API access.
- Advanced Mode was not faster than Basic Mode in the problem-solving task.
Experimental setup
Datasets: Multi-News
Benchmarks: Multi-News
Metrics: SUS score usefulness questionnaire score USE questionnaire score problem-solving correct rate time spent
Baselines: Basic Mode
Builds on: FactScore SummHelper
Closest papers by meaning
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.88)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.84)
- Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.82)
- Document Summarization with Conformal Importance Guarantees (NeurIPS 2025 · similarity 0.81)
- Automatic Summarization of Long Documents (Student Abstract) (AAAI 2025 · similarity 0.81)
- Tell Me What Is Good about This Property: Leveraging Reviews for Segment-Personalized Image Collection Summarization (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.