Research map / Large language models / Language, safety and interpretability
Learning from Failure: Improving Meeting Summarization without Good Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Score Tuning, a cold-start method that fine-tunes an LLM on (transcript, summary, score) triplets built from asynchronous numerical human ratings of low-quality summaries, to generate high-scoring meeting summaries without any good samples.
Problem
Aligning language models for meeting summarization when no positively-rated (good) samples are available and public or private data are scarce, while SFT and RLHF/CoH require pairwise or gold-standard data.
Approach
Treat all generated summaries, regardless of quality, as training data by conditioning on their asynchronous numerical human scores. The model is trained with a confidence-aware score-prediction loss and a hindsight-score summary-generation loss, then at inference is fixed to the maximum score (100) to produce better summaries.
Key results
- On VCSum, Score Tuning reaches 42.74 average ROUGE (R-Avg.), +5.30 over the pre-trained model and above RLHF (+2.87) and CoH (+3.27).
- On the in-house dataset, human evaluation win rates show Score Tuning significantly preferred over all other methods.
- On CWMT 2018 machine translation, Score Tuning averages 29.35 BLEU, +2.24 over pre-trained, versus +1.09 for RLHF and +1.20 for CoH.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Score Tuning instead of the pre-trained model | ChatGLM-6B · VCSum | R-Avg: 37.44 -> 42.74 | Helped |
| Apply Score Tuning instead of CoH | ChatGLM-6B · VCSum | R-Avg: 40.71 -> 42.74 | Helped |
| Add confidence-aware loss (LCA) (ablation) | Score Tuning · VCSum | R-Avg: 41.85 -> 42.74 | Helped |
| Add hindsight-score loss (LHS) (ablation) | Score Tuning · VCSum | R-Avg: 42.12 -> 42.74 | Helped |
| Apply Score Tuning instead of the pre-trained model | ChatGLM-6B · TL;DR | R-L: 22.54 -> 31.23 | Helped |
| Apply Score Tuning instead of CoH | ChatGLM-6B · TL;DR | R-L: 28.79 -> 31.23 | Helped |
| Apply Score Tuning instead of the pre-trained model | ChatGLM-6B · CWMT 2018 | BLEU: 27.11 -> 29.35 | Helped |
| Apply Score Tuning instead of CoH | ChatGLM-6B · CWMT 2018 | BLEU: 28.31 -> 29.35 | Helped |
| Add confidence-aware loss (LCA) (ablation) | Score Tuning · CWMT 2018 | BLEU: 28.36 -> 29.35 | Helped |
| Add hindsight-score loss (LHS) (ablation) | Score Tuning · CWMT 2018 | BLEU: 29.12 -> 29.35 | Helped |
Surprising findings
- Inter-rater agreement on the numerical scores was only moderate (ICC 0.57 in-house, 0.61 VCSum), yet the method still proved effective.
- RLHF performance dropped in later iterations of the iterative rounds, while Score Tuning kept improving with diminishing gains.
Limitations
- Inter-rater agreement for numerical feedback is only moderate, which the authors acknowledge as a challenge for comparability of scores.
- The in-house dataset is private and not released, and the reward model used in RLHF had to be an off-the-shelf model because of limited data.
- Human evaluation uses only 100 sampled examples with two evaluators.
Experimental setup
Datasets: in-house Chinese meeting dataset VCSum TL;DR CWMT 2018
Benchmarks: VCSum TL;DR CWMT 2018
Metrics: ROUGE-1 ROUGE-2 ROUGE-L human evaluation (accuracy, coherence, coverage, overall on win rate BLEU
Baselines: Pre-trained ChatGLM-6B SFT RLHF Chain of Hindsight
Builds on: Chain of Hindsight RLHF Learning to Summarize from Human Feedback
Closest papers by meaning
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.86)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.85)
- QuRating: Selecting High-Quality Data for Training Language Models (ICML 2024 · similarity 0.85)
- Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.85)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.85)
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.