Research map / Large language models / Language, safety and interpretability
Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills structured rationales (essential aspects, associated sentences, triple entity relations) extracted by Llama-3-70B into a BART-large student via multi-view hierarchical alignment and weak-gated fusion for abstractive summarization.
Problem
Small language models for abstractive summarization lack the structured reasoning of large LLMs, and effectively distilling structured rationales from LLMs into SLMs while keeping deployment cost low is challenging.
Approach
Uses an LLM to dig three-view structured rationales (EA, AS, TER), selects high-quality rationales via ROUGE-based multi-step sub-summary evaluation, and fuses hierarchically aligned rationale features into BART decoder cross-attention through learnable weak-gated units instead of fixed ReLU gates.
Key results
- LSR-MWF (439M) reaches 48.54 R-1 / 23.91 R-2 on CNNDM, outperforming BART-large by 11.2% (per paper's stated relative gain).
- On XSum, LSR-MWF achieves 47.47 R-1 / 24.32 R-2 / 39.28 R-L, a 5.8% gain over BART-large.
- Removing TER causes the largest ablation drop on CNNDM (R-1 48.54 to 46.37).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add essential aspects (EA) (ablation) | LSR-MWF · CNN/DailyMail | ROUGE-1: 48.08 -> 48.54 | Helped |
| Add associated sentences (AS) (ablation) | LSR-MWF · CNN/DailyMail | ROUGE-1: 47.55 -> 48.54 | Helped |
| Add triple entity relations (TER) (ablation) | LSR-MWF · CNN/DailyMail | ROUGE-1: 46.37 -> 48.54 | Helped |
| Add AS and TER (ablation) | LSR-MWF · CNN/DailyMail | ROUGE-1: 44.58 -> 48.54 | Helped |
| Add EA and TER (ablation) | LSR-MWF · CNN/DailyMail | ROUGE-1: 45.22 -> 48.54 | Helped |
| Add EA and AS (ablation) | LSR-MWF · CNN/DailyMail | ROUGE-1: 44.76 -> 48.54 | Helped |
| Add essential aspects (EA) (ablation) | LSR-MWF · XSum | ROUGE-1: 47.34 -> 47.47 | Helped |
| Add associated sentences (AS) (ablation) | LSR-MWF · XSum | ROUGE-1: 46.93 -> 47.47 | Helped |
| Add triple entity relations (TER) (ablation) | LSR-MWF · XSum | ROUGE-1: 45.22 -> 47.47 | Helped |
| Add AS and TER (ablation) | LSR-MWF · XSum | ROUGE-1: 45.09 -> 47.47 | Helped |
| Add EA and TER (ablation) | LSR-MWF · XSum | ROUGE-1: 44.73 -> 47.47 | Helped |
| Add EA and AS (ablation) | LSR-MWF · XSum | ROUGE-1: 46.69 -> 47.47 | Helped |
Surprising findings
- Relying on TER alone performs worse than AS alone on CNNDM, but better than AS alone on XSum, attributed to XSum's higher abstraction level.
Limitations
- Relies on a 70B teacher LLM for rationale generation, so rationale extraction cost is high.
- Gains reported against a BART-large baseline with relative percentage improvements rather than absolute differences; limited to two news datasets.
Experimental setup
Datasets: CNN/DailyMail XSum
Benchmarks: CNN/DailyMail XSum
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BERTScore
Baselines: BERTSumAbs T5 BART PEGASUS GSum BigBird SimCLS SeqCo GLM BRIO GPT-3.5 TriSum
Builds on: BART TriSum GSum chain-of-thought
Closest papers by meaning
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.88)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.87)
- Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.86)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.86)
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.86)
- Learning from Failure: Improving Meeting Summarization without Good Samples (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.