Research map / Large language models / Language, safety and interpretability
Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MiDAS-PRo, which uses code-LLM few-shot prompting to plan a hierarchical citation layout and generate sentence-wise entities, selects in-context examples via a GAT, and then generates attribution-inclusive multi-document summaries.
Problem
Multi-document summarization of scientific documents with citation attribution, where naively concatenated sources yield poorly structured, redundant summaries with unreliable source attribution.
Approach
Treat planning (hierarchical reference layout tree) and reasoning (sentence-wise topic entities) as code completion tasks for a code LLM under in-context learning, and choose in-context examples with a graph attention network trained to mimic tree-edit similarity of HRLTs.
Key results
- On Multi-XScience 1-shot with Llama-3-8B-Instruct, MiDAS-PRo raises average AMI from 0.22 (BM25) to 0.33 and ROUGE-L from 0.1512 to 0.1542.
- With GPT-4o 1-shot on Multi-XScience, MiDAS-PRo achieves BLEU 0.0993 vs 0.0839 for best baseline (BM25) and average AMI 0.254 vs 0.09.
- Human evaluation on 12 Multi-XScience samples gives MiDAS-PRo the highest average score (4.68 vs 4.63 for BM25).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MiDAS-PRo instead of BM25 in-context example selection | GPT-4o summarization · Multi-XScience, 1-shot | BLEU: 0.0839 -> 0.0993 (+18.4%) | Helped |
| Use MiDAS-PRo instead of BM25 in-context example selection | GPT-4o summarization · Multi-XScience, 5-shot | BLEU: 0.0882 -> 0.1105 (+25.3%) | Helped |
| Use MiDAS-PRo instead of BM25 in-context example selection | GPT-4o summarization · Multi-XScience, 10-shot | BLEU: 0.089 -> 0.1113 (+25.1%) | Helped |
| Use MiDAS-PRo instead of SciBERT example selection | GPT-4o summarization · MiDAS, 1-shot | BLEU: 0.0571 -> 0.0692 | Helped |
| Add sentence-level planning and reasoning beyond citation-bracket-only planning (ablation) | MiDAS-PRo · Multi-XScience, 1-shot | BLEU: 0.0894 -> 0.0917 | Helped |
| Add reasoning to sentence-level planning (ablation) | MiDAS-PRo · Multi-XScience, 1-shot | AVG NMI: 0.388 -> 0.421 | Helped |
| Add reasoning to planning (ablation) | MiDAS-PRo · Multi-XScience, 1-shot | BLEU: 0.0913 -> 0.0917 | Mixed |
| Add hierarchical planning beyond sentence-level planning with reasoning (ablation) | MiDAS-PRo · Multi-XScience, 1-shot | AVG NMI: 0.39 -> 0.421 | Helped |
Surprising findings
- Larger models (Llama-3-70B, GPT-4o) do not necessarily improve multi-document summarization over smaller ones (Mistral-7B, Llama-3-8B) when using MiDAS-PRo.
- BM25 performs best among the baseline ICE selection methods for most metrics.
Limitations
- Human evaluation covers only 12 samples and one scientific expert.
- The MiDAS subset is small (96 examples) and annotated by the authors.
- Relies on GPT-4o for generating planning code, which is a closed API.
Experimental setup
Datasets: Multi-XScience BigSurvey-MDS
Benchmarks: Multi-XScience MiDAS
Metrics: ROUGE-L BLEU BERTScore NMI AMI human rating of relevance, fluency, informativeness, attribu
Baselines: Random ICE selection BM25 ICE selection SciBERT ICE selection PEGASUS BART PRIMERA
Builds on: Madaan et al. 2022 code LLMs as commonsense reasoners Graph Attention Networks Synchromesh BM25
Closest papers by meaning
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.87)
- Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.86)
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.85)
- Document Summarization with Conformal Importance Guarantees (NeurIPS 2025 · similarity 0.85)
- Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization (ICLR 2026 · similarity 0.84)
- Learning from Failure: Improving Meeting Summarization without Good Samples (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.