Research map / Large language models / Language, safety and interpretability

BooookScore: A systematic exploration of book-length summarization in the era of LLMs

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies coherence of LLM book-length summarizers (hierarchical merging vs. incremental updating) on 100 newly published books, using 1193 human span annotations and an automatic GPT-4-based metric, BooookScore.

Problem

Book-length summarization (>100K tokens) with LLMs lacks principled evaluation: existing datasets like BookSum are likely in pretraining data, and existing metrics miss coherence errors made by modern LLM summarizers.

Approach

Build a contamination-avoiding corpus of newly published books, derive an eight-type coherence error taxonomy from human point-of-confusion annotations, and automate it with a sentence-by-sentence few-shot GPT-4 detector whose score is the fraction of error-free sentences.

Key results

• Claude 2 (88K chunk, hierarchical) achieves the highest BooookScore at 91.1 with 2048 chunks; GPT-4 hierarchical reaches 89.1 and GPT-4 hierarchical with 4096 chunks yields 90.8 with LLM annotations.
• BooookScore annotation precision is 78.2% versus 79.7% for human annotators, and system-level scores from GPT-4 and human annotations are very similar.
• Incremental updating yields lower coherence than hierarchical merging for most models (e.g., GPT-4 82.5 vs 89.1), but annotators preferred incremental summaries for level of detail (83% vs 11%).

Measured effects reported in the paper

Surprising findings

• Human preference judgments on overall quality do not correlate with BooookScore; annotators sometimes preferred incremental summaries for detail despite lower coherence.
• LLaMA2-7B-Instruct could not perform incremental updating, copying chunk text until the length limit and failing the compression instruction.

Limitations

• Error taxonomy derived only from GPT-4 summaries and the metric uses GPT-4 as annotator, which may introduce systematic bias.
• BooookScore is expensive to run because it prompts GPT-4 separately for each sentence and does not weight error severity.
• No validation of annotation recall; only precision was measured.

Experimental setup

Datasets: BookSum SQuALITY

Benchmarks: 100 newly-published books (custom) SQuALITY

Metrics: BooookScore (proportion of error-free sentences) precision of annotations average summary length % novel 3-grams % repeated 3-grams ROUGE-L human preference

Baselines: GPT-3.5-Turbo Claude 2 Mixtral-8x7B LLaMA2-7B-Instruct BLANC SUPERT

Builds on: chain-of-density prompting BookSum SNaC FActScore AlpacaEval

Closest papers by meaning

• PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.87)
• Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.87)
• Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.87)
• Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation (NeurIPS 2025 · similarity 0.86)
• LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.86)
• Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.86)