Research map / Large language models / Language, safety and interpretability

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies SFT output-length caps as the cause of ~2k-word generation limits, proposes AgentWrite (plan-then-write agent pipeline) to build the LongWriter-6k dataset, and trains LongWriter models (9B/8B, plus DPO) that generate 10k+ word outputs.

Problem

Long-context LLMs can read 100k+ tokens but cannot generate outputs beyond roughly 2,000 words; the paper investigates why and how to extend output length.

Approach

The effective generation length is bounded by the maximum output length in SFT data. AgentWrite uses off-the-shelf LLMs to plan and then serially write paragraphs, producing long outputs that are used as SFT data (LongWriter-6k) to unlock 10k+ word generation in existing models.

Key results

• LongWriter-9B-DPO achieves overall score 84.0 on LongBench-Write, the highest among evaluated models, above GPT-4o at 78.6.
• AgentWrite extends GPT-4o's effective output from about 2k words to about 20k words on LongWrite-Ruler, raising its [4k,20k) length score S_l from 5.6 to 86.8.
• LongWriter models generate outputs reaching 10k-20k words; humans preferred LongWriter-9B-DPO over LongWriter-9B in 58% of cases.

Measured effects reported in the paper

Surprising findings

• Plan-augmented SFT data (writing plan before the content) did not significantly improve performance and slightly reduced quality.
• Instruction-backtranslated long texts from pretraining data hurt output length and quality rather than helping.

Limitations

• Maximum generation length remains limited to 10k-20k words due to few SFT examples above 20k words.
• AgentWrite serial generation is slower and parallel variant reduces coherence.
• GPT-4o is both a data generator and a quality judge, which may bias self-evaluation.

Experimental setup

Datasets: LongWriter-6k GLM-4 chat SFT data WildChat-1M GLM-4 chat DPO data LongBench-Write LongWrite-Ruler

Benchmarks: LongBench-Write LongWrite-Ruler

Metrics: length score S_l quality score S_q overall score win rate (human pairwise) generation length

Baselines: Claude 3.5 Sonnet GPT-4 Turbo GPT-4o mini GPT-4o GLM-4-9B-chat Llama-3.1-8B-Instruct Llama-3.1-70B-Instruct Mistral-Large-Instruct Suri-I-ORPO

Builds on: AgentWrite-style plan-then-write DPO LLM-as-a-judge instruction backtranslation

Closest papers by meaning

• LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.92)
• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.91)
• Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.90)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)
• LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.89)
• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions (NeurIPS 2025 · similarity 0.89)