Research map / Large language models / Language, safety and interpretability

LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LongGenBench, a synthetic benchmark testing whether long-context LLMs can generate 16K/32K-token outputs that follow single, range, and periodic instructions across diary, menu, skyscraper, and urban planning scenarios.

Problem

Existing long-context benchmarks (NIAH, RULER, NeedleBench) test long-input comprehension, not the generation of coherent, instruction-compliant long-form text, which is needed for applications like design proposals and documentation.

Approach

Reverse-NIAH style benchmark: prompts embed single, range, and periodic instructions that must be placed at specified positions in a very long generated output, scored by completion rate and instruction-completion metrics (STIC-1/STIC-2) using a binary LLM judge on extracted segments.

Key results

• Llama3.1-8B achieves the highest 32K completion rate (77.6%) and wAvg of 14.6%, outperforming larger models at that length.
• GPT-4o-mini reaches 97.0% CR and 26.9% wAvg at 16K, while GPT-4o scores 12.5% wAvg at 16K.
• Pearson correlation between RULER input scores and LongGenBench output scores is 0.51 at 16K and 0.66 at 32K, indicating only partial overlap of skills.

Measured effects reported in the paper

Surprising findings

• Models with strong RULER results still struggle heavily on long-form generation; several models stop early or answer only some subtasks.
• Approximately 45% of long outputs showed significant repetition, and repetition_penalty adjustment gave only limited mitigation.

Limitations

• Evaluation focuses on explicit instruction adherence in four synthetic scenarios and does not assess creativity, rationality, or coherence in depth.
• Outputs are judged by a binary Llama3.1-8B judge validated on only 300 manually labeled points.
• Reported results contain internal inconsistencies (e.g., duplicated FILM-7B rows, model count of nine vs ten, Mixtral listed as 8x22B in table) and the metric definitions are ambiguous.

Experimental setup

Datasets: LongGenBench

Benchmarks: LongGenBench RULER

Metrics: Completion Rate STIC-1 STIC-2 weighted average (wAvg) output length

Baselines: GPT-4o GPT-4o-mini Llama3.1-8B-Instruct Llama3.1-70B-Instruct Qwen2-7B-Instruct Qwen2-72B-Instruct Mistral-7B-Instruct-v0.2 Mixtral-8x7B FILM-7B LongWriter-llama3.1-8B

Builds on: Needle-in-a-Haystack RULER vLLM

Closest papers by meaning

• LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.92)
• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions (NeurIPS 2025 · similarity 0.92)
• LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.92)
• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.92)
• WritingBench: A Comprehensive Benchmark for Generative Writing (NeurIPS 2025 · similarity 0.91)
• Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.89)