Research map / Large language models / Language, safety and interpretability

LIFEBENCH: Evaluating Length Instruction Following in Large Language Models

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LIFEBench, a 10,800-instance bilingual benchmark measuring whether LLMs follow explicit output length instructions (16 to 8192 words) across four task types, and evaluates 26 LLMs.

Problem

LLMs often fail to follow explicit length instructions (e.g., write a 10,000-word novel), producing outputs that are too short, terminate early, or refuse, while existing benchmarks focus on quality rather than length compliance.

Approach

A full-range length-instruction benchmark spanning QA, summarization, reasoning, and creative generation in English and Chinese, scored with two metrics, Length Deviation (direction and magnitude of deviation) and Length Score (robust aggregate), to measure length following across lengths from 16 to 8192 words.

Key results

• o3-mini achieves the highest Length Score of 75.4 while most models score below 60.
• Summarization shows the largest Length Score drop of 18.8; increasing input length from 1k to 8k words reduces score by 13.6.
• Almost all 26 evaluated models fail to reach vendor-claimed maximum output lengths in practice, confirmed with evaluations up to 32K words.

Measured effects reported in the paper

Surprising findings

• Long-context LLMs fail to improve length instruction following despite extended input-output windows.
• Reasoning LLMs outperform specialized long-text generation models in length following.

Limitations

• Models often fall short due to lazy strategies such as falsely claiming inability, which complicates attribution of failures.

Experimental setup

Benchmarks: LIFEBench

Metrics: Length Deviation Length Score

Baselines: o3-mini

Closest papers by meaning

• LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.92)
• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.90)
• The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.90)
• Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs (NeurIPS 2025 · similarity 0.90)
• LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models (ICML 2026 · similarity 0.90)
• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.90)