Research map / Large language models / Agents, code and math
WritingBench: A Comprehensive Benchmark for Generative Writing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WritingBench, a 1,000-query, 6-domain/100-subdomain generative writing benchmark, plus a query-dependent LLM-as-judge framework with a fine-tuned 7B critic and data curation that lets 7-8B models match or beat larger models.
Problem
Existing writing benchmarks are narrow in domain and task formulation and use static, generic evaluation criteria that fail to capture diverse, requirement-rich real-world writing.
Approach
Generate five instance-specific evaluation criteria per query with an LLM (Claude-3.7), score responses with a fine-tuned Qwen-2.5-7B critic model on those criteria, and use the same scores to filter SFT data so that small models gain writing ability.
Key results
- The fine-tuned critic model reaches 84% human alignment, versus 67% (Claude-3.7 with static global criteria) and 58% (static domain-specific criteria).
- Qwen-2.5-7B-filtered scores 7.44 on WritingBench, above Qwen-2.5-72B-Instruct (6.40) and GPT-4o (6.81).
- Llama-3.1-8B-filtered scores 7.39 on WritingBench, above Llama-3.3-70B-Instruct (5.05), with Claude-3.7-thinking top at 7.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic query-dependent criteria instead of static global criteria | Claude judge · WritingBench human consistency | human consistency: 67% -> 87% | Helped |
| Use dynamic query-dependent criteria instead of static domain-specific criteria | Claude judge · WritingBench human consistency | human consistency: 58% -> 87% | Helped |
| Use dynamic query-dependent criteria instead of static global criteria | GPT-4o judge · WritingBench human consistency | human consistency: 69% -> 79% | Helped |
| Use dynamic query-dependent criteria instead of static domain-specific criteria | GPT-4o judge · WritingBench human consistency | human consistency: 40% -> 79% | Helped |
| Use the critic model for dynamic query-dependent scoring | WritingBench evaluation · WritingBench human consistency | human consistency: 67% -> 84% | Helped |
| Fine-tune on filtered 12K data instead of the full 24K dataset (ablation) | Qwen-2.5-7B-Instruct · WritingBench | score: 7.36 -> 7.44 | Helped |
| Fine-tune on filtered 12K data instead of the full 24K dataset (ablation) | Qwen-2.5-7B-Instruct · LongBench-Write | score: 4.69 -> 4.70 | Helped |
| Fine-tune on filtered 12K data instead of the full 24K dataset (ablation) | Llama-3.1-8B-Instruct · WritingBench | score: 7.34 -> 7.39 | Helped |
| Fine-tune on filtered 12K data instead of the full 24K dataset (ablation) | Llama-3.1-8B-Instruct · LongBench-Write | score: 4.65 -> 4.65 | No clear effect |
| Add CoT-formatted instructions during fine-tuning (ablation) | Qwen-2.5-32B-Instruct · WritingBench-D4 | score: 7.54 -> 7.58 | Helped |
| Add CoT-formatted instructions during fine-tuning (ablation) | Qwen-2.5-32B-Instruct · EQBench | score: 79.43 -> 82.48 | Helped |
Surprising findings
- Filtering SFT data with the framework yields only marginal gains over unfiltered data (e.g., Qwen-2.5-7B 7.44 filtered vs 7.36 all), with the stated gap mainly in how the 12K subset compares.
- Most models cap output near 3,000 tokens, and small models show repetitive outputs in long generations.
Limitations
- Criteria are generated by Claude-3.7 and the critic is trained on Claude-3.7 scores, so the judge inherits that model's preferences.
- Length control and long-output generation remain weak for most models.
- Human alignment study uses 300 pairwise judgments with five annotators, a limited sample.
Experimental setup
Datasets: WritingBench LongBench-Write EQ-Bench 155K critic scoring instances 24K generated writing queries (12K filtered)
Benchmarks: WritingBench LongBench-Write EQ-Bench
Metrics: rubric score (1-10) human alignment agreement % Kappa
Baselines: GPT-4o o1-Preview Claude-3.7 Claude-3.7-thinking Gemini-1.5-Pro Qwen-Max Deepseek-R1 Deepseek-V3 Mistral-Large-Instruct Qwen-2.5-72B-Instruct Llama-3.3-70B-Instruct Suri
Builds on: Arena-Hard-Auto LongBench-Write EQ-Bench
Closest papers by meaning
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.91)
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.88)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.88)
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.87)
- LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Controlled Text Generation via Language Model Arithmetic (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.