Research map / Large language models / Agents, code and math
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
Jie Ruan, Inderjeet Jayakumar Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Yune-Ting Tiffany Chiang, Lucy R. Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate frontier LLMs on end-to-end expert long-form tasks | 13 LLMs · ExpertLongBench | average checklist F1: best model 33.4 | Hurt |
| use larger model within the same family | Mistral-Large vs Mistral-Nemo · T4 and T10 | per-task rank: T4: 2nd vs 11th; T10: 6th vs 1st | Mixed |
| use larger model within the same family | Mistral, Llama, Qwen families · ExpertLongBench average | average checklist F1: Mistral-Large 24.0 vs Mistral-Nemo 23.8 | Improved |
| use proprietary instead of open-weight models | Claude-3.7-Sonnet and Claude-3.5-Haiku · ExpertLongBench average | average checklist F1: 23.2 and 19.3 | Hurt |
| limited 32K context length for Qwen models | Qwen2.5-72B · T10 FinanceESG (64K average input) | checklist F1: trails the leading model by 17.5 | Hurt |
| higher checklist item coverage in generated text | all evaluated models · ExpertLongBench | F1 vs coverage | Hurt |
| replace GPT-4o with Qwen2.5-72B as judge in CLEAR (ablation) | CLEAR evaluation · all tasks | Pearson correlation with GPT-4o: 0.88 | No clear effect |
| majority-pool three small open-weight judges (ablation) | CLEAR judge · all tasks | Pearson correlation with GPT-4o: 0.65, 0.80, 0.78 -> 0.82 | Improved |
| majority pooling instead of mean pooling for judge ensembles (ablation) | CLEAR judge · all tasks | Pearson correlation with GPT-4o | Improved |
| use Qwen2.5-72B as open-weight checklist mapper instead of GPT-4o (ablation) | CLEAR mapper · T1, T6, T7, T8 | mapping F1: 90.1 | Improved |
| raise reasoning difficulty and graduate-level knowledge | evaluated LLMs · ExpertLongBench skill decomposition | checklist F1 | Hurt |
Closest papers by meaning
- InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models (NeurIPS 2024 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.89)
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly (ICLR 2025 · similarity 0.88)
- TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning (AAAI 2025 · similarity 0.88)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
