AI Paper Atlas

Research map / Large language models / Agents, code and math

ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

Jie Ruan, Inderjeet Jayakumar Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Yune-Ting Tiffany Chiang, Lucy R. Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang and 5 more

ICLR 2026 · Large language models › Agents, code and math · approach: Code generation benchmarks

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
evaluate frontier LLMs on end-to-end expert long-form tasks13 LLMs · ExpertLongBenchaverage checklist F1: best model 33.4Hurt
use larger model within the same familyMistral-Large vs Mistral-Nemo · T4 and T10per-task rank: T4: 2nd vs 11th; T10: 6th vs 1stMixed
use larger model within the same familyMistral, Llama, Qwen families · ExpertLongBench averageaverage checklist F1: Mistral-Large 24.0 vs Mistral-Nemo 23.8Improved
use proprietary instead of open-weight modelsClaude-3.7-Sonnet and Claude-3.5-Haiku · ExpertLongBench averageaverage checklist F1: 23.2 and 19.3Hurt
limited 32K context length for Qwen modelsQwen2.5-72B · T10 FinanceESG (64K average input)checklist F1: trails the leading model by 17.5Hurt
higher checklist item coverage in generated textall evaluated models · ExpertLongBenchF1 vs coverageHurt
replace GPT-4o with Qwen2.5-72B as judge in CLEAR (ablation)CLEAR evaluation · all tasksPearson correlation with GPT-4o: 0.88No clear effect
majority-pool three small open-weight judges (ablation)CLEAR judge · all tasksPearson correlation with GPT-4o: 0.65, 0.80, 0.78 -> 0.82Improved
majority pooling instead of mean pooling for judge ensembles (ablation)CLEAR judge · all tasksPearson correlation with GPT-4oImproved
use Qwen2.5-72B as open-weight checklist mapper instead of GPT-4o (ablation)CLEAR mapper · T1, T6, T7, T8mapping F1: 90.1Improved
raise reasoning difficulty and graduate-level knowledgeevaluated LLMs · ExpertLongBench skill decompositionchecklist F1Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy