Research map / Large language models / Agents, code and math
Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ComplexBench, a benchmark of 1,150 manually constructed instructions with multiple constraint compositions (And, Chain, Selection, nested), scored by a rule-augmented LLM evaluator with dependency-based aggregation.
Problem
Existing complex instruction-following benchmarks model individual constraints but neglect how constraints are structurally composed (e.g., sequential chains, conditional selection), leading to insufficient evaluation of LLMs.
Approach
A hierarchical taxonomy (4 constraint types, 19 dimensions, 4 composition types) with yes/no scoring questions verified by a rule-augmented LLM evaluator (LLM extraction plus rules), and final scores aggregated over the dependency structure induced by compositions.
Key results
- GPT-4-1106 achieves the highest overall DRFR of 0.800 on ComplexBench, still failing about 20% of complex instructions.
- Performance drops with composition complexity; Chain and Selection are hardest, e.g., GPT-4 Selection & Chain (>=3 depth) avg 0.675.
- The RAL evaluator reaches 87.82% question-level agreement with humans versus 75.18% for direct scoring; pairwise agreement 0.614 vs 0.512.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Dependency Aggregation rather than omitting it (ablation) | RAL evaluation · Overall Pairwise Agreement with human | Pairwise agreement: 0.574 -> 0.614 | Helped |
| Add rule arguments to the LLM evaluator (ablation) | RAL evaluation · Rule-defined questions | Question-level agreement: 82.02% -> 95.36% | Helped |
| Add rule arguments to the LLM evaluator (ablation) | RAL evaluation · Open-ended questions | Question-level agreement: 86.28% -> 86.28% | No clear effect |
| Add rule arguments to the LLM evaluator (ablation) | RAL evaluation · Overall question-level agreement | Question-level agreement: 85.56% -> 87.82% | Helped |
| Use RAL rather than Direct Scoring | ComplexBench human agreement evaluation · Overall Pairwise Agreement with human | Pairwise agreement: 0.512 -> 0.614 | Helped |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · And, Chain, and Selection instructions | DRFR: Overall: 0.682 -> 0.652 | Hurt |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · And (depth 1) | DRFR: 0.845 -> 0.845 | No clear effect |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · Chain (depth 1) | DRFR: 0.686 -> 0.655 | Hurt |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · Chain (depth 2) | DRFR: 0.630 -> 0.583 | Hurt |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · Selection (depth 1) | DRFR: 0.661 -> 0.631 | Hurt |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · Selection (depth 2) | DRFR: 0.561 -> 0.520 | Hurt |
| Decompose instructions and execute them step-by-step (ablation) | GPT-3.5-Turbo-1106 · Selection (depth ≥ 3) | DRFR: 0.475 -> 0.411 | Hurt |
Surprising findings
- Decomposing instructions into multi-round step-by-step execution lowered GPT-3.5-Turbo performance, with the largest drops on deeply nested compositions (e.g., -0.067 on Selection & Chain depth >=3).
- GPT-4 achieves only 14.9% accuracy on the coherent test for multi-layer nested Selection instructions.
Limitations
- Constructed primarily from Chinese reference instructions, limiting multilingual coverage.
- Relies on GPT-4-1106 as LLM judge, with known biases (verbosity, self-enhancement) and cost/data-leakage concerns.
- Decomposition analysis suggests cumulative errors in multi-round execution; benchmark cannot be solved by simple decomposition.
Experimental setup
Datasets: reference instructions from real-world application scenarios open-source instruction following benchmarks
Benchmarks: ComplexBench IFEval HumanEval MATH
Metrics: DRFR pairwise agreement question-level agreement Pearson correlation
Baselines: GPT-4-1106 Claude-3-Opus GLM-4 ERNIEBot-4 GPT-3.5-Turbo-1106 Qwen1.5-72B-Chat Llama-3-70B-Instruct Direct Scoring
Builds on: InfoBench IFEval AlignBench CELLO
Closest papers by meaning
- Can Large Language Models Understand Real-World Complex Instructions? (AAAI 2024 · similarity 0.93)
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.91)
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.89)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.89)
- LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.