Research map / Large language models / Language, safety and interpretability
Can Large Language Models Understand Real-World Complex Instructions?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CELLO, a benchmark of 523 real-world complex instructions (eight complexity features, nine tasks) with four automatic criteria and metrics, used to evaluate 34 Chinese- and English-oriented LLMs.
Problem
LLMs struggle with complex instructions (multi-task or constrained task descriptions, long/noisy/heterogeneous/multi-turn inputs), but existing benchmarks are close-ended, simple, and use coarse binary or GPT-4-judged metrics.
Approach
A systematic benchmark built from real chatbot logs and expert prompts, with data evolution (in-breadth and in-depth) and four objective, fine-grained criteria (count limit, answer format, task-prescribed phrases, input-dependent query) scored by rule-based sub-metrics such as keyword coverage, parseability, and COPY-BLEU penalty.
Key results
- OpenChat-V3.2 (13B) was the best open model with 0.720 average score, while GPT-4 reached 0.822 and GPT-3.5-turbo 0.794.
- Vicuna-V1.5-13B scored 0.699 average, rivaling ChatGLM (0.673) and showing small models can match larger ones.
- Under the same Llama2-7B base, models varied widely (e.g., Llama2-chat-7B 0.385 vs. Llama2-LinkSoul 0.629), suggesting instruction tuning matters strongly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenChat-V3.2 rather than the other evaluated open-source models | open-source LLMs · CELLO All | average score: 0.699 -> 0.720 | Helped |
| Use a close-source model rather than an open-source small to medium-scale model | LLMs · CELLO All | average score: 0.720 -> 0.794; 0.720 -> 0.822 | Helped |
| Increase model size within the same model series | LLMs on complex input · CELLO Complex Input | average score | Helped |
| Increase model size within the same model series | LLMs on complex task descriptions · CELLO Complex Task Description | average score | Mixed |
| Use a longer supported context length | LLMs on complex input · CELLO Complex Input | average score | Hurt |
| Use more Chinese training data | LLMs on complex input text · CELLO Complex Input | average score | Helped |
Surprising findings
- Small open models (13B) such as OpenChat-V3.2 and Vicuna-V1.5-13B can follow complex instructions about as well as larger models in the same family.
- Performance differences between models on the count-limit criterion are small compared to other criteria, suggesting similar numerical comprehension.
Limitations
- Open-source small to medium models show a large performance gap relative to GPT-3.5-turbo and GPT-4.
- Synthetic data evolved with GPT-3.5-turbo, which the authors note is far from a perfect automated agent; manual correction was needed for planning tasks.
Experimental setup
Datasets: real-world chatbot user logs (90,000 interactions) expert-crafted meta-prompts CN-DBpedia
Benchmarks: CELLO
Metrics: count limit score answer format score task-prescribed phrases score input-dependent query score keyword coverage COPY-BLEU penalty
Baselines: C-Eval AGIEval Kola WizardLM Testset ToolBench AgentBench HumanEval
Builds on: Evol-Instruct COPY-BLEU
Closest papers by meaning
- Benchmarking Complex Instruction-Following with Multiple Constraints Composition (NeurIPS 2024 · similarity 0.93)
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.91)
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR 2024 · similarity 0.90)
- LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.90)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.89)
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.