Research map / Large language models / Language, safety and interpretability
XIFBench: Evaluating Large Language Models on Multilingual Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces XIFBench, a constraint-based multilingual instruction-following benchmark (558 instructions, six languages across high/medium/low resource levels) that evaluates LLMs with English requirements as shared semantic anchors.
Problem
Multilingual instruction-following is poorly evaluated: existing evaluations use simple instructions or coarse scoring, and constraint-based benchmarks cover mainly English and Chinese and omit semantically rich constraints.
Approach
Augments seed instructions with 0-5 constraints from a 5-category, 21-dimension taxonomy excluding language-dependent constraints, decomposes them into binary English evaluation requirements, and uses those fixed English requirements as semantic anchors for judging responses in all six languages, with cultural accessibility annotation and constraint-level translation validation.
Key results
- GPT-4o and Gemini-2.0-Flash reach about 92% average RFR but only about 72% and 75% average IFR across six languages.
- Open-source models show steep IFR drops in low-resource languages; Qwen-2.5-72B falls to 10.4% IFR in Swahili versus 67.7% in English.
- The proposed English-requirement judging protocol achieves 94.7% average agreement with humans (std 1.6), versus 60.7% for direct scoring.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use shared English evaluation requirements instead of translated requirements | XIFBench evaluation protocol · XIFBench agreement study, Avg. | Agreement rate: 88.5% vs. 94.7% | Helped |
| Use shared English evaluation requirements instead of translated requirements | XIFBench evaluation protocol · XIFBench agreement study | Cross-lingual standard deviation: 3.10 vs. 1.58 | Helped |
| Use requirement-based evaluation rather than direct holistic scoring | XIFBench evaluation protocol · XIFBench agreement study, Avg. | Agreement rate: 60.7 vs. 94.7 | Helped |
| Increase model capacity | LLMs evaluated on XIFBench · XIFBench multilingual evaluation | IFR | Helped |
| Increase instruction complexity by adding constraints | Gemini-2.0-Flash, Qwen-2.5-72B, and GLM-4-9B · XIFBench Hard Set, across languages | IFR | Hurt |
| Evaluate style and situation constraints rather than format and numerical constraints | LLMs evaluated on XIFBench · XIFBench Hard Set, lower-resource languages | RFR | Hurt |
| Evaluate culturally specific rather than culturally universal instructions | LLMs evaluated on XIFBench · XIFBench, especially lower-resource languages and smaller models | IFR | Hurt |
| Evaluate responses in lower-resource languages instead of higher-resource languages | LLMs evaluated on XIFBench · XIFBench, Hindi and Swahili versus English and Chinese | IFR | Hurt |
Surprising findings
- Translating evaluation requirements into the response language lowers judge-human agreement (88.5% vs 94.7%) and cross-lingual consistency.
- Direct scoring's high agreement in Swahili reflects poor model performance coinciding with human labels rather than greater reliability.
Limitations
- Relies on LLMs for construction and evaluation, which may carry LLM-evaluate-LLM biases.
- Cultural accessibility annotations derived from English contexts rather than native speakers.
- Excludes reasoning-specialized models and code-mixed instructions.
Experimental setup
Datasets: AlpacaEval WizardLM LIMA
Benchmarks: XIFBench
Metrics: Requirement Following Rate (RFR) Instruction Following Rate (IFR) agreement rate with human annotation
Baselines: Direct Scoring (DS) Ours w/ Trans. Reqs.
Builds on: InfoBench IFEval FollowBench AlpacaEval
Closest papers by meaning
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.91)
- Can Large Language Models Understand Real-World Complex Instructions? (AAAI 2024 · similarity 0.91)
- Benchmarking Complex Instruction-Following with Multiple Constraints Composition (NeurIPS 2024 · similarity 0.91)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.90)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.90)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.