Research map / Large language models / Instruction and fine-tuning
DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DINGO, a 5026-sample instruction-following evaluation dataset with a manually annotated 130-node fine-grained category tree and GPT-4-generated diverse instruction styles, and evaluates 10 LLMs with GPT-4 single-answer grading.
Problem
Existing instruction-following evaluations use coarse general skills and singular instruction expressions, so they cannot pinpoint task-level weaknesses or reflect diverse real-world user requests.
Approach
Builds a fine-grained multi-level task taxonomy from real ShareGPT user requests and uses GPT-4 in-context generation conditioned on style, attitude, and language constraints (with over-generate-then-filter) to create diverse instructions per category, grading with GPT-4 against reference answers.
Key results
- DINGO contains 5026 instructions organized in a 130-node category tree with 4 levels, derived from 7265 ShareGPT seed samples.
- Instruction-tuned models do not consistently improve over base models on fine-grained categories; e.g., vicuna2-7B does not beat Llama2-7B on Biology and Medicine.
- Models score lower on DINGO's diverse instructions than on basic questions across four categories, including Mathematical Theorem and Logical Reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction-tune Llama-13B as vicuna-13B | Llama-13B · DINGO first-level categories | average score | Helped |
| use Llama2-chat-7B instead of vicuna2-7B | Llama2-7B-based instruction-tuned models · DINGO first-level categories | average score | Helped |
| increase the size of the pre-trained model from 7B to 13B | Llama2 · DINGO first-level categories | average score | Helped |
| increase the size of the instruction-tuned model from 7B to 13B | Llama2-chat · DINGO first-level categories | average score | No clear effect |
| instruction-tune Llama2-7B as vicuna2-7B | Llama2-7B · DINGO Knowledge Utilization → Open-Book Questions → Knowledge-Intensive Questions: Biology | average score | No clear effect |
| instruction-tune Llama2-7B as vicuna2-7B | Llama2-7B · DINGO Knowledge Utilization → Open-Book Questions → Knowledge-Intensive Questions: Medicine | average score | No clear effect |
| use diverse DINGO instructions instead of the corresponding basic questions | four evaluated LLMs · DINGO Word Problems, Mathematical Theorem, Common Sense Reasoning, and Logical Reasoning | instruction-following score | Hurt |
Surprising findings
- Instruction-tuned models can underperform their base models on some fine-grained categories despite higher coarse-grained scores.
- Models follow diverse instruction styles worse than plain basic questions, e.g., ignoring concise-output requests.
Limitations
- Relies on GPT-4 both to generate instructions and to judge responses, which may introduce bias.
- GPT-4 has difficulty scoring math/code, mitigated only by including reference answers.
- Agreement with humans is checked on only 100 examples with six experts.
Experimental setup
Datasets: ShareGPT GSM8K ProofNet MATH StrategyQA LogiQA MBPP
Benchmarks: DINGO
Metrics: average score (single-answer grading, 1-10 scale) pairwise win rate agreement with human experts
Baselines: Llama-7B Llama-13B Llama2-7B Llama2-13B Vicuna-7B Vicuna-13B Llama2-chat-7B Llama2-chat-13B Vicuna2-7B Vicuna2-13B
Builds on: Vicuna FLASK InstructEval Chatbot Arena
Closest papers by meaning
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.92)
- SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models (ICLR 2025 · similarity 0.89)
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.89)
- FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets (ICLR 2024 · similarity 0.89)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.89)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.