Research map / Large language models / Alignment and preferences
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FLASK, a fine-grained, skill-set-based evaluation protocol for LLM alignment that scores 12 skills per instance (human or GPT-4 evaluators), showing higher human-model correlation and robustness to verbosity bias than coarse overall scoring.
Problem
Coarse-grained overall preference scoring of instruction-following LLMs is hard to interpret and unreliable because the skills required vary by instance and single scores miss multiple evaluation axes.
Approach
Decompose evaluation into instance-wise skill-set scoring over 12 alignment skills under 4 abilities, with skill-specific rubrics, plus an instance-specific checklist rubric for the hardest subset (FLASK-Hard); metadata (skills, domains, difficulty) is annotated automatically by GPT-4.
Key results
- Skill-specific FLASK evaluation with GPT-4 as judge reaches Spearman 0.680 with human labelers vs 0.641 for skill-agnostic evaluation.
- Open-source 13B models lag proprietary models mainly on Logical Thinking and Background Knowledge skills, while Problem Handling and User Alignment are closer.
- Even GPT-4 performs poorly on Logical Correctness and Factuality in FLASK-Hard, with up to 50% degradation for some skills versus the full set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use skill-specific fine-grained rubrics instead of skill-agnostic scoring | Model-based FLASK evaluation · Human-model evaluation | Spearman correlation: 0.641 -> 0.680 | Helped |
| use skill-specific fine-grained rubrics instead of skill-agnostic scoring | Model-based FLASK evaluation · Human-model evaluation | Kendall-Tau correlation: 0.495 -> 0.541 | Helped |
| use skill-specific fine-grained rubrics instead of skill-agnostic scoring | Model-based FLASK evaluation · Human-model evaluation | Pearson correlation: 0.673 -> 0.732 | Helped |
| add a reference answer, rationale generation, and score rubric (ablation) | Model-based FLASK evaluation · Human-model evaluation | correlation: 0.516; 0.634; 0.646 | Helped |
| add RLHF after supervised fine-tuning (ablation) | Vicuna-13B · FLASK Logical Thinking | skill score: 3.96; 3.13; 1.57 | Helped |
| add RLHF after supervised fine-tuning (ablation) | Vicuna-13B · FLASK factuality, commonsense, problem handling, and user alignment skills | skill score: -6.96; -3.92; -6.41; -11.86; -9.18; -9.90; -2.49; -3.48; -1.37 | Hurt |
| train on more difficult instructions (ablation) | LLaMA-13B fine-tuned on Evol-Instruct versus ShareGPT · FLASK | skill performance | No clear effect |
| train on Code-Alpaca instead of Alpaca (ablation) | LLaMA-13B · FLASK Logical Thinking | skill performance | Helped |
| train on FLAN V2 instead of other fine-tuning datasets (ablation) | LLaMA-13B · FLASK long-form generation | skill performance | Hurt |
| train on FLAN V2 (ablation) | LLaMA-13B · FLASK subset with short reference answers | skill performance | No clear effect |
| use GPT-4-generated responses instead of InstructGPT-generated responses for instruction tuning (ablation) | Alpaca-13B · FLASK | skill performance | Helped |
| use GPT-4-generated responses for instruction tuning (ablation) | Alpaca-13B · FLASK compared with GPT-3.5 | skill performance | Hurt |
Surprising findings
- Open-source models imitate the style of proprietary models for many skills but not factuality or logical thinking.
- Fine-grained rubrics increase robustness of model-based evaluation to verbosity changes, with instance-specific rubrics most robust.
Limitations
- Human evaluation covers only 200 instances and 4 models with moderate inter-labeler agreement (0.488).
- Instance-specific rubrics are applied only to the 89-instance FLASK-Hard subset due to cost.
- Scope limited to English, single-turn, zero-shot instructions.
Experimental setup
Datasets: MMLU GSM8K ShareGPT FLAN V2 Alpaca Code-Alpaca Evol-Instruct
Benchmarks: FLASK FLASK-Hard
Metrics: Spearman correlation Kendall-Tau correlation Pearson correlation Krippendorff's alpha score robustness ratio
Baselines: GPT-3.5 Bard Vicuna-13B Alpaca-13B Claude InstructGPT WizardLM-13B LLaMA2-Chat 70B ROUGE-L Skill-agnostic evaluation
Builds on: Skill-agnostic reference-guided evaluation (Vicuna/MT-Bench style judging) Bloom's-style skill taxonomy for QA
Closest papers by meaning
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.89)
- DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation (AAAI 2024 · similarity 0.89)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.89)
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · similarity 0.88)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.