Research map / Large language models / AI and society

SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models

ICLR 2024 · Large language models › AI and society · approach: Social-science view of AI evaluation · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SKILL-MIX, an evaluation that asks LLMs to write short text combining random k-subsets of language skills on a random topic, auto-graded by GPT-4 and LLaMA-2-70B with human spot-checks.

Problem

Existing LLM benchmarks are vulnerable to training-set contamination and leaderboard cramming and saturate quickly, so they fail to measure whether models can flexibly combine basic skills in novel ways.

Approach

Generate evaluation prompts on the fly by randomly sampling k skills from a list of N skills plus a low-frequency topic, so the number of combinations (C(N,k)·T) makes memorization unlikely; grade with a rubric using an LLM grader and use a probability bound to argue for beyond-stochastic-parrot behavior.

Key results

• GPT-4 keeps a Ratio of Full Marks of 0.36 at k=5 and 0.29 at k=6 under GPT-4 grading, the only model that stays above 0.05 at k=5 with that grading.
• Under GPT-4 grading, most open models (LLaMA-2 7B/13B/70B, Mistral, Falcon-180B, Qwen-14B, Tigerbot) saturate by k=2 or k=3, and Falcon-180B-Chat and Tigerbot-70B-Chat score below LLaMA-2 70B despite higher Open LLM Leaderboard ranks.
• Probability calculation suggests GPT-4's k=5 and k=6 correct outputs include novel skill-topic combinations for at least one-third of cases under the stated assumptions.

Measured effects reported in the paper

Surprising findings

• Models ranked higher on the Open LLM Leaderboard (Falcon-180B-Chat, Tigerbot-70B-Chat, Mistral-7B-Instruct) score worse than LLaMA-2 70B on SKILL-MIX.
• LLaMA-2 70B as grader is more generous than GPT-4 and shows family bias toward smaller LLaMA-2 siblings.

Limitations

• Human grading showed high variance among five graders, so human spot-checking is noisy.
• Probability estimate of beyond-stochastic-parrot behavior relies on an independence assumption among skill occurrences, verified only for k=2.

Experimental setup

Datasets: RedPajama

Benchmarks: SKILL-MIX(k)

Metrics: ratio of full marks ratio of all skills skill fraction

Baselines: GPT-3.5-turbo LLaMA-2 7B LLaMA-2 13B LLaMA-2 70B Falcon-180B-Chat Xwin-LM-70B-V0.1 Mistral-7B-Instruct-v0.1 Qwen-14B-Chat Tigerbot-70B-Chat

Builds on: Arora & Goyal (2023) skill-emergence theory Skill-It AlpacaEval Open LLM Leaderboard

Closest papers by meaning

• Can Models Learn Skill Composition from Examples? (NeurIPS 2024 · similarity 0.90)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.88)
• SkillOpt: Executive Strategy for Self-Evolving Agent Skills (NeurIPS 2026 · similarity 0.88)
• A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
• From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (NeurIPS 2026 · similarity 0.88)
• Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.87)