Research map / Large language models / Agents, code and math
Unearthing Skill-level Insights for Understanding Trade-offs of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes parsing model-generated step-by-step rationales (GPT-4o) to tag each evaluation instance with latent skills, aggregates them into cross-benchmark skill-slices over 12 benchmarks, and uses them to analyze model trade-offs, route instances, and probe skills.
Problem
Aggregate accuracy hides skill-level differences between foundation models, and manually annotating fine-grained skills for evaluation instances does not scale across benchmarks.
Approach
Use a strong model's rationales, with each step labeled by a skill named at multiple granularities, to automatically infer per-instance skills; group instances across benchmarks into skill-slices and analyze accuracy per slice, validated by automatic verifiers and human checks.
Key results
- Gemini 1.5 Pro is about 18% more accurate than GPT-4o and Claude 3.5 Sonnet on computing molar mass but 19% less accurate on applying constitutional law, though overall accuracies differ by 0.4%.
- Skill-based routing of each instance to the best model on its relevant skills improves overall accuracy by up to 3.2% over each single frontier model across 12 datasets, with 3.5-6.8% gains on MMLU Pro.
- 94.1% of GPT-4o-annotated skills are verified as relevant by automatic verifiers; human validation gives 95.7% relevancy and 92.8% agreement with automatic verifiers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Route each instance to the model strongest on its annotated skills instead of selecting one model for all instances | GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet · All 12 datasets | accuracy: up to $3.2\%$ | Helped |
| Route instances using skill-slice accuracies rather than random model selection | GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet · MMLU Pro | accuracy: 58.02 -> 62.94 | Helped |
| Use Gemini 1.5 Pro instead of GPT-4o | Foundation model performance · computing molar mass skill | accuracy: $18\%$ more accurate | Helped |
| Use Gemini 1.5 Pro instead of GPT-4o and Claude 3.5 Sonnet | Foundation model performance · applying constitutional law skill | accuracy: $19\%$ less accurate | Hurt |
| Replace direct skill prompting with rationale parsing (ablation) | GPT-4o skill annotation · 1195 instances from 12 benchmarks | skills annotated per instance: $6.32$ -> $9.98$; $58\%$ relative improvement | Helped |
| Replace rationale parsing with direct prompting (ablation) | GPT-4o skill annotation · All 12 benchmarks, skill slices with 100–250 instances | number of skills: $491$ -> $288$; a reduction of $41\%$ | Hurt |
| Replace rationale parsing with direct prompting (ablation) | GPT-4o skill annotation · All 12 benchmarks, skill slices with fewer than 10 instances | number of skills: $2399$ -> $2494$ | Mixed |
Surprising findings
- Skill annotations stay highly relevant even when the annotating model answers the underlying instance incorrectly.
- Skill-slice accuracies computed mostly on multimodal benchmarks correlate strongly (r>=0.79) with accuracies on the language-only MMLU Pro.
Limitations
- Skills may be incomplete, since completeness is not evaluated and inter-annotator overlap is imperfect.
- Validity of rationale-listed skills may degrade on harder benchmarks where the annotator model fails.
- Rationale generation is more costly than direct prompting or embedding-based approaches.
Experimental setup
Datasets: MMLU Pro MMMU MathVista MMC MMVP MMBench MMTBench MME MMVet SEEDBench RealWorld-QA VibeEval
Benchmarks: MMLU Pro MMMU MathVista MMC MMVP MMBench MMTBench MME MMVet SEEDBench RealWorld-QA VibeEval
Metrics: accuracy skill relevancy rate precision@20 inconsistency rate Pearson correlation
Baselines: direct prompting skill annotation CLIP embedding retrieval GPT-4o attribute tagging ground-truth benchmark attributes random model selection
Builds on: slice discovery chain-of-thought rationales MMLU-Pro
Closest papers by meaning
- Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models (NeurIPS 2026 · similarity 0.87)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.86)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.86)
- SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · similarity 0.85)
- Can Models Learn Skill Composition from Examples? (NeurIPS 2024 · similarity 0.85)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.