Research map / Large language models / Instruction and fine-tuning
Capability Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes capability instruction tuning and MODEL-SAT, a router that uses natural-language aptitude-test descriptions of candidate LLMs plus a capability encoder and a Phi-3-Mini LLM to predict, per instruction, which model in a zoo will answer correctly, without running candidates.
Problem
Selecting the best LLM for each instruction from a model zoo without running every candidate on the instruction (no candidate inference, no target ground truth), to improve overall performance at low cost.
Approach
Represent each candidate model by its natural-language results on 50 MMLU-derived 20-shot aptitude tasks, form capability instructions (capability representation + user instruction + yes/no performance inquiry), and train a lightweight LLM with a capability encoder to score whether a model can handle the instruction, using in-batch negative sampling over positive and negative instructions.
Key results
- On the smaller-scale LLM zoo (5 models under 10B), MODEL-SAT reaches a mean of 75.28 vs 76.90 for Meta-Llama-3-70B and 74.16 for the best single model in the zoo.
- On the high-performance zoo of three 70B-class models, MODEL-SAT achieves 81.17 mean, with 85.64 on MMLU and 73.63 on ARC-Challenge.
- Compared with re-ranking baselines BGE Large (69.88) and GTE Large (70.67) on the smaller-scale zoo, MODEL-SAT reaches 75.28 mean accuracy without candidate inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MODEL-SAT for instruction-level selection in the smaller-scale LLM zoo | Smaller-scale LLM zoo · MMLU, WinoGrande, ARC-C, BoolQ, TruthfulQA, MRPC, MNLI | Mean: 74.16 -> 75.28 | Helped |
| Add three earlier-released, weaker, smaller LLMs to the smaller-scale zoo | MODEL-SAT · Smaller-Mixed LLM zoo | Mean: 75.28 -> 74.65 | Hurt |
| Add two medium-scale models to the smaller-scale zoo | MODEL-SAT · Middle-Mixed LLM zoo | Mean: 75.28 -> 76.28 | Helped |
| Add three 70B-class models to the mixed zoo | MODEL-SAT · Larger-Mixed LLM zoo | Mean: 76.28 -> 80.03 | Helped |
| Route among only three models above 70B parameters | MODEL-SAT · High-Performance LLM zoo | Mean: 80.03 -> 81.17 | Helped |
| Add the capability encoder to map capability representations to instruction semantics (ablation) | Capability instruction tuning · MMLU | score: 74.71 -> 79.86 | Helped |
| Use MODEL-SAT rather than the best-performing individual model in the smaller-scale zoo | Smaller-scale LLM zoo · ARC-C | score: 71.08 -> 72.53 | Helped |
| Use MODEL-SAT for multimodal model selection | Multimodal LLM zoo · MMMU-VAL, AI2D-TEST, CCBench | score | Mixed |
Surprising findings
- Combined coverage of smaller early-release LLMs (Phi-1 plus others) covers about 90% of the instructions GPT-4o answers correctly, even though each is much weaker.
- Re-ranking methods (Cappy, BGE, GTE) often fail to beat simple selection, suggesting they retrieve semantically similar outputs rather than optimizing performance.
Experimental setup
Datasets: MMLU WinoGrande ARC-Challenge BoolQ TruthfulQA MRPC MNLI GLUE MMMU AI2D CCBench
Benchmarks: MMLU WinoGrande ARC-Challenge BoolQ TruthfulQA MRPC MNLI MMMU AI2D CCBench
Metrics: accuracy average accuracy
Baselines: Random Selection Cappy BGE Large GTE Large Best-Performing single model RoBERTa + MLP RoBERTa + k Nearest Neighbors RoBERTa + Random Forest Phi-3 Mini-128K
Builds on: Cappy Model Spider Wings learnware
Closest papers by meaning
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.89)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.88)
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.88)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- Focus On This, Not That! Steering LLMs with Adaptive Feature Specification (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.