Research map / Large language models / Instruction and fine-tuning
Is In-Context Learning Sufficient for Instruction Following in LLMs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that URIAL in-context alignment of base LLMs still trails instruction fine-tuning on MT-Bench, that decoding parameters matter a lot, and that adding greedy-selected high-quality demonstrations narrows the gap, with ICL competitive with IFT only in the low-data single-turn regime.
Problem
In-context learning (URIAL) can align base LLMs to follow instructions without weight updates, but it is unclear how far it can go compared to instruction fine-tuning, which factors matter, and how it scales with more demonstrations.
Approach
Systematically evaluates URIAL across many base models, identifies decoding parameters and demonstration quality as key drivers, scales many-shot ICL with high-quality SkillMix examples, uses greedy search to pick demonstrations, and directly compares ICL vs IFT on the same small data budgets.
Key results
- URIAL with Llama-3-70B reaches 6.40 average MT-Bench vs 8.74 for Llama-3-70B-Instruct, showing ICL still lags fine-tuned models.
- Greedy search adds a 4th demonstration that raises Llama-3.1-8B URIAL 1st-turn MT-Bench from 6.95 to 7.61 and reaches 7.81 with 6 examples.
- With 3 examples, ICL and IFT achieve nearly identical 1st-turn MT-Bench on Mistral-7B-v0.2 up to 50 examples, but IFT clearly wins on 2nd-turn scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace URIAL with instruction fine-tuning | Base LLMs evaluated on MT-Bench · MT-Bench average | average score: 4.83 -> 6.53 | Helped |
| Use URIAL decoding parameters (temperature=0, top-p=1) (ablation) | Mistral-7B-v0.2 base model · MT-Bench | score: 6.61 vs 7.00 | Helped |
| Add URIAL's rule text to the prompt (ablation) | Base model with URIAL demonstrations · MT-Bench | score | No clear effect |
| Add correctly matched query-answer demonstrations (ablation) | In-context alignment · MT-Bench 1st-turn | score: 6.52 -> 6.99; 6.25 -> 6.95 | Helped |
| Provide questions without their answers (ablation) | In-context alignment · MT-Bench 1st-turn | score: 6.52 -> 5.90; 6.25 -> 4.48 | Hurt |
| Use answers only, without their questions (ablation) | In-context alignment · MT-Bench 1st-turn | score: 6.52 -> 6.94; 6.25 -> 6.83 | Helped |
| Pair demonstrations with circularly shifted answers (ablation) | In-context alignment · MT-Bench 1st-turn | score: 6.95 -> 2.13 | Hurt |
| Replace single-turn examples with two-turn examples (ablation) | URIAL · MT-Bench 2nd-turn | score: 5.31 -> 6.44 | Helped |
| Add a separation tag between groups of two-turn demonstrations (ablation) | URIAL + 3 two-turn SkillMix examples · MT-Bench 2nd-turn | score: 5.88 -> 6.31; 6.43 -> 6.71 | Helped |
| Increase two-turn demonstrations from 3 to 6 (ablation) | URIAL + two-turn SkillMix examples · MT-Bench | score | No clear effect |
| Add greedily selected high-quality SkillMix examples | URIAL on Llama-3.1-8B · MT-Bench 1st-turn | score: 6.95 -> 7.61 | Helped |
| Add single-turn demonstrations to improve multi-turn behavior (ablation) | In-context alignment · MT-Bench 2nd-turn | score | Hurt |
Surprising findings
- Base models with proper decoding parameters (temperature 0, no repetition penalty settings) already achieve reasonable MT-Bench scores without any in-context examples (6.61 vs 7.00 with URIAL).
- Scaling random in-context demonstrations beyond about 30 examples plateaus, unlike many-shot ICL on other tasks, and 2nd-turn performance falls below the base model.
Limitations
- Greedy search was run with limited compute and a small candidate pool.
- Focus is only on instruction following; safety-oriented alignment is not studied.
- Relies on GPT-4 and GPT-4-Turbo as judges for MT-Bench and AlpacaEval.
Experimental setup
Datasets: SkillMix-4k Evol-Instruct-70k MT-Bench AlpacaEval 2.0
Benchmarks: MT-Bench AlpacaEval 2.0
Metrics: MT-Bench 1st-turn score MT-Bench 2nd-turn score length-controlled win rate
Baselines: URIAL Llama-2-7B-Instruct Llama-3-8B-Instruct Llama-3.1-8B-Instruct Mistral-7B-Instruct-v0.2 GPT-4 (March 2023) Evol-Instruct IFT
Builds on: URIAL SkillMix MT-Bench AlpacaEval 2.0
Closest papers by meaning
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.90)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.88)
- The Missing Alignment Link of In-context Learning on Sequences (ICML 2025 · similarity 0.88)
- Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning (ICLR 2025 · similarity 0.88)
- Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.88)
- Teaching LLMs How to Learn with Contextual Fine-Tuning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.