Research map / Large language models / Instruction and fine-tuning
The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares the internal representation geometry of few-shot in-context learning and LoRA fine-tuning on MMLU across Llama2, Llama3 and Mistral models using density-peak clustering and intrinsic dimension analysis.
Problem
Although in-context learning (ICL) and supervised fine-tuning (SFT) often yield similar task performance, it is unknown whether they induce similar internal representations in LLMs.
Approach
Applies a density-based mode-seeking clustering (Advanced Density Peaks, with intrinsic dimension estimated by Gride) to last-token hidden representations, showing that ICL organizes early layers hierarchically by MMLU subject while fine-tuning sharpens answer-letter clusters in later layers, with a transition at an intrinsic-dimension peak around the middle layers.
Key results
- Few-shot (5-shot) Llama3-8b early-layer clusters reach ARI of 0.82 with MMLU subjects, versus below 0.27 at 0-shot.
- Fine-tuned Llama3-8b reaches ARI of about 0.25 with answer letters in the second half of the network, while 5-shot ICL stays below 0.05 in Llama3-8b and Mistral.
- Intrinsic dimension peaks around layer 16 in Llama3-8b (14 base, 16.5 with 5-shot, 21 fine-tuned), marking a transition between the two phases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a 5-shot prompt instead of using 0-shot prompting | MMLU evaluation across the reported models · MMLU test | micro accuracy: 62.1 -> 65.1 (Llama3-8b); 75.0 -> 78.5 (Llama3-70b); 55.9 -> 62.1 (Mistral-7b); 37.5 -> 45.9 (Llama2-7b); 52.0 -> 54.8 ( | Helped |
| Add a 5-shot prompt to organize early representations by subject | Llama3-8b, Llama3-70b, and Mistral-7b · MMLU subjects, early layers | ARI: below 0.27 -> 0.82 (Llama3-8b); below 0.27 -> 0.72 (Llama3-70b); below 0.27 -> 0.63 (Mistral-7b) | Helped |
| Use LoRA fine-tuning rather than few-shot prompting to structure later representations by answer letter | Llama3-8b, Llama3-70b, and Mistral-7b · MMLU answers, later layers | ARI: few-shot below 0.05; fine-tuned approximately 0.25 (Llama3-8b), 0.45 (Llama3-70b), 0.2 (Mistral-7b) | Helped |
| Increase few-shot context length | Early hidden representations across the analyzed LLMs · MMLU subjects, early layers | ARI: Llama2-7b: 0.76; Llama2-13b: 0.64; Llama2-70b: 0.80 (5-shot) | Helped |
| Use 5-shot prompting instead of fine-tuning on the additional QA dataset mixture | Llama3-8b · Additional QA dataset mixture | accuracy: 57% -> 58% (5-shot to fine-tuned) | No clear effect |
Surprising findings
- In Llama3-70b, the late-layer representation landscape of few-shot ICL begins to resemble that of fine-tuned models, with ARI around 0.35-0.40.
- Zero-shot performance improvement alone also increases subject alignment of early-layer clusters (ARI rises with accuracy).
Limitations
- Density estimation reliability is limited at intrinsic dimensions above roughly 15-20, and ID values here are mostly below 16.
- Only MMLU (plus one additional STEM mixture) and only few-shot ICL were analyzed; other QA datasets and prompt structures were not tested.
- The mechanism behind the transition and information flow from context to last token is not interpreted.
Experimental setup
Datasets: MMLU
Benchmarks: MMLU
Metrics: adjusted rand index intrinsic dimension accuracy (macro/micro) number of density peaks core point fraction neighborhood overlap
Baselines: 0-shot 5-shot in-context learning
Builds on: Advanced Density Peaks Gride LoRA DADApy
Closest papers by meaning
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.90)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.89)
- Why Larger Language Models Do In-context Learning Differently? (ICML 2024 · similarity 0.88)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.88)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.