Research map / Architectures and efficiency / Pruning and efficient tuning
Where does In-context Learning Happen in Large Language Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses layer-wise context masking on GPT-Neo, BLOOM, StarCoder2 and Llama-3.1 models to locate a "task recognition" layer after which attention to the prompt context is redundant, and links this to LoRA layer adaptability.
Problem
Where inside causal LLMs the model transitions from recognizing the in-context task to performing it, with respect to instructions and demonstration examples.
Approach
Mask attention to instructions, examples, or query from a given layer onward and measure task performance per layer to find a task-recognition point after which context attention is unnecessary; exploit this for inference savings and for choosing layers for lightweight LoRA fine-tuning.
Key results
- Layer-wise context masking shows models reach ceiling translation performance by around layer 13-16 of Llama-3.1-8B, allowing about 45-50% computational savings with 5 examples.
- Around 10% of attention heads in GPT-Neo and BLOOM can be masked via L0 gates, indicating attention heads are not redundant while attention over context is.
- LoRA fine-tuning of single early-to-middle layers performs best before the task recognition point, correlating with where task location occurs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mask attention to instructions and examples after the task-recognition layer (ablation) | LLM in-context translation · FLORES en↔fr | translation performance | No clear effect |
| Increase the number of prompt examples (ablation) | GPTNEO2.7B and BLOOM3B · FLORES en→fr | task-recognition layer | No clear effect |
| Provide instructions while masking examples (ablation) | GPTNEO2.7B and BLOOM3B · FLORES en→fr | translation performance at earlier layers | Helped |
| Mask the query rather than only task context (ablation) | LLAMA models · FLORES translation | translation performance | Hurt |
| Use instruction-tuned rather than base LLAMA3.1-8B (ablation) | LLAMA3.1 · FLORES en→fr, instructions present and examples masked | translation performance in earlier layers | Helped |
| Use 400 rather than 800 training examples for lightweight LoRA tuning (ablation) | LLAMA3.1-8B and LLAMA3.1-8B-INSTRUCT · FLORES en→fr | successful fine-tuning of task location | Hurt |
| Train LoRA on earlier-to-middle layers rather than later layers (ablation) | LLAMA3.1-8B and LLAMA3.1-8B-INSTRUCT · FLORES en↔fr | translation performance | Helped |
| Train attention head gates with L0 regularisation instead of unregularised gates (ablation) | GPTNEO2.7B and BLOOM3B · FLORES en→fr and en→pt | BLEU: 18.3; 21.4; 20.1; 24.3; 24.5; 24.1; 6.7; 15.7; 8.6; 25.9; 19.6; 25.8 | Mixed |
| Use 0-prompt rather than 5-prompt training for L0(λ=.01) gates (ablation) | GPTNEO2.7B and BLOOM3B · FLORES en→fr and en→pt | BLEU: 18.9; 19.3; 13.2; 6.4; 23.6; 24.2; 26.0 | Mixed |
Surprising findings
- Attention heads themselves are not highly redundant (only about 10% can be masked), contrasting with supervised MT pruning literature.
- Later-layer attention to the query may also be redundant in Llama models, not only attention to the context.
Limitations
- Limited explanation of why different models show different task recognition points and critical layers.
- Lacks a theoretical framework explaining the observed phenomenon.
- Only two tasks (machine translation and code generation) studied, with a few model families.
Experimental setup
Datasets: FLORES HumanEval MBPP WMT06 Europarl
Benchmarks: FLORES HumanEval
Metrics: BLEU Pass@1
Baselines: no-masking full-context performance layer-dropping (Sajjad et al.)
Builds on: LoRA in-context learning as task recognition L0 regularisation for attention heads
Closest papers by meaning
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.89)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.89)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.