Research map / Large language models / Language, safety and interpretability
Implicit In-context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Implicit In-context Learning (I2CL), which compresses demonstrations into a context vector and injects a learned linear combination of it into residual streams, giving few-shot-level accuracy at zero-shot inference cost.
Problem
In-context learning adds large compute and KV-cache memory overhead proportional to demonstration length and is sensitive to demonstration choice and order; zero-shot inference is cheap but weak.
Approach
Extract per-demonstration MHA and MLP activations at the end token across layers, average them into a permutation-invariant context vector, and inject a learned linear combination (scalars per layer, calibrated with noisy self-calibration on the same few-shot examples) into the residual stream of the query, replacing attention to demonstration tokens.
Key results
- On Llama2-7b, I2CL reaches 75.12% average accuracy across nine tasks versus 58.13% zero-shot and 76.37% few-shot ICL, at zero-shot inference cost.
- I2CL with 0.13K trainable parameters outperforms LoRA (73.15%) and IA3 (73.00%) and prompt-tuning (54.94%) on the same nine tasks.
- Under poorly chosen demonstrations, ICL drops about 7% while I2CL drops about 0.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace zero-shot inference with I2CL context vectorization and injection | Llama2-7b · Nine-task average | accuracy: 58.13 -> 75.12 | Helped |
| Replace standard few-shot ICL with I2CL | Llama2-7b · Nine-task average | accuracy: 76.37 -> 75.12 | Hurt |
| Replace random noise with demonstration-derived context vectors | I2CL · Llama2-7b, nine-task average | accuracy: 34.61 -> 75.12 | Helped |
| Transfer similar-task context vectors and coefficients | I2CL · SST-5 | accuracy: 39.12 -> 43.24 | Helped |
| Transfer similar-task context vectors and coefficients | I2CL · MR | accuracy: 87.68 -> 89.99 | Helped |
| Add I2CL while retaining the ICL demonstration tokens | ICL · Nine-task average | accuracy: 75.21 -> 81.96 | Helped |
| Use MHA+MLP activations rather than MHA alone (ablation) | I2CL · Llama2-7b, nine-task average | accuracy: 66.97 -> 75.12 | Helped |
| Use MHA+MLP activations rather than MLP alone (ablation) | I2CL · Llama2-7b, nine-task average | accuracy: 70.27 -> 75.12 | Helped |
| Use MHA+MLP activations rather than hidden states (ablation) | I2CL · Llama2-7b, nine-task average | accuracy: 56.80 -> 75.12 | Helped |
| Inject into all layers rather than early layers only (ablation) | I2CL · Llama2-7b, nine-task average | accuracy: 58.18 -> 75.12 | Helped |
| Inject into all layers rather than middle layers only (ablation) | I2CL · Llama2-7b, nine-task average | accuracy: 64.07 -> 75.12 | Helped |
| Inject into all layers rather than late layers only (ablation) | I2CL · Llama2-7b, nine-task average | accuracy: 64.03 -> 75.12 | Helped |
Surprising findings
- Random input-label mappings barely hurt context vector formation, mirroring ICL findings, while randomly permuting words in a demonstration degrades it.
- Allowing negative coefficients (deleting information from residual streams) outperforms pure addition or convex-combination injection formulas.
Limitations
- Evaluated only on standard text classification tasks, not open-ended generation or multi-hop reasoning.
- Requires access to and caching of intermediate activations, which is infeasible for commercial closed models.
- Experiments limited to modest-sized open models (up to ~8B parameters).
Experimental setup
Datasets: SST-2 SST-5 EmoC TREC AGNews MR DBPedia Subj hate_speech18
Benchmarks: SST-2 SST-5 TREC AGNews Subj HateSpeech18 DBPedia EmoC MR
Metrics: macro-average accuracy inference speed cached memory cached parameter count
Baselines: Zero-shot Few-shot (ICL) Noise vector Label-anchor Task-vector ICV AutoCompressors ICAE CEPE Prompt-tuning LoRA IA3
Builds on: in-context vector task vector label words are anchors linear representation hypothesis IA3
Closest papers by meaning
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.91)
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.91)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.90)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.90)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.89)
- Rethinking Invariance in In-context Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.