Research map / Large language models / Language, safety and interpretability
Rethinking Invariance in In-context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Invariant ICL (InvICL), a permutation-invariant in-context learning method that satisfies information non-leakage and context interdependence via leave-one-out pre-encoding with a parallel unrolled implementation.
Problem
In-context learning is sensitive to the ordering of context examples, and existing permutation-invariant ICL variants underperform auto-regressive ICL.
Approach
Identifies information non-leakage and context interdependence as necessary properties for invariant ICL; achieves both with a two-stage leave-one-out (LOO) bag-of-examples pre-encoding, implemented in one forward pass by duplicating the context with a LOO attention mask.
Key results
- On GPT-2 Large, InvICL averages 42.4 on all target tasks vs 41.9 for AR ICL and 38.1 for PCW, and 48.4 vs 43.6 for AR ICL on unseen-domain target tasks.
- Ablation on HR→LR: AR ICL with invariant mask and symmetric PE reaches 45.1 with order sensitivity 0.00 (InvICL) vs 43.4 and 0.25 for AR ICL.
- In synthetic linear regression, InvICL outperforms AR ICL in length extrapolation beyond the training length of 40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use InvICL instead of AR ICL | GPT-2 Large · All target tasks (Avg.) | accuracy: 41.9 -> 42.4 | Helped |
| use InvICL instead of AR ICL | GPT-2 Large · Target tasks in unseen domains (Avg.) | accuracy: 43.6 -> 48.4 | Helped |
| use InvICL instead of Auto-regressive ICL | GPT-Neo 2.7B · All target tasks (Avg.) | accuracy: 41.2 -> 41.3 | Helped |
| use InvICL instead of Auto-regressive ICL | GPT-Neo 2.7B · Target tasks in unseen domains (Avg.) | accuracy: 42.1 -> 42.7 | Helped |
| use InvICL instead of Auto-regressive ICL | Pythia-2.8B · All target tasks (Avg.) | accuracy: 30.8 -> 31.9 | Helped |
| use InvICL instead of Auto-regressive ICL | Pythia-2.8B · Target tasks in unseen domains (Avg.) | accuracy: 30.5 -> 32.4 | Helped |
| add symmetric positional encoding (ablation) | AR ICL · HR→LR | accuracy: 43.4 -> 38.4 (-5.0) | Hurt |
| add symmetric positional encoding (ablation) | AR ICL · HR→LR | Sensitivity: 0.25 -> 0.30 (+0.05) | Hurt |
| add invariant mask (ablation) | AR ICL · HR→LR | accuracy: 43.4 -> 44.8 (+1.4) | Helped |
| add invariant mask (ablation) | AR ICL · HR→LR | Sensitivity: 0.25 -> 0.10 (-0.15) | Helped |
| add invariant mask and symmetric positional encoding (ablation) | AR ICL · HR→LR | accuracy: 43.4 -> 45.1 (+1.7) | Helped |
| add invariant mask and symmetric positional encoding (ablation) | AR ICL · HR→LR | Sensitivity: 0.25 -> 0.00 (-0.25) | Helped |
Surprising findings
- Symmetric positional encoding alone degrades AR ICL performance (43.4 to 38.4 on HR→LR), while combined with the invariant mask it achieves the best result.
Limitations
- Duplicating the input raises memory cost and reduces effective training data under truncation, requiring reduced context examples (16 to 8) for fair comparison.
- Gains on real-world tasks are modest on several settings; InvICL trails AR ICL on some individual tasks (e.g., Class→Class).
- The synthetic ICL training objective does not match LLM pretraining.
Experimental setup
Datasets: MetaICL 142 tasks synthetic linear regression, sparse linear regression, decision tree
Benchmarks: MetaICL 7 evaluation settings (HR→LR, Class→Class, non-Class→Class, QA→QA, non-QA→QA, non-NLI→NLI, non-Para→Para)
Metrics: accuracy MSE order sensitivity
Baselines: AR ICL NoPE Prefix ICL PCW SAICL BatchICL
Builds on: MetaICL Prefix LM PCW SAICL BatchICL
Closest papers by meaning
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.89)
- Implicit In-context Learning (ICLR 2025 · similarity 0.89)
- In-Context Learning Learns Label Relationships but Is Not Conventional Learning (ICLR 2024 · similarity 0.88)
- In-Context Learning State Vector with Inner and Momentum Optimization (NeurIPS 2024 · similarity 0.88)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.88)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.