Research map / Label-efficient and robust learning / Recognition and long-tail
Disentangling Latent Shifts of In-Context Learning with Weak Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WILDA, which uses ICL outputs as pseudo-labels to train a LoRA adapter (student) that reproduces demonstration-induced behavior from the query alone, and shows the student often beats its ICL teacher on GLUE and MMLU subsets.
Problem
In-context learning is unstable and scales poorly with demonstration count due to context length, ordering sensitivity and inference cost; existing latent-shift disentanglement methods rely on linear-attention approximations that intervene in attention heads or hidden states.
Approach
Treat ICL predictions as weak supervision: a teacher with demonstrations generates pseudo-labels on unlabeled queries, and a student (same LLM with a LoRA adapter toggled on, seeing only the query) is trained with cross-entropy to match them, encoding the demonstration-induced latent shift into reusable adapter weights that can be arithmetically merged and composed with in-context demonstrations.
Key results
- WILDA-S with Llama 3 (8B) improves over 16-shot ICL by 2.6% to 11.9% absolute across ID datasets, with statistically significant gains.
- On ARC-Challenge, WILDA-S reaches 59.2% vs 54.5% for 16-shot ICL with Llama 3 (8B).
- WILDA-S shows lower standard deviation across demonstration orderings than ICL on all datasets for Llama 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WILDA-S instead of 16-shot ICL | Llama 3 (8B) · GLUE and MMLU, 16-shot ID | generalization score: absolute improvements ranging from $2.6\%$ to $11.9\%$ | Helped |
| Shuffle the same demonstrations at the beginning of each training epoch (WILDA-S versus WILDA-F) (ablation) | WILDA · GLUE and MMLU, 16-shot ID | generalization score and stability | Helped |
| Resample demonstrations from the labeled pool each epoch (WILDA-R instead of fixed demonstrations) (ablation) | WILDA · GLUE and MMLU, 16-shot ID | stability | Hurt |
| Keep one fixed demonstration set throughout training (WILDA-F) rather than shuffling its order (ablation) | WILDA · GLUE and MMLU, 16-shot ID | stability | Hurt |
| Increase unlabeled training instances from 200 to 500 (ablation) | WILDA-S · GLUE, 16-shot ID | generalization score | Mixed |
| Sum adapters trained on multiple demonstration subsets (8×16 rather than a single subset) | WILDA-S · GLUE and MMLU ID | generalization score | Helped |
| Combine 16 encoded demonstrations with 16 demonstrations at inference rather than use standard 32-shot ICL | WILDA-S · GLUE and MMLU, 16/16 configuration | accuracy | Helped |
| Use 32 encoded demonstrations rather than 16 encoded plus 16 in-context demonstrations | WILDA-S · GLUE and MMLU | accuracy | Mixed |
| Use WILDA-S (0/16) rather than 16-shot ICL | Llama 3 (8B) · RTE, SST, QNLI, MNLI, CoLA, MRPC, QQP, MMLU Math and Misc | accuracy: 75.3 vs 86.0; 93.2 vs 96.1; 77.7 vs 81.4; 69.1 vs 73.1; 58.3 vs 64.3; 76.4 vs 77.7; 74.2 vs 73.1; 43.0 vs 49.5; 84.5 vs | Helped |
| Increase the number of ICL demonstrations while retaining WILDA-S training (ablation) | WILDA-S · GLUE and MMLU ID, Llama 3 (8B) | generalization score | Helped |
| Enable a task-specific adapter during evaluation on an unrelated domain (ablation) | WILDA-S · Cross-domain GLUE dataset pairs | accuracy | No clear effect |
Surprising findings
- The student often outperforms its own teacher (standard ICL), attributed to pseudo-label correction and coverage expansion.
Limitations
- Adapter fine-tuning adds computational overhead; inference with 16 demonstrations is about 120x slower than 0-shot.
- Requires a supply of unlabeled data.
- Effectiveness with very large demonstration sets is unexplored.
Experimental setup
Datasets: GLUE MMLU elementary math MMLU miscellaneous
Benchmarks: GLUE MMLU ARC-Challenge
Metrics: accuracy Matthew's correlation F1 standard deviation across seeds
Baselines: zero-shot standard n-shot ICL PBFT ICV Batch-ICL
Builds on: LoRA in-context vectors Batch-ICL weak-to-strong generalization
Closest papers by meaning
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.89)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Implicit In-context Learning (ICLR 2025 · similarity 0.89)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.88)
- MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.