Research map / Large language models / Language, safety and interpretability

Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition

ICML 2025 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LLMs (GPT-3.5, Llama-3, Qwen-1.5) can in-context learn multiple distinct tasks simultaneously in one prompt ("task superposition"), that it emerges even when training on one task at a time, that transformers can express it theoretically, and that convex combinations of task vectors reproduce it.

Problem

Whether LLMs can perform multiple computationally distinct in-context learning tasks at once in a single inference call, and how this capability arises and is represented internally.

Approach

Defines "task superposition": when a prompt mixes in-context examples from several tasks, the model's output probabilities reflect a mixture of those tasks, weighted roughly by the in-context task proportions. The paper shows this empirically in pretrained LLMs, in small GPT-2 models trained one task at a time, constructs a transformer that implements it, and shows task-vector interpolation can re

Key results

• Llama-3 70B, GPT-3.5 and Qwen-1.5 72B all assign non-negligible median probability to at least two of four tasks in mixed-task prompts (Finding 1).
• GPT-2 (~86M) trained on one retrieval task at a time produces output probabilities for ret2 and ret6 near [0.5, 0.5] at a 50/50 in-context mixture (Finding 2).
• Within Qwen-1.5, larger models (7B, 14B) have higher r and lower KL divergence to the in-context task distribution than smaller ones (Finding 5).

Measured effects reported in the paper

Surprising findings

• Task superposition emerges even when the model is trained to in-context learn only one task at a time.
• Patching convex combinations of task vectors reproduces the superposition effect, though with more mass on irrelevant outputs than the in-context mixture does.

Limitations

• Standard decoding suffers 'generation collapse': after the first token the model tends to commit to a single task.
• Convex task-vector interpolation does not fully explain superposition and yields more irrelevant output mass.
• The transformer construction is existential and does not guarantee training finds these parameters.

Experimental setup

Datasets: synthetic retrieval tasks (ret1-ret8) synthetic addition tasks (plus0-plus9) country capital/continent/capitalization prompts multilingual addition prompts copy and add tasks

Benchmarks: synthetic multi-task ICL prompts (addition, translation, country facts, letter tasks)

Metrics: output probability of task answers r (number of tasks in top-K outputs) KL divergence to in-context task distribution

Builds on: task vectors Bayesian in-context learning superposition of simulators

Closest papers by meaning

• Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.89)
• From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When (NeurIPS 2024 · similarity 0.88)
• Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.88)
• The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.88)
• Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence (ICML 2025 · similarity 0.88)
• Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack (NeurIPS 2024 · similarity 0.88)