Research map / Large language models / Language, safety and interpretability

Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning

ICLR 2025 · Large language models › Language, safety and interpretability · approach: In-context learning analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how task recognition and task learning compete during LLM pre-training, correlates this competition with final ICL performance, analyzes pre-training factors, and proposes adaptive ensemble fusion of TR and TL checkpoints at inference.

Problem

The mechanism behind the emergence of in-context learning during pre-training, specifically how task recognition and task learning relate to each other and to ICL performance, is unclear.

Approach

Measures TR (random-label) and TL (abstract-label) accuracy across pre-training checkpoints, defines competition metrics where the two abilities change in opposite directions, finds competition intensity negatively correlates with final ICL, and fuses TR-best and TL-best checkpoints via performance-weighted ensembling.

Key results

• Competition between TR and TL is negatively correlated with final ICL performance, with Pearson coefficient -0.591.
• Adaptive fusion of Pythia-1B (TR) and Pythia-2.8B (TL), 3.9B total parameters, reaches 56.83 average accuracy vs 55.66 for CrystalCoder-7B.
• Adaptive fusion of Pythia-1B and MiniCPM-2B (3.8B total) reaches 55.85 average accuracy with best-TR/best-TL checkpoints, outperforming single models.

Measured effects reported in the paper

Surprising findings

• Scaling model size makes competition appear earlier but reduces its average intensity, with the average intensity scaling as a power law with model size.
• Two small fused models (e.g., Pythia-1B + Pythia-2.8B, 3.9B total) outperform a 7B model in average accuracy.

Limitations

• Focuses on classification tasks only; other tasks such as chain-of-thought are not studied.
• Limited to LLMs up to 12B parameters due to compute constraints.
• Evaluates on 1000 sampled examples per dataset due to computational constraints.

Experimental setup

Datasets: SST-2 financial_phrasebank emotion poem_sentiment TREC tweet_eval_atheist tweet_eval_feminist tweet_eval_hate ethos_race ethos_gender ethos_national_origin ethos_religion

Benchmarks: SST-2 financial_phrasebank emotion poem_sentiment TREC tweet_eval_atheist tweet_eval_feminist tweet_eval_hate ethos_race ethos_gender ethos_national_origin ethos_religion

Metrics: accuracy competition intensity competition ratio Pearson correlation

Baselines: Amber-7B Pythia-6.9B Pythia-12B OLMo-7B Baichuan2-7B CrystalCoder-7B Pythia-1B Pythia-2.8B MiniCPM-2B fixed-weight fusion

Builds on: task recognition and task learning decomposition of ICL Pythia suite

Closest papers by meaning

• In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.92)
• Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.91)
• Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.91)
• Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.90)
• Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.90)
• Strategy Coopetition Explains the Emergence and Transience of In-Context Learning (ICML 2025 · similarity 0.90)