AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

On the Power of Context-Enhanced Learning in LLMs

Xingyu Zhu, Abhishek Panigrahi, Sanjeev Arora

ICML 2025 · Large language models › Language, safety and interpretability · approach: In-context learning analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes context-enhanced learning (gradient training where curriculum text is in context but receives no loss) and shows on a synthetic multi-level translation task that it can be exponentially more sample-efficient than SFT, with the benefit tied to more accurate gradients.

Problem

Whether gradient-based LLM training can benefit from helpful material placed in context without auto-regressive loss on that material, and whether such learning is more sample-efficient than standard SFT and leaks less recoverable information.

Approach

Train with standard auto-regressive loss on targets while conditioning on curriculum text (phrasebook excerpts with dropout) that receives no loss; this is a gradient-based analog of in-context learning. Theory uses a surrogate transformer model on a multi-level translation task to show exponential sample-complexity gap, attributed to more accurate gradient signals.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use annealed rule dropout with correct phrasebook context instead of no contextLlama 3.2-3B ICL-capable model on multi-level translation · Multi-level translation, n=8 and n=10training samples needed: 10x less samplesImproved
Train with subsets of phrasebook rules and 20% dropoutLlama 3.2-3B ICL-capable model · Multi-level translation heldout testnext-token prediction accuracy: perfect heldout test-time accuracy with 100% dropout rateImproved
Use fixed rule dropout instead of annealed rule dropoutLlama 3.2-3B ICL-capable model · Multi-level translation, n=8 and n=10sample efficiencyHurt
Remove dropout while providing the relevant phrasebook rules (ablation)Context-enhanced learning · Multi-level translationtest accuracy without contextHurt
Replace correct phrasebook rules with incorrect rules (ablation)Annealed-dropout context-enhanced learning · Multi-level translationtest accuracyHurt
Start from a non-ICL-capable 3B base model (ablation)Annealed-dropout context-enhanced learning · Multi-level translationtest accuracyHurt
Include unused phrasebook rules in context during training (ablation)Annealed-dropout context-enhanced model · Multi-level translation, heldout rule splittranslation accuracy on unused rulesNo clear effect
Filter generation to tokens in alphabet A_{i+1} (ablation)Context-enhanced learning model queried for phrasebook rules · Phrasebook rule recovery, d=5, n=10rule recovery success rate: 0.00% -> 1.66%; 0.00% -> 1.28%; 0.89% -> 0.94%Mixed
Increase the rule dropout rateICL-capable surrogate model · Gradient prediction accuracygradient prediction accuracyHurt
Drop rules from multiple phrasebooks rather than fewer phrasebooksICL-capable surrogate model · Gradient prediction accuracygradient prediction accuracyHurt
Increase the gradient-computation batch sizeICL-capable surrogate model · Gradient prediction accuracygradient prediction accuracyImproved

Surprising findings

Limitations

Experimental setup

Datasets: synthetic multi-level translation (MLT) samples

Benchmarks: held-out synthetic MLT translation tasks

Metrics: next-token accuracy on final answer tokens recovery success rate for phrasebook rules gradient prediction accuracy l2 norm of hidden representation change

Baselines: vanilla SFT no-dropout context ablation wrong-context ablation non-ICL-capable base model

Builds on: learning using privileged information CoT internalization statistical query framework

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy