Research map / Large language models / Language, safety and interpretability
The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that test-time training with LoRA adapters on leave-one-out in-context tasks, combined with augmented inference, substantially boosts few-shot accuracy of LMs on ARC and BIG-Bench Hard.
Problem
Language models struggle with structurally novel reasoning tasks even with in-context demonstrations; the paper asks whether updating parameters at test time using the few-shot examples can improve few-shot learning on ARC and BBH.
Approach
Build a test-time training dataset from each task's own in-context demonstrations (leave-one-out ICL tasks with permuted demonstration order and rule-based geometric augmentations), fine-tune per-task LoRA adapters at inference, and aggregate predictions with hierarchical voting over transformations.
Key results
- On an 80-task ARC validation subset, TTT raised the fine-tuned 8B model from 5% to 29% pass@2 (about 6x).
- With the BARC model plus TTT, ARC validation reaches 53.0%, and ensembling with program synthesis reaches 61.9%, comparable to average human performance of 60.2%.
- On BBH 10-shot, TTT reaches 57.8% versus 50.5% for standard ICL (7.3 points).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TTT and augmented inference to the fine-tuned 8B model | Fine-tuned Llama 3 8B · ARC validation set | Pass@2 accuracy: \num{18.25}\% -> \num{47.125}\% | Helped |
| Apply our TTT pipeline to the BARC neural model | BARC neural model · ARC validation set | Pass@2 accuracy: \num{54.375}\% -> \num{53}\% | No clear effect |
| Combine our TTT pipeline with BARC's neural model and synthesizer | BARC · ARC validation set | Pass@2 accuracy: \num{54.375}\% -> \num{62.8}\% | Helped |
| Use leave-one-out in-context tasks instead of direct input-output training (ablation) | TTT · BIG-Bench Hard, 10-shot | accuracy: \num{51.5}\% -> \num{57.8}\% | Helped |
| Use multiple shuffled versions of the in-context prompt (ablation) | TTT · BIG-Bench Hard, 10-shot | accuracy: 55.7\% -> \num{57.8}\% | Helped |
| Add loss on demonstration outputs (ablation) | TTT · BIG-Bench Hard, 10-shot | accuracy: 54.4\% -> \num{57.8}\% | Helped |
| Add loss on inputs as well as outputs (ablation) | TTT · BIG-Bench Hard, 10-shot | accuracy: \num{57.8}\% -> 55.9\% | Hurt |
| Use a shared adapter across tasks instead of task-specific adapters (ablation) | TTT · BIG-Bench Hard | accuracy | Helped |
| Use a shared adapter across tasks instead of task-specific adapters (ablation) | TTT · ARC | accuracy: 7 tasks | Hurt |
| Add geometric transformations to the TTT dataset (ablation) | TTT · ARC validation subset, 80 tasks | accuracy: 55\% -> 29\% | Helped |
| Use in-context leave-one-out tasks instead of direct I/O data (ablation) | TTT · ARC validation subset, 80 tasks | accuracy: 38\% | Helped |
| Add demonstration-output loss (ablation) | TTT · ARC validation subset, 80 tasks | accuracy: 26\% -> 29\% | Helped |
Surprising findings
- Excluding LM-generated fine-tuning data outperformed training on all data after TTT on ARC.
- For smaller models, the 1B and 3B models reach identical final accuracy after TTT, so scaling after TTT is unclear.
Limitations
- Hyperparameters for ARC were tuned on the 80-task validation subset, which may bias results.
- Possible pre-training data leakage since ARC and BBH are public.
- Semi-private ARC evaluation dropped to 47.5%, suggesting distribution shift sensitivity.
Experimental setup
Datasets: ARC public training/validation ReARC BIG-Bench Hard LARC
Benchmarks: ARC public validation ARC public evaluation BIG-Bench Hard
Metrics: pass@2 accuracy accuracy
Baselines: ICL Direct I/O Zero-shot BARC Claude 3.5 Sonnet GPT-4o OpenAI o1 preview DeepSeek r1 OpenAI o3
Builds on: LoRA test-time training meta-ICL self-consistency BARC
Closest papers by meaning
- Test-Time Training Provably Improves Transformers as In-context Learners (ICML 2025 · similarity 0.90)
- In-Place Test-Time Training (ICLR 2026 · similarity 0.90)
- Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.90)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.89)
- In-Context Principle Learning from Mistakes (ICML 2024 · similarity 0.89)
- From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.