Research map / Large language models / Reasoning and chain of thought
PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
Zeming Chen, Angelika Romanou, Gail Weiss, Antoine Bosselut
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PERK, which meta-trains a model to encode long contexts into low-rank adapter weights via gradient updates at test time, enabling long-context reasoning as parameter-efficient test-time learning.
Problem
Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts.
Approach
Uses two nested optimization loops during meta-training: the inner loop rapidly encodes a context into a low-rank adapter via gradient updates at test time, while the outer loop meta-learns to make this encoding effective for reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Meta-learn test-time LoRA encoding (PERK) instead of long-context finetuning | Qwen2.5-7B · Average over 8 tasks (8K ctx) | accuracy: 66.3 -> 78.2 | Improved |
| Use PERK instead of FT-ICR when extrapolating to 128K | Qwen2.5-0.5B · BabiLong QA1 (128K ctx, extrapolation) | accuracy: 0 -> 61.4 | Improved |
| Add Yarn + DCA to FT-ICR for length extrapolation | FT-ICR Qwen2.5-0.5B · BabiLong QA1 (128K ctx, extrapolation) | accuracy: 0 -> 25.4 | Improved |
| Reduce LoRA rank from 256 to 16 (ablation) | PERK Qwen2.5-0.5B · BabiLong QA3 (1K ctx) | accuracy: 89.1 -> 84.7 | Hurt |
| Increase inner-loop steps from 2 to 4 (ablation) | PERK Qwen2.5-0.5B · BabiLong QA2 (1K ctx) | accuracy: 94.5 -> 98.1 | Improved |
| Truncate gradient unrolling for up to 3 of 4 inner steps (ablation) | PERK Qwen2.5-0.5B · BabiLong QA1 (truncation ablation) | accuracy: 100 -> 98.8 | No clear effect |
| Truncate all 4 inner steps (first-order approximation) (ablation) | PERK Qwen2.5-0.5B · BabiLong QA1 (truncation ablation) | accuracy: 100 -> 92.4 | Hurt |
| Train on contexts with relevant info at random positions, PERK vs FT-ICR | Qwen2.5-0.5B · Lost-in-the-Middle API retrieval | accuracy: +24% | Improved |
| Fixed-position training: PERK vs FT-ICR when train and test position match at beginning or end | Qwen2.5-0.5B · Lost-in-the-Middle API retrieval (fixed positions) | accuracy | Hurt |
| Increase gradient accumulation from 1 to 16 steps in PERK inference (ablation) | PERK 8K context · Inference efficiency | memory (GB) / runtime (s): 35.2GB -> 5.9GB; 1.9s -> 8.5s | Mixed |
| PERK instead of FT-ICR at 64K context | inference cost · Inference efficiency (64K tokens) | runtime (s): 32.6s -> 11.4s | Improved |
| PERK vs FT-ICR on GPT-2 to LLaMA-8B scales | BabiLong QA2 · BabiLong QA2 (8K, model scales) | accuracy: +9.3 points on LLaMA-8B (PERK 99.1) | Improved |
Experimental setup
Builds on: LoRA
Closest papers by meaning
- Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.92)
- LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.90)
- Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning (NeurIPS 2025 · similarity 0.90)
- MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning (NeurIPS 2026 · similarity 0.89)
- InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models (ICLR 2026 · similarity 0.89)
- Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
