AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning

Zeming Chen, Angelika Romanou, Gail Weiss, Antoine Bosselut

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PERK, which meta-trains a model to encode long contexts into low-rank adapter weights via gradient updates at test time, enabling long-context reasoning as parameter-efficient test-time learning.

Problem

Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts.

Approach

Uses two nested optimization loops during meta-training: the inner loop rapidly encodes a context into a low-rank adapter via gradient updates at test time, while the outer loop meta-learns to make this encoding effective for reasoning.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Meta-learn test-time LoRA encoding (PERK) instead of long-context finetuningQwen2.5-7B · Average over 8 tasks (8K ctx)accuracy: 66.3 -> 78.2Improved
Use PERK instead of FT-ICR when extrapolating to 128KQwen2.5-0.5B · BabiLong QA1 (128K ctx, extrapolation)accuracy: 0 -> 61.4Improved
Add Yarn + DCA to FT-ICR for length extrapolationFT-ICR Qwen2.5-0.5B · BabiLong QA1 (128K ctx, extrapolation)accuracy: 0 -> 25.4Improved
Reduce LoRA rank from 256 to 16 (ablation)PERK Qwen2.5-0.5B · BabiLong QA3 (1K ctx)accuracy: 89.1 -> 84.7Hurt
Increase inner-loop steps from 2 to 4 (ablation)PERK Qwen2.5-0.5B · BabiLong QA2 (1K ctx)accuracy: 94.5 -> 98.1Improved
Truncate gradient unrolling for up to 3 of 4 inner steps (ablation)PERK Qwen2.5-0.5B · BabiLong QA1 (truncation ablation)accuracy: 100 -> 98.8No clear effect
Truncate all 4 inner steps (first-order approximation) (ablation)PERK Qwen2.5-0.5B · BabiLong QA1 (truncation ablation)accuracy: 100 -> 92.4Hurt
Train on contexts with relevant info at random positions, PERK vs FT-ICRQwen2.5-0.5B · Lost-in-the-Middle API retrievalaccuracy: +24%Improved
Fixed-position training: PERK vs FT-ICR when train and test position match at beginning or endQwen2.5-0.5B · Lost-in-the-Middle API retrieval (fixed positions)accuracyHurt
Increase gradient accumulation from 1 to 16 steps in PERK inference (ablation)PERK 8K context · Inference efficiencymemory (GB) / runtime (s): 35.2GB -> 5.9GB; 1.9s -> 8.5sMixed
PERK instead of FT-ICR at 64K contextinference cost · Inference efficiency (64K tokens)runtime (s): 32.6s -> 11.4sImproved
PERK vs FT-ICR on GPT-2 to LLaMA-8B scalesBabiLong QA2 · BabiLong QA2 (8K, model scales)accuracy: +9.3 points on LLaMA-8B (PERK 99.1)Improved

Experimental setup

Builds on: LoRA

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy