Research map / Large language models / Reasoning and chain of thought

In-Place Test-Time Training

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes In-Place TTT, which turns the final MLP projection of pretrained LLMs into chunk-wise updated fast weights with an NTP-aligned target, enabling drop-in test-time training without architectural changes.

Problem

Test-time training for LLMs is hindered by needing specialized layers that require pretraining from scratch, sequential per-token updates that underuse parallel hardware, and generic reconstruction objectives misaligned with next-token prediction.

Approach

Treat the MLP down-projection W_down of existing Transformer blocks as fast weights updated chunk-wise during inference, using a value target built from a causal Conv1D over token embeddings plus a learned projection so the update aligns with next-token prediction.

Key results

• On RULER, Qwen3-4B-Base with In-Place TTT reaches 77.0 at 128k vs 74.8 for the baseline, and 43.9 vs 41.7 at 256k extrapolation.
• Extension to LLaMA-3.1-8B gives +2.1 at 64k (81.6 to 83.7); Qwen3-14B-Base gives +2.7 at 64k (67.9 to 70.6).
• From-scratch 4B full-attention model: RULER-16k rises from 6.58 to 19.99 with In-Place TTT.

Surprising findings

• Zero-initialized Conv1D and near-zero W_target keep the fast-weight update negligible at start, so pretrained behavior is preserved at initialization.

Limitations

• Proof relies on strong simplifying assumptions (near-orthogonal embeddings, expected key-query alignment) and a single-step induction setting.
• Core principles claimed orthogonal to loss and optimizer choice, but only one loss (similarity) and plain gradient step were explored.
• Gains on short contexts are small or negative in some settings (e.g., 4k RULER drops from 96.6 to 96.1 for Qwen3-4B).

Experimental setup

Datasets: In-house pretraining corpus In-house continual pretraining corpus (books, repo-level code, synthetic long-context QA) Pile Proof-Pile-2

Benchmarks: RULER HellaSwag ARC-E ARC-C MMLU PIQA Pile sliding window perplexity

Metrics: average accuracy perplexity prefill throughput peak memory

Baselines: Mistral-7B GLM3-6B Phi3-medium-14B Llama3-8B Qwen3-4B (Instruct) Sliding-window attention Transformer Gated Linear Attention DeltaNet LaCT

Builds on: Test-Time Training LaCT Gated MLP YaRN Multi-Token Prediction

Closest papers by meaning

• Rethinking Expressivity and Efficiency in Test-Time Training (NeurIPS 2026 · similarity 0.90)
• Test-Time Training Provably Improves Transformers as In-context Learners (ICML 2025 · similarity 0.90)
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.90)
• Learning What to Remember: Test-Time Training via Context Distillation (NeurIPS 2026 · similarity 0.90)
• Test-Time Learning for Large Language Models (ICML 2025 · similarity 0.90)
• Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.89)