Research map / Large language models / Reasoning and chain of thought

Doc-to-LoRA: Learning to Instantly Internalize Contexts

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Doc-to-LoRA, a Perceiver-based hypernetwork meta-trained to emulate context distillation by mapping a document in one forward pass to a LoRA adapter that internalizes its content for a frozen target LLM.

Problem

Long contexts make LLM inference memory- and latency-heavy, while context distillation (CD) can internalize information into weights but per-document distillation is too slow and memory-hungry for interactive use.

Approach

Meta-train a Perceiver-style hypernetwork with the context-distillation KL objective across millions of contexts so that a single forward pass produces a context-specific rank-r LoRA; long contexts are chunked and their LoRAs concatenated along the rank dimension for composability.

Key results

• On NIAH, D2L achieves near-perfect zero-shot retrieval at contexts beyond 32K tokens (over 4x the 8K base window) despite training on up to 256 tokens.
• On SQuAD, D2L reaches 82.5% of the in-context upper bound and updates in under a second versus about 40 s for oracle CD and over 100 s for generated-query CD.
• On 2WikiMultihopQA, D2L attains 0.857 normalized performance vs 0.901 for oracle CD, with 0.209 s mean update latency versus 40.171 s.

Surprising findings

• A VLM (gemma-3-4b-it) used as context encoder lets the text-only gemma-2-2b-it classify Imagenette images at 75.03% accuracy, without any image training.
• Internalized knowledge can hurt unrelated queries: with replaced (distracting) contexts, D2L performance drops well below the base model, suggesting a bias toward assuming queries relate to the internalized context.

Limitations

• Meta-training is expensive (about 5 days on 8 H200 GPUs) and the hypernetwork must be retrained for each new target LLM.
• A performance gap remains between in-context learning and in-parameter knowledge methods, including D2L.
• Knowledge interference: internalized contexts degrade performance on unrelated queries, and no retention mechanism is included.

Experimental setup

Datasets: FineWeb-Edu PwC SQuAD ROPES DROP RULER LongBench

Benchmarks: SQuAD DROP ROPES 2WikiMultihopQA MultiFieldQA QASPER RULER needle-in-a-haystack Imagenette

Metrics: ROUGE-L F1 ROUGE-L recall ROUGE-L precision update latency peak update memory normalized performance

Baselines: Context distillation (oracle) Context distillation (generated queries) T2L LLMLingua-2 Generative Adapter Base model with context Base model without context

Builds on: context distillation T2L Generative Adapter Perceiver LoRA

Closest papers by meaning

• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
• Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.89)
• Text-to-LoRA: Instant Transformer Adaption (ICML 2025 · similarity 0.89)
• Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.89)
• Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.89)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.88)