Research map / Large language models / Instruction and fine-tuning
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GenerativeAdapter, a generator trained self-supervised on frozen LMs that maps streamed context to low-rank LoRA-style adapters in a single forward pass for knowledge injection, ICL and personalization.
Problem
Adapting frozen LLMs to new contexts currently requires either costly fine-tuning or prompting that adds inference overhead over long contexts.
Approach
A layer-specific adapter generator maps hidden states of the context (accumulated via outer products, with SVD normalization) to low-rank additive weight updates of a frozen base LM, trained end-to-end with reconstruction and completion next-token losses; the adapter is produced by forward passes only and updated incrementally as chunks arrive.
Key results
- On StreamingQA with Mistral-7B-Instruct at 32K context, GenerativeAdapter reaches F1 32.0 versus 19.5 for supervised fine-tuning (63.5% relative improvement reported in abstract).
- On MSC, GenerativeAdapter achieves F1 40.2 at 0.505 TFLOPS and 32M floats storage, versus full-conversation prompting at F1 66.0 with 2.059 TFLOPS and 128M+ floats.
- On MetaICL, GenerativeAdapter outperforms few-shot prompting in most cases, with larger gains on non-classification tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenerativeAdapter instead of supervised fine-tuning | Mistral-7B-Instruct · StreamingQA | F1: 19.5 -> 31.5 | Helped |
| use GenerativeAdapter instead of in-context prompting | Mistral-7B-Instruct · StreamingQA (512 tokens) | F1: 47.2 -> 51.5 | Helped |
| use GenerativeAdapter instead of in-context prompting | Mistral-7B-Instruct · SQuAD (512 tokens) | F1: 45.4 -> 48.8 | Helped |
| replace full-conversation prompting with GenerativeAdapter | Mistral-7B-Instruct · MSC | Inference computation (TFLOPS): 2.059 -> 0.505 | Helped |
| replace full-conversation prompting with GenerativeAdapter | Mistral-7B-Instruct · MSC | F1: 66.0 -> 40.2 | Hurt |
| replace Ultragist (512 Tokens) with GenerativeAdapter | Mistral-7B-Instruct · MSC | F1: 40.8 -> 40.2 | No clear effect |
| use both reconstruction and completion tasks instead of reconstruction task only (ablation) | GenerativeAdapter · Validation set (SlimPajama) | Completion perplexity: 34.34 -> 7.40 | Helped |
| use both reconstruction and completion tasks instead of completion task only (ablation) | GenerativeAdapter · Validation set (SlimPajama) | Reconstruction perplexity: 6.38 -> 1.75 | Helped |
| replace Frobenius normalization with SVD normalization (ablation) | GenerativeAdapter · Validation set (SlimPajama) | Reconstruction perplexity: 7.72 -> 1.75 | Helped |
| replace Frobenius normalization with SVD normalization (ablation) | GenerativeAdapter · Validation set (SlimPajama) | Completion perplexity: 7.32 -> 7.40 | No clear effect |
| add the adapter to the feedforward down projection instead of the default attention output projection (ablation) | GenerativeAdapter · Validation set (SlimPajama) | Reconstruction perplexity: 1.75 -> 1.68 | Helped |
| add the adapter to the feedforward down projection instead of the default attention output projection (ablation) | GenerativeAdapter · Validation set (SlimPajama) | Completion perplexity: 7.40 -> 7.26 | Helped |
Surprising findings
- Full-context in-context prompting degrades sharply beyond 8K tokens on SQuAD for both models, while the adapter keeps higher F1 at long lengths.
- Reconstruction-only training yields a large completion perplexity increase (7.40 to 34.34), indicating loss of general LM ability.
Limitations
- Only the attention output projection is adapted in main experiments; other modules left for future work.
- Generator has about 500M parameters and adapter about 32M, adding storage per context.
- Gains on long-context QA are relative to in-context prompting; closed-book fine-tuning baselines remain weaker than prompting at short lengths.
Experimental setup
Datasets: SlimPajama SQuAD StreamingQA MetaICL MSC COQA DROP NarrativeQA PubMedQA Quail MS MARCO BookSum
Benchmarks: SQuAD StreamingQA MetaICL MSC
Metrics: F1 accuracy TFLOPS extra storage (M floats) perplexity
Baselines: Zero-shot prompting Supervised fine-tuning Continuous pretraining In-context prompting UltraGist
Builds on: LoRA fast weight programmers Generative adapter-style meta-learning (amortization networks)
Closest papers by meaning
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.91)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.89)
- Doc-to-LoRA: Learning to Instantly Internalize Contexts (ICML 2026 · similarity 0.89)
- Online Adaptation of Language Models with a Memory of Amortized Contexts (NeurIPS 2024 · similarity 0.89)
- Large Language Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Self-Updatable Large Language Models by Integrating Context into Model Parameters (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.