Research map / Large language models / Instruction and fine-tuning
Self-Adapting Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEAL, where an LLM generates its own finetuning data ("self-edits") for new inputs and is trained with ReST-EM-style RL using downstream performance after LoRA updates as reward, evaluated on SQuAD knowledge incorporation and ARC few-shot learning.
Problem
Static LLMs cannot adapt their weights to new tasks, knowledge, or examples; existing synthetic-data and test-time-training approaches use fixed heuristics rather than learned, task-specific adaptation strategies.
Approach
The model generates self-edits (implications, rewrites, or tool/hyperparameter specifications) that are applied as SFT updates, and the generation policy is trained via on-policy filtered behavior cloning (ReST-EM) with the post-update task accuracy as reward, so the model's own generation parameterizes its adaptation.
Key results
- On single-passage SQuAD knowledge incorporation with Qwen2.5-7B, SEAL reaches 47.0% vs 33.5% for passage-only finetuning and 46.3% for GPT-4.1 synthetic data.
- On ARC few-shot subset with Llama-3.2-1B, SEAL achieves 72.5% adaptation success vs 20% for self-edits without RL and 0% for ICL.
- In the n=200 continued pretraining setting SEAL reaches 58.2%, below GPT-4.1 synthetic data at 59.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the self-edit policy with RL instead of using base-model self-edits | Llama-3.2-1B-Instruct · ARC evaluation tasks | Success Rate (%): 20 -> 72.5 | Helped |
| Apply adaptation using RL-trained self-edits rather than no adaptation | Llama-3.2-1B-Instruct · ARC evaluation tasks | Success Rate (%): 0 -> 72.5 | Helped |
| Finetune on the passage only instead of leaving the model frozen | Qwen2.5-7B · SQuAD single passage (n = 1; LoRA) | accuracy: 32.7 -> 33.5 | No clear effect |
| Add base-model-generated synthetic implications to passage training | Qwen2.5-7B · SQuAD single passage (n = 1; LoRA) | accuracy: 33.5 -> 39.7 | Helped |
| Use RL-trained SEAL synthetic data instead of GPT-4.1 synthetic data | Qwen2.5-7B · SQuAD single passage (n = 1; LoRA) | accuracy: 46.3 -> 47.0 | Helped |
| Use RL-trained SEAL synthetic data instead of GPT-4.1 synthetic data | Qwen2.5-7B · SQuAD continued pretraining (n = 200; full-FT) | accuracy: 59.4 -> 58.2 | Hurt |
| Use RL-trained SEAL synthetic data instead of GPT-4.1 synthetic data | Qwen2.5-7B · SQuAD continued pretraining (n = 2067; full-FT) | accuracy: 49.2 -> 46.4 | Hurt |
| Replace the full post-finetuning reward with a rubric-based proxy reward (ablation) | SEAL · SQuAD single-passage knowledge incorporation | accuracy: 47.0 -> 45.6 | Hurt |
| Replace the short implications prompt with an implications-long prompt (ablation) | Qwen2.5-7B · SQuAD single-passage knowledge incorporation | accuracy: 39.7 -> 49.3 | Helped |
| Train the implications-long self-edit policy with RL (ablation) | Qwen2.5-7B · SQuAD single-passage knowledge incorporation | accuracy: 49.3 -> 54.4 | Helped |
| Train the rewrite self-edit policy with RL (ablation) | Qwen2.5-7B · SQuAD single-passage knowledge incorporation | accuracy: 49.4 -> 55.6 | Helped |
| Remove the self-edit format prompt (ablation) | Qwen2.5-7B · SQuAD single-passage knowledge incorporation | accuracy: 33.5 -> 18.9 | Hurt |
Surprising findings
- SEAL-generated self-edits outperform GPT-4.1-generated synthetic data in the single-passage setting despite the much smaller base model.
- Performance on earlier passages degrades as sequential self-edits accumulate, indicating catastrophic forgetting.
Limitations
- Catastrophic forgetting when applying sequential self-edits.
- High computational overhead because each reward requires finetuning and evaluating a model.
- Requires paired downstream evaluation questions, limiting scaling to unlabeled corpora.
Experimental setup
Datasets: SQuAD v1.1 ARC training set ARC evaluation set
Benchmarks: SQuAD (no-passage-in-context) ARC subset
Metrics: accuracy adaptation success rate
Baselines: ICL TTT + Self-Edit (w/o prior RL) Oracle TTT Base Model Train on Passage Train on Passage + Synthetic Train on Passage + GPT-4.1 Synthetic Generative Adapter Entigraph
Builds on: ReST-EM test-time training deductive closure training Generative Adapter
Closest papers by meaning
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.89)
- Transformer-Squared: Self-adaptive LLMs (ICLR 2025 · similarity 0.89)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.