Research map / Large language models / Instruction and fine-tuning
Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning
Wenhang Shi, Yiren Chen, Shuqing Bian, Zhe Zhao, Jinhao Dong, Pengfei Hu, WEI LU, Xiaoyong Du
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with different semantically equivalent paraphrased prompts (ablation) | LLM fine-tuning on current task · NI-Probe sequences | in-task performance | No clear effect |
| vary paraphrase of the training prompt (ablation) | Llama-2-7b-chat · NI-Probe-M1 | forgetting / generalization change: largest difference across paraphrases 156% forgetting, 110% generalization | Mixed |
| select training prompt by lowest pre-update loss | continual learning methods (LoraInc, EWC, O-Lora, InsCL) · NI-Seq-G1/C1/M1, TRACE | AP, BWT, FWT: e.g. EWC AP +4.93 on NI-Seq-G1 (Llama2-7b-chat) | Improved |
| add SAPO to InsCL | InsCL on Llama2-7b-chat · NI-Seq-G1 | AP: 45.53 -> +0.29 | Improved |
| select the highest-loss prompt (SAPP) instead of the lowest (ablation) | SAPO · NI-Seq-G1, NI-Seq-C1 | AP: LoraInc AP -0.67 vs SAPO +2.36 (NI-Seq-G1, Llama2-7b-chat) | Hurt |
| use pre-update loss as selection signal (vs 9 other metrics) (ablation) | prompt selection · 120 task sequences, Llama2-7b-chat | Pearson correlation with cross-task performance | Improved |
| use Rouge-L zero-shot as selection signal instead of loss (ablation) | prompt selection · 120 task sequences | Pearson correlation | Mixed |
| swap paraphraser from Gemini-2.5-Pro to GPT-OSS-120B or Qwen3-32B (ablation) | SAPO with LoraInc / O-Lora · NI-Seq-M1, Qwen3-8b | AP: LoraInc +1.89 / +1.47 / +1.66; O-Lora +1.07 / +1.11 / +0.55 | No clear effect |
| select lower-loss training prompts (ablation) | Llama2-7b-chat gradients · NI-Probe-G1/M1 | inter-task gradient cosine similarity | Improved |
| add SAPO to LoraInc | LoraInc Llama2-7b-chat · NI-Seq-G1 | FWT: 6.87 -> -0.13 | No clear effect |
Closest papers by meaning
- Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.90)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.90)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.90)
- Understanding Prompt Tuning and In-Context Learning via Meta-Learning (NeurIPS 2025 · similarity 0.90)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.90)
- Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
