AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning

Wenhang Shi, Yiren Chen, Shuqing Bian, Zhe Zhao, Jinhao Dong, Pengfei Hu, WEI LU, Xiaoyong Du

ICML 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train with different semantically equivalent paraphrased prompts (ablation)LLM fine-tuning on current task · NI-Probe sequencesin-task performanceNo clear effect
vary paraphrase of the training prompt (ablation)Llama-2-7b-chat · NI-Probe-M1forgetting / generalization change: largest difference across paraphrases 156% forgetting, 110% generalizationMixed
select training prompt by lowest pre-update losscontinual learning methods (LoraInc, EWC, O-Lora, InsCL) · NI-Seq-G1/C1/M1, TRACEAP, BWT, FWT: e.g. EWC AP +4.93 on NI-Seq-G1 (Llama2-7b-chat)Improved
add SAPO to InsCLInsCL on Llama2-7b-chat · NI-Seq-G1AP: 45.53 -> +0.29Improved
select the highest-loss prompt (SAPP) instead of the lowest (ablation)SAPO · NI-Seq-G1, NI-Seq-C1AP: LoraInc AP -0.67 vs SAPO +2.36 (NI-Seq-G1, Llama2-7b-chat)Hurt
use pre-update loss as selection signal (vs 9 other metrics) (ablation)prompt selection · 120 task sequences, Llama2-7b-chatPearson correlation with cross-task performanceImproved
use Rouge-L zero-shot as selection signal instead of loss (ablation)prompt selection · 120 task sequencesPearson correlationMixed
swap paraphraser from Gemini-2.5-Pro to GPT-OSS-120B or Qwen3-32B (ablation)SAPO with LoraInc / O-Lora · NI-Seq-M1, Qwen3-8bAP: LoraInc +1.89 / +1.47 / +1.66; O-Lora +1.07 / +1.11 / +0.55No clear effect
select lower-loss training prompts (ablation)Llama2-7b-chat gradients · NI-Probe-G1/M1inter-task gradient cosine similarityImproved
add SAPO to LoraIncLoraInc Llama2-7b-chat · NI-Seq-G1FWT: 6.87 -> -0.13No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy