AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention

Haonan Wang, Brian K Chen, Siquan Li, Liang Xinhe, Hwee Kuan Lee, Kenji Kawaguchi, Tianyang Hu

ICLR 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
move prefix out of attention head (PMT vs Prefix-Tuning)Qwen2.5-3B-Instruct · BigBench Date Understandingaccuracy: 52.0 -> 76.6Improved
move prefix out of attention head (PMT vs Prefix-Tuning)LLaMA2-7B-Chat · GoEmotionsaccuracy: 5.6 -> 45.2Improved
PMT versus full fine-tuningQwen2.5-3B-Instruct · GoEmotionsaccuracy: 37.8 -> 37.3No clear effect
PMT versus LoRALLaMA2-7B-Chat / Qwen2.5-3B · few-shot classification (3 datasets)accuracy: LoRA 36.2 -> PMT 45.2 (GoEmotions, LLaMA2)Improved
use GELU instead of ELU feature map (ablation)PMT · GoEmotions / DBpedia / BigBenchaccuracy: GoEmotions LLaMA2 45.2 -> 47.0; DBpedia Qwen 96.9 -> 96.4Mixed
use MLP kernel feature map instead of ELU (ablation)PMT · BigBench Date Understandingaccuracy: LLaMA2 71.2 -> 64.5; Qwen 76.6 -> 77.1Mixed
PMT vs LoRA under DPOQwen2.5-3B · AlpacaEval 2win-rate delta: +3.52 -> +4.66Improved
PMT vs LoRA under SFTQwen2.5-3B · AlpacaEval 2win-rate delta: +0.49 -> +0.76Improved
PMT vs LoRA under SimPOQwen2.5-3B · AlpacaEval 2win-rate delta: +1.24 -> +1.74Improved
PMT vs LoRA with 50K CFT samplesQwen2.5-Math-7B · Minerva-Mathaccuracy: 23.9 -> 62.5Improved
PMT vs LoRA with 4K CFT samplesQwen2.5-Math-7B · AMC23accuracy: 40.0 -> 42.5Improved
increase prefix length 16 to 64 in Prefix-Tuning (ablation)Prefix-Tuning on Qwen2.5-3B · CKA similarity to baseCKA: 0.8802 -> 0.7957Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy