Research map / Large language models / Instruction and fine-tuning
PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
Haonan Wang, Brian K Chen, Siquan Li, Liang Xinhe, Hwee Kuan Lee, Kenji Kawaguchi, Tianyang Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move prefix out of attention head (PMT vs Prefix-Tuning) | Qwen2.5-3B-Instruct · BigBench Date Understanding | accuracy: 52.0 -> 76.6 | Improved |
| move prefix out of attention head (PMT vs Prefix-Tuning) | LLaMA2-7B-Chat · GoEmotions | accuracy: 5.6 -> 45.2 | Improved |
| PMT versus full fine-tuning | Qwen2.5-3B-Instruct · GoEmotions | accuracy: 37.8 -> 37.3 | No clear effect |
| PMT versus LoRA | LLaMA2-7B-Chat / Qwen2.5-3B · few-shot classification (3 datasets) | accuracy: LoRA 36.2 -> PMT 45.2 (GoEmotions, LLaMA2) | Improved |
| use GELU instead of ELU feature map (ablation) | PMT · GoEmotions / DBpedia / BigBench | accuracy: GoEmotions LLaMA2 45.2 -> 47.0; DBpedia Qwen 96.9 -> 96.4 | Mixed |
| use MLP kernel feature map instead of ELU (ablation) | PMT · BigBench Date Understanding | accuracy: LLaMA2 71.2 -> 64.5; Qwen 76.6 -> 77.1 | Mixed |
| PMT vs LoRA under DPO | Qwen2.5-3B · AlpacaEval 2 | win-rate delta: +3.52 -> +4.66 | Improved |
| PMT vs LoRA under SFT | Qwen2.5-3B · AlpacaEval 2 | win-rate delta: +0.49 -> +0.76 | Improved |
| PMT vs LoRA under SimPO | Qwen2.5-3B · AlpacaEval 2 | win-rate delta: +1.24 -> +1.74 | Improved |
| PMT vs LoRA with 50K CFT samples | Qwen2.5-Math-7B · Minerva-Math | accuracy: 23.9 -> 62.5 | Improved |
| PMT vs LoRA with 4K CFT samples | Qwen2.5-Math-7B · AMC23 | accuracy: 40.0 -> 42.5 | Improved |
| increase prefix length 16 to 64 in Prefix-Tuning (ablation) | Prefix-Tuning on Qwen2.5-3B · CKA similarity to base | CKA: 0.8802 -> 0.7957 | Hurt |
Closest papers by meaning
- Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (ICLR 2025 · similarity 0.90)
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.90)
- From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning (ICML 2024 · similarity 0.89)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations (ICLR 2024 · similarity 0.89)
- The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
