Research map / Large language models / Instruction and fine-tuning
Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
Zhengxiang Shi, Aldo Lipani
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TAPT continued pre-training | Prompt-based FT (hard) · CoLA, fully supervised | Matthews corr.: 54.7 -> 44.0 | Hurt |
| add TAPT continued pre-training | Prompt-based FT (soft) · QQP, fully supervised | F1: 73.9 -> 79.9 | Improved |
| add TAPT continued pre-training | Prompt-based FT (hard) · MNLI, semi-supervised | accuracy: 67.3 -> 50.7 | Hurt |
| add TAPT continued pre-training | CLS-based FT · 8 single-sentence tasks | accuracy: 7 out of 8 tasks | Improved |
| add TAPT continued pre-training | CLS-based FT · 8 sentence-pair tasks | accuracy: 6 out of 8 tasks | Hurt |
| add PCP continued pre-training | Prompt-based FT (hard) · CoLA, semi-supervised | Matthews corr.: 7.2 -> 25.0 | Improved |
| add PCP continued pre-training | Prompt-based FT (soft) · QNLI, semi-supervised | accuracy: 64.2 -> 84.3 | Improved |
| add PCP continued pre-training | Prompt-based FT · 16 tasks, fully supervised | accuracy: more than 1.0 | Improved |
| add PCP continued pre-training | Prompt-based FT · 16 tasks, semi-supervised | accuracy: 6.8 | Improved |
| replace semi-supervised methods with PCP-enhanced prompt FT | Prompt-based FT (hard) · Amazon Review, 200 labels | Macro-F1: 5.5 | Improved |
| compare PCP with FixMatch | Prompt-based FT (hard) + PCP · IMDB, 20 labels | Macro-F1: 95.26 vs 92.49 | Hurt |
| use only pseudo labels or only template in PCP (ablation) | Prompt FT + PCP (soft) · 8 single-sentence tasks, mean | mean score: 77.5 -> 73.5 (labels only), 72.2 (template only) | Hurt |
Closest papers by meaning
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.90)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.89)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.89)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.89)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.88)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
