Research map / Large language models / Instruction and fine-tuning
Soft Prompt Recovers Compressed LLMs, Transferably
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a vanilla soft prompt on C4 that is attached to pruned or quantized LLMs to recover their perplexity and downstream accuracy, and shows the prompt transfers across datasets, tasks, and compression levels.
Problem
Compressed LLMs (pruned or quantized) suffer significant accuracy degradation, and existing recovery methods require intricate engineering; the question is whether cheap prompt tuning can restore compressed model performance.
Approach
Apply vanilla soft prompt tuning (100 learnable prompt embeddings prepended to inputs) on top of a frozen compressed model, so the prompt is learned with gradients flowing through compressed weights; the learned prompt is then shown to transfer zero-shot across datasets, tasks, and compression settings.
Key results
- On LLaMA-7B with joint 50% sparsity and 4-bit quantization, a C4-learned soft prompt reduces Wikitext-2 PPL from 9.67 to 7.31 (full model 6.34).
- A prompt learned on C4 transfers zero-shot to Wikitext-2 and PTB, bringing 3-bit LLaMA-7B to on-par or better PPL than the full model on PTB and Wikitext-2 in the paper's transfer setting.
- On LLaMA-2-13B 3-bit, soft prompt (0.5M params) reaches C4 PPL 6.75 versus 8.15 for LoRA (26M params) and 6.96 for fp16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a learned soft prompt to 3-bit LLaMA-2-13B | LLaMA-2-13B · C4 validation | PPL: 9.24 -> 6.75 | Helped |
| Add a learned soft prompt to 3-bit BLOOM-7B | BLOOM-7B · C4 validation | PPL: 18.40 -> 13.54 | Helped |
| Use a transferred C4-trained soft prompt rather than no prompt | 3-bit LLaMA-2-13B · Wikitext2 test | PPL: 7.88 -> 5.89 | Helped |
| Use a transferred C4-trained soft prompt rather than no prompt | 3-bit BLOOM-7B · Wikitext2 test | PPL: 16.06 -> 12.42 | Helped |
| Add a C4-trained soft prompt to LLaMA-7B compressed with 50% sparsity and 4-bit quantization | LLaMA-7B · C4 validation | PPL: 10.94 -> 7.38 | Helped |
| Add a C4-trained soft prompt to LLaMA-7B compressed with 50% sparsity and 4-bit quantization | LLaMA-7B · Wikitext-2 test | PPL: 9.67 -> 7.31 | Helped |
| Add a C4-trained soft prompt to LLaMA-7B compressed with 50% sparsity and 4-bit quantization | LLaMA-7B · PTB test | PPL: 17.39 -> 10.64 | Helped |
| Use a target-trained rather than C4-transferred prompt | 4-bit LLaMA-7B · PTB test | PPL: 9.25 -> 7.04 | Helped |
| Add a transferred C4-trained soft prompt | 4-bit LLaMA-7B · PTB test | PPL: 11.65 -> 9.25 | Helped |
| Add a transferred C4-trained soft prompt | 3-bit LLaMA-7B · PTB test | PPL: 15.74 -> 10.81 | Helped |
| Add a transferred C4-trained soft prompt | 2-bit LLaMA-7B · PTB test | PPL: 5883.13 -> 29.82 | Helped |
| Increase the soft prompt from 0 to 100 tokens (ablation) | 3-bit LLaMA-7B · PTB test | PPL: 15.74 -> 7.76 | Helped |
Surprising findings
- Learned soft prompts transfer across datasets, tasks, and compression types, contrary to the belief that soft prompts are task-specific.
- Learned prompt embeddings have no close nearest-neighbor words in the vocabulary (cosine similarity below about 0.16), so they are not interpretable as natural language.
Limitations
- Relies on GPUs, which are expensive for resource-limited researchers.
- Transferred prompts are less competitive than prompts trained directly on the target domain.
- Prompt tokens are not interpretable as natural language words.
Experimental setup
Datasets: C4 Wikitext-2 Penn Treebank OpenbookQA Hellaswag PIQA High School European History
Benchmarks: C4 Wikitext-2 Penn Treebank OpenbookQA Hellaswag PIQA High School European History
Metrics: perplexity normalized accuracy
Baselines: SparseGPT GPTQ LoRA hard prompt
Builds on: prompt tuning (Lester et al., 2021) SparseGPT GPTQ LoRA
Closest papers by meaning
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.91)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
- No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization (NeurIPS 2025 · similarity 0.89)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.89)
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.