Research map / Large language models / Instruction and fine-tuning
Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that prepending a fixed, task-agnostic prefix of cross-task classification demonstrations (TAPP) to every input improves zero-shot instruction following of base and instruction-tuned GPT-3, GPT-J, GPT-NeoX and OPT models on SuperNI.
Problem
Instruction-following in LLMs is usually obtained through fine-tuning (instruction tuning or RLHF), which needs backpropagation and weight access; the paper asks whether inference-time prompting alone can improve zero-shot instruction following.
Approach
A fixed prefix of K cross-task demonstrations (instruction, input, output), built with simple heuristics (classification tasks with explicit answer choices, non-overlapping answer choices, length limit, ordering), is prepended to every target-task input regardless of task, with no task-specific information needed.
Key results
- TAPP improves base LLMs by 34.58% and instruction-tuned models by 12.26% on average on SuperNI (119 held-out tasks).
- GPT-J (6B) with TAPP outperforms GPT-3 davinci (175B) without TAPP.
- TAPP boosts text-davinci-003 by 9.3%, the best-performing model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prepend fixed task-agnostic demonstrations instead of using no prefix prompt | GPT-3 davinci (175B) · SuperNI (12 tasks) | average score: 29.66 -> 44.24 | Helped |
| Prepend TAPP to 4-shot target-task demonstrations | GPT-3 davinci (175B) · SuperNI (12 tasks) | average score: 56.65 -> 60.21 | Helped |
| Replace demonstration inputs with random text (ablation) | TAPP on GPT-3 davinci (175B) · SuperNI (12 tasks) | average score: 44.24 -> 44.27 | No clear effect |
| Replace demonstration instructions with random text (ablation) | TAPP on GPT-3 davinci (175B) · SuperNI (12 tasks) | average score: 44.24 -> 31.18 | Hurt |
| Replace demonstration outputs with random words (ablation) | TAPP on GPT-3 davinci (175B) · SuperNI (12 tasks) | average score: 44.24 -> 38.30 | Hurt |
| Allow answer choices to overlap across demonstrations (ablation) | TAPP on GPT-3 davinci (175B) · SuperNI (12 tasks) | average score: 44.24 -> 42.30 | Hurt |
| Use demonstrations with overlapping answer choices (ablation) | TAPP on GPT-3 davinci (175B) · SuperNI classification tasks | classification score: 33.86 -> 35.14 | Mixed |
| Use demonstrations with overlapping answer choices (ablation) | TAPP on GPT-3 davinci (175B) · SuperNI generation tasks | generation score: 58.77 -> 52.32 | Hurt |
| Use output-only task-specific demonstrations instead of TAPP | GPT-3 davinci (175B) · SuperNI classification tasks | classification score: 43.67 -> 36.83 | Hurt |
| Use output-only task-specific demonstrations instead of TAPP | GPT-3 davinci (175B) · SuperNI generation tasks | generation score: 24.98 -> 51.93 | Helped |
| Prepend TAPP to increase attention to the target instruction | OPT-6.7B · 5 generation tasks | instruction-to-input attention ratio: 0.6172 -> 0.6758 | Helped |
| Apply TAPP to OPT-IML (30B) and FLAN-T5 | OPT-IML (30B); FLAN-T5 · SuperNI | performance | Hurt |
Surprising findings
- Corrupting the input instances of the demonstrations barely hurts performance, while corrupting instructions or outputs does.
- Demonstrations built from classification tasks with explicit answer choices help even generation target tasks more than retrieved similar demonstrations.
Limitations
- Increased inference cost from longer input sequences (mitigated by caching of the fixed prefix).
- Evaluation relies mainly on heuristic metrics (EM, ROUGE-L) with no human evaluation.
- Proposed mechanism (focus on instruction) is hypothetical and supported only by a preliminary attention analysis.
Experimental setup
Datasets: Super-NaturalInstructions CC-News
Benchmarks: Super-NaturalInstructions
Metrics: Exact Match ROUGE-L INSTRUCTSCORE
Baselines: Base Input (zero-shot) Nearest PP Category PP Output PP Few-shot ICL Z-ICL
Builds on: Super-NaturalInstructions Z-ICL few-shot in-context learning
Closest papers by meaning
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.90)
- SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models (ICLR 2025 · similarity 0.90)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
- Recursive Language Models (NeurIPS 2026 · similarity 0.89)
- PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs (NeurIPS 2025 · similarity 0.89)
- Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.