Research map / Large language models / Instruction and fine-tuning
Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes instruction-data separation for single-turn LMs, proposes an empirically computable separation score and the SEP dataset, and finds that all tested LLMs separate poorly and that prompt engineering, prompt optimization, and fine-tuning each have limits.
Problem
LLMs do not explicitly separate instructions from data, leaving them vulnerable to indirect prompt injection; no established definition or benchmark quantifies this separation.
Approach
Define a separation score as the divergence between model behavior when a probe is placed in the instruction argument versus the data argument, and approximate it with an output-only empirical score based on surprise witnesses (words that appear when the probe is executed but rarely when it is processed).
Key results
- Naive empirical separation scores range from 13.3% (Phi-3-mini-4k) to 73.2% (Gemma-2B), indicating all evaluated models separate instructions from data poorly.
- Prompt engineering raised GPT-4 separation from 20.8% to 95.3% with utility rising from 83.3% to 96.6%, while Gemma-2B's utility dropped from 36.7% to 15.3%.
- DPO fine-tuning reached an average separation of 96% across open models but reduced utility by about 17.8 points on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply prompt engineering instead of the naive prompt setup | evaluated models (average excluding GPT models) · SEP | empirical separation score: +24%pt | Helped |
| Apply prompt engineering instead of the naive prompt setup | evaluated models (average excluding GPT models) · SEP | utility score: +1.3%pt | Mixed |
| Apply prompt engineering | GPT-4 · SEP | empirical separation score: 20.8% -> 95.3% | Helped |
| Apply prompt engineering | Gemma-2B · SEP | utility score: 36.7% -> 15.3% | Hurt |
| Apply prompt engineering | Gemma-7B · SEP | empirical separation score: 0.569 ± 0.008 -> 0.569 ± 0.008 | No clear effect |
| Apply prompt optimization instead of the naive prompt setup | models with white-box access (average) · SEP | empirical separation score: +7.2%pt | Helped |
| Apply prompt optimization instead of the naive prompt setup | models with white-box access (average) · SEP | utility score: +2.1%pt | No clear effect |
| Fine-tune with DPO instead of using the naive model | evaluated models (average) · SEP | empirical separation score: 96% | Helped |
| Fine-tune with DPO instead of using the naive model | evaluated models (average) · SEP | utility score: -17.8% | Hurt |
| Use larger or more capable models within the same family | GPT, Gemma, and Llama model families · SEP | empirical separation score | Hurt |
| Use prompt engineering rather than the naive prompt setup | Gemma-2B · SEP | empirical separation score: 73.2% -> 92.4% | Helped |
| Use fine-tuning rather than the naive prompt setup | Llama-2 7B · SEP | utility score: 0.833 ± 0.003 -> 0.165 ± 0.005 | Hurt |
Surprising findings
- Smaller or less capable models in a family often show higher separation than larger ones (e.g., GPT-3.5 over GPT-4, Llama-2 over Llama-3), suggesting scaling does not solve the problem.
- Fine-tuning achieves high separation but at a sharp cost to utility, creating a roughly negative trade-off across methods.
Limitations
- Empirical separation relies on a naive system/user prompt proxy and on short single-word witnesses, so it may not reflect true instruction-data separation.
- GPT-4 was used to generate SEP, which may give GPT models an unfair advantage in prompt engineering results.
- Experiments cannot prove impossibility of separation; only provide evidence for mitigation failure within the tested strategies.
Experimental setup
Datasets: SQuAD Alpaca SEP
Benchmarks: SEP
Metrics: empirical separation score utility score
Baselines: GPT-3.5 GPT-4 Gemma-2B Gemma-7B Phi-3-mini-4k Llama-3 (8B) Llama-2 (7B) Starling-LM-7B-beta Zephyr (7B) beta
Builds on: direct preference optimization low-rank adaptation gradient-based prompt optimization
Closest papers by meaning
- Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy (ICLR 2025 · similarity 0.90)
- Can Large Language Models Understand Real-World Complex Instructions? (AAAI 2024 · similarity 0.89)
- Benchmarking Complex Instruction-Following with Multiple Constraints Composition (NeurIPS 2024 · similarity 0.88)
- Do LLMs estimate uncertainty well in instruction-following? (ICLR 2025 · similarity 0.88)
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.88)
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.