Research map / Large language models / Retrieval and knowledge
Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that instruction-tuned LMs in retrieval-in-context RAG systems can be prompt-injected to regurgitate datastore text verbatim, scaling with model size, and tests mitigations and attacks on production GPTs.
Problem
Private non-parametric datastores in retrieval-augmented generation (RAG) systems can be extracted by black-box adversaries exploiting instruction-following in LMs via prompt injection.
Approach
Prompt-injected data extraction: an adversarial prompt asks the RAG LM to copy the text preceding the query (where retrieved chunks are prepended), exploiting instruction following; stronger instruction-following models leak more. Mitigations combine a safety-aware system prompt with position-insensitive encoding (PINE), and the attack is adapted to production GPTs with a tool-call prompt.
Key results
- Qwen1.5-Chat-72b reaches ROUGE-L 99.154 and BLEU 98.412 under the attack; Llama2-Chat-7b reaches ROUGE-L 80.369.
- Attack achieves 100% success rate on 25 customized GPTs with at most 2 queries.
- With 100 generated queries, reconstruction rate reaches 41.73% for the Harry Potter book and 3.22% for the 1.57M-word Wikipedia corpus on GPTs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the instruction-tuned model from 7b to 70b | Llama2-Chat · Wikipedia datastore | ROUGE-L: 80.369_{\pm 1.679} -> 89.567_{\pm 0.958} | Helped |
| Instruction-tune the base model (ablation) | Llama2-7b/13b, Mistral-7b, and Mixtral-8x7b · Wikipedia datastore | ROUGE: +65.76 | Helped |
| Use Harry Potter instead of Wikipedia as datastore (ablation) | Llama2-Chat · Harry Potter datastore | ROUGE-L: +12.446 (7b); +10.083 (13b); +5.743 (70b) | Helped |
| Increase the retrieved context size (ablation) | Mistral-Instruct-7b · Wikipedia datastore | Absolute reconstruction length | Helped |
| Use fewer, longer chunks rather than more, shorter chunks at fixed retrieved-context size (ablation) | Mistral-Instruct-7b · Wikipedia datastore | Absolute reconstruction length | Helped |
| Use semantic-aware chunking rather than fixed-size chunking (ablation) | Mistral-Instruct-7b · Wikipedia datastore | Reconstruction rate | Helped |
| Place the adversarial instruction near the beginning or end instead of the middle of the context (ablation) | Mistral-Instruct-7b · Wikipedia datastore | Reconstruction length | Helped |
| Add a safety-aware system prompt (ablation) | Llama3 8b Instruct · Wikipedia datastore | Reconstruction Rate: 88.88 -> 87.57 | No clear effect |
| Apply Position-Insensitive Encoding (PINE) (ablation) | Llama3 8b Instruct · Wikipedia datastore | Reconstruction Rate: 88.88 -> 58.03 | Helped |
| Combine safety-aware prompting with PINE (ablation) | Llama3 8b Instruct · Wikipedia datastore | Reconstruction Rate: 88.88 -> 52.34 | Helped |
| Attack customized GPTs by prompting execution of the myfiles_browser search function | 25 customized GPTs · GPTs datastore leakage | Attack success rate: 100\% | Helped |
| Use 100 GPT-generated questions as extraction queries | Harry Potter GPT · Harry Potter and the Sorcerer's Stone | Reconstruction rate: 41.73\% | Helped |
Surprising findings
- Instruction tuning raised ROUGE by about 65.76 on average, so stronger instruction following increases leakage.
- Leakage is higher when the datastore content is likely seen in pre-training (Harry Potter) than for unseen Wikipedia data.
Limitations
- Mitigation evaluated only on Llama3-8b-Instruct; safety-aware prompt alone gives marginal protection.
- GPT attack relies on daily usage limits, so only 100 queries were used for reconstruction.
- Anchor queries from WikiQA are obsolete; attack effectiveness with informed queries not fully measured.
Experimental setup
Datasets: Wikipedia articles created after November 1st 2023 (1165 articles, 1,569,000 words) WikiQA questions (230) as anchor queries Harry Potter and the Sorcerer's Stone text
Benchmarks: Wikipedia datastore extraction (WikiQA anchor queries) Harry Potter GPT reconstruction
Metrics: ROUGE-L BLEU F1 BERTScore absolute reconstruction length reconstruction rate attack success rate
Baselines: Llama2-Chat Mistral-Instruct Mixtral-Instruct Vicuna SOLAR WizardLM Qwen1.5-Chat Platypus2-Instruct
Builds on: retrieval-augmented generation PINE BM25
Closest papers by meaning
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.91)
- On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains (ICML 2025 · similarity 0.91)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.90)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.90)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.90)
- RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.